2.2 Configuration

Configuration files aren't mandatory but they provide a convenient way to specify options that you use regularly when running EventQL programs, so you don't have to enter them on the command line each time you run the program.

Configuration File Format

EventQL uses the INI file format, that is a simple text file grouped into sections that consists of key-value pairs.

[client]
host = prod.example.com ; execute queries on this server

You can override every option set in the configuration file by using the command line option -C followed by the corresponding section.key=value pair.

$ evql -C client.host=localhost

Configuration File Path

If no explicit config path is provided using the --config option, EventQL will search for the configuration file at the following locations:

Binaries Config Search Paths
evqld /etc/evqld.conf
evql, evqlctl /etc/evql.conf
~/.evql.conf

Configuration Options

The EventQL configuration options are grouped in three sections: client, server and cluster.

Option Default Value Description
cluster.*
cluster.name The name of the cluster
cluster.coordinator The cluster coordinator service. Legal values: "zookeeper"
cluster.zookeeper_hosts A comma-separated list of zookeeper hosts (only used when cluster.coordinator=zookeeper)
cluster.rebalance_interval 60000000
cluster.allowed_hosts A comma-separated list of CIDR network ranges that are allowed to connect as internal nodes to the cluster. This setting does not affect which hosts are allowed to connect as a client. You can set this option to "0.0.0.0/0" to allow all hosts to connect as internal nodes.
cluster.allow_anonymous true Allow anonymous users to connect to the cluster Note: this does not circumvent client auth or any other ACLs. It merely controls if an anonymous user is even allowed to connect, let alone execute an operation.
cluster.allow_drop_table true If false, DROP TABLE is globally forbidden, regardless ACLs.
cluster.allow_create_database true If false, CREATE DATABASE is globally forbidden, regardless of ACLs.
server.*
server.datadir The location of the EvenQL data directory (mandatory)
server.listen The address (host:port) on which the server should listen. NOTE that this address is published to the coordinator service and must be a reachable by all other servers in the cluster. I.e. you can't use localhost or 0.0.0.0. (mandatory)
server.name The name of the server (optional)
server.pidfile If set, the server will write a pidfile to the provided path and aquire an exclusive lock on the pidfile. If the exclusive lock fails, the server will exit.
server.daemonize false
server.indexbuild_threads 2 The number of background compaction threads to start
server.replication_threads_max 4 The max number of background replication threads to start
server.client_auth_backend
server.internal_auth_backend
server.noleader false If set to true, this server will not partake in leader election
server.noalloc false If set to true, no partitions/data will be allocated to this server
server.gc_mode MANUAL
server.gc_interval 30000000
server.cachedir_maxsize 68719476736 Unit: Bytes
server.disk_capacity The maximum number of bytes that the server is allowed to write/use on disk. Unit is Bytes. This is an optional limit, if it is unset, the server will use the actual number of free bytes on disk as the limit. Even if the limit is set and allows using more disk space than is actually available, the server will use the (smaller) real limit.
server.loadinfo_publish_interval 15m

How often should the server publish it's current load info (i.e disk usage and other stats) to the cluster. Unit is microseconds. The load info is used when deciding on which server to allocate new chunks, so a shorter interval and therefore more up-to-date load info is usually better.

However, making the interval smaller will increase the load on the coordination service (e.g. ZooKeeper). The QPS to to the coordination service can be calculated using "num_servers / interval_in_s". So with the default value of 15 minutes and 1,000 servers we will have roughly 1 write QPS to Zookeeper (good). With 10,000 servers we have 10 write QPS (still okay).

server.load_limit_soft 0.95 This value controls the upper limit on a servers disk utilization. If the soft load limit was reached, now new partitions will be allocated on the server. (Default: 95%)
server.load_limit_hard 0.98 This value controls the upper limit on a servers disk utilization. If the soft load limit was reached, now new partitions will be allocated on the server and the cluster will slowly start to remove existing partitions from the server until the disk usage falls below the soft limit.
server.partitions_loading_limit_soft 4 The maximum number of partitions that can be loading on a server at the same time before allocations with priorty "IDLE" will stop being placed on the server. Allocations with priorty "BEST_EFFORT" or "MUST_ALLOCATE" may still be placed on the server even after the soft limit was reached.
server.partitions_loading_limit_hard 64 The maximum number of partitions that can be loading on a server at the same time before allocations with priorty "IDLE" and "BEST_EFFORT" will stop being placed on the server. Allocations with priority "MUST_ALLOCATE" may still be placed on the server even after the soft limit was reached.
server.c2s_io_timeout 60s How long should the server wait for data on a connection to a client when it expects the data to arrive immediately. (optional, unit: microseconds)
server.c2s_idle_timeout 30min How long should the server wait for new data on an idle connection to a client. An idle connection is a connection where no data is expected to arrive immediately. (optional, unit: microseconds)
server.s2s_io_timeout 10s How long should the server wait for data on a connection to another server when it expects the data to arrive immediately. (optional, unit: microseconds)
server.s2s_idle_timeout 10s How long should the server wait for new data on an idle connection to another server. An idle connection is a connection where no data is expected to arrive immediately. (optional, unit: microseconds)
server.s2s_pool_max_connections unlimited Limit how many (internal) connections each server should keep in its connection pool. Set this value to zero to turn off connection pooling.
server.s2s_pool_max_connections_per_host unlimited Limit how many (internal) connections each server should keep in its connection pool per each distinct remote endpoint.
server.s2s_pool_linger_timeout 1s Controls after which time an idle connection is removed (and closed) from the connection pool. Unit is microseconds.
server.http_io_timeout 1s Configures the HTTP I/O timeout. The timeout controls how long the server will wait for the client to send the next byte of the request while reading the http request as well as how long the server will wait for the client to read the next byte of the response while writing the response. (optional, unit: microseconds)
server.heartbeat_interval 1s How often should the server send a keepalive/heartbeat frame on a busy connection. Note that this value must be lower than the idle timeout and also puts a lower limit on the idle timeout that a connection client may choose. (optional, unit: microseconds)
server.query_progress_rate_limit 250ms How often should the server send a progress event. (optional, unit: microseconds)
server.query_max_concurrent_shards 256 The default maximum number of shards to be executed in parallel/ concurrently for a single query. In other words this setting limits the maximum parallelism for a query. You should consider increasing the value if you're running on more than 64 machines.
server.query_max_concurrent_shards_per_host 6 The default maximum number of shards to be executed on any given host for a single query.
server.query_failed_shard_policy tolerate The failed shard policy can either be "tolerate" or "error". If the value is "tolerate" failed shards will be ignore/excluded from the query result (the percentage of 'missing data' will be returned with each result). If the value is "error" any failed shard will result in a query error. Valid values: "tolerate", "error"
client.*
client.host localhost The hostname of the EventQL server
client.port 9175 The port of the EventQL server
client.database The database that should be used for following queries (optional)
client.user $USER Username to use when connecting to server (optional)
client.password Password to use when connecting to server (optional)
client.auth_token Auth-Token to use when connecting to server (optional)
client.timeout 60s Timeout to use when connecting to server (unit is microseconds)
client.history_file $HOME/.evql_history Where to write the interactive shell history file
client.history_maxlen 1024 Maximum number of entries in the interactive shell history file