Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: supports auto as default value for some mito configs #4513

Closed
wants to merge 4 commits into from
Closed

feat: supports auto as default value for some mito configs #4513

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
252752c
feat: supports auto as default value for several numeric configuratio…
killme2008 Aug 5, 2024
43bb7bc
fix: config api test failure and wrong configurations
killme2008 Aug 5, 2024
5758a64
fix: forgot macros
killme2008 Aug 5, 2024
f55f1de
chore: revert files
killme2008 Aug 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 26 additions & 20 deletions config/config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,11 +106,12 @@
| `region_engine.mito.compress_manifest` | Bool | `false` | Whether to compress manifest and checkpoint file by gzip (default false). |
| `region_engine.mito.max_background_jobs` | Integer | `4` | Max number of running background jobs |
| `region_engine.mito.auto_flush_interval` | String | `1h` | Interval to auto flush a region if it has not flushed yet. |
| `region_engine.mito.global_write_buffer_size` | String | `1GB` | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB. |
| `region_engine.mito.global_write_buffer_reject_size` | String | `2GB` | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size` |
| `region_engine.mito.sst_meta_cache_size` | String | `128MB` | Cache size for SST metadata. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/32 of OS memory with a max limitation of 128MB. |
| `region_engine.mito.vector_cache_size` | String | `512MB` | Cache size for vectors and arrow arrays. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/16 of OS memory with a max limitation of 512MB. |
| `region_engine.mito.page_cache_size` | String | `512MB` | Cache size for pages of SST row groups. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/16 of OS memory with a max limitation of 512MB. |
| `region_engine.mito.global_write_buffer_size` | String | `auto` | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB.<br/>- `auto`: automatically determine the value based on the system memory size (default)<br/>- `[size]` e.g. `1GB`: fixed size |
| `region_engine.mito.global_write_buffer_reject_size` | String | `auto` | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size`<br/>- `auto`: automatically determine the value based on the system memory size (default)<br/>- `[size]` e.g. `2GB`: fixed size |
| `region_engine.mito.sst_meta_cache_size` | String | `auto` | Cache size for SST metadata. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/32 of OS memory with a max limitation of 128MB.<br/>- `auto`: automatically determine the value based on the system memory size (default)<br/>- `[size]` e.g. `128MB`: fixed size |
| `region_engine.mito.vector_cache_size` | String | `auto` | Cache size for vectors and arrow arrays. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/16 of OS memory with a max limitation of 512MB.<br/>- `auto`: automatically determine the value based on the system memory size (default)<br/>- `[size]` e.g. `512MB`: fixed size |
| `region_engine.mito.page_cache_size` | String | `auto` | Cache size for pages of SST row groups. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/8 of OS memory.<br/>- `auto`: automatically determine the value based on the system memory size (default)<br/>- `[size]` e.g. `1GB`: fixed size |
| `region_engine.mito.selector_result_cache_size` | String | `auto` | Cache size for time series selector (e.g. `last_value()`). Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/16 of OS memory with a max limitation of 512MB.<br/>- `auto`: automatically determine the value based on the system memory size (default)<br/>- `[size]` e.g. `512MB`: fixed size |
| `region_engine.mito.enable_experimental_write_cache` | Bool | `false` | Whether to enable the experimental write cache. |
| `region_engine.mito.experimental_write_cache_path` | String | `""` | File system path for write cache, defaults to `{data_home}/write_cache`. |
| `region_engine.mito.experimental_write_cache_size` | String | `512MB` | Capacity for write cache. |
Expand Down Expand Up @@ -152,7 +153,7 @@
| `export_metrics` | -- | -- | The datanode can export its metrics and send to Prometheus compatible service (e.g. send to `greptimedb` itself) from remote-write API.<br/>This is only used for `greptimedb` to export its own metrics internally. It's different from prometheus scrape. |
| `export_metrics.enable` | Bool | `false` | whether enable export metrics. |
| `export_metrics.write_interval` | String | `30s` | The interval of export metrics. |
| `export_metrics.self_import` | -- | -- | For `standalone` mode, `self_import` is recommend to collect metrics generated by itself |
| `export_metrics.self_import` | -- | -- | For `standalone` mode, `self_import` is recommend to collect metrics generated by itself<br/>You must create the database by yourself. |
| `export_metrics.self_import.db` | String | `None` | -- |
| `export_metrics.remote_write` | -- | -- | -- |
| `export_metrics.remote_write.url` | String | `""` | The url the metrics send to. The url example can be: `http://127.0.0.1:4000/v1/prometheus/write?db=information_schema`. |
Expand All @@ -168,12 +169,6 @@
| Key | Type | Default | Descriptions |
| --- | -----| ------- | ----------- |
| `default_timezone` | String | `None` | The default timezone of the server. |
| `runtime` | -- | -- | The runtime options. |
| `runtime.global_rt_size` | Integer | `8` | The number of threads to execute the runtime for global read operations. |
| `runtime.compact_rt_size` | Integer | `4` | The number of threads to execute the runtime for global write operations. |
| `heartbeat` | -- | -- | The heartbeat options. |
| `heartbeat.interval` | String | `18s` | Interval for sending heartbeat messages to the metasrv. |
| `heartbeat.retry_interval` | String | `3s` | Interval for retrying to send heartbeat messages to the metasrv. |
| `http` | -- | -- | The HTTP server options. |
| `http.addr` | String | `127.0.0.1:4000` | The address to bind the HTTP server. |
| `http.timeout` | String | `30s` | HTTP request timeout. Set to 0 to disable timeout. |
Expand Down Expand Up @@ -212,6 +207,12 @@
| `prom_store` | -- | -- | Prometheus remote storage options |
| `prom_store.enable` | Bool | `true` | Whether to enable Prometheus remote write and read in HTTP API. |
| `prom_store.with_metric_engine` | Bool | `true` | Whether to store the data from Prometheus remote write in metric engine. |
| `runtime` | -- | -- | The runtime options. |
| `runtime.global_rt_size` | Integer | `8` | The number of threads to execute the runtime for global read operations. |
| `runtime.compact_rt_size` | Integer | `4` | The number of threads to execute the runtime for global write operations. |
| `heartbeat` | -- | -- | The heartbeat options. |
| `heartbeat.interval` | String | `18s` | Interval for sending heartbeat messages to the metasrv. |
| `heartbeat.retry_interval` | String | `3s` | Interval for retrying to send heartbeat messages to the metasrv. |
| `meta_client` | -- | -- | The metasrv client options. |
| `meta_client.metasrv_addrs` | Array | -- | The addresses of the metasrv. |
| `meta_client.timeout` | String | `3s` | Operation timeout. |
Expand All @@ -237,7 +238,7 @@
| `export_metrics` | -- | -- | The datanode can export its metrics and send to Prometheus compatible service (e.g. send to `greptimedb` itself) from remote-write API.<br/>This is only used for `greptimedb` to export its own metrics internally. It's different from prometheus scrape. |
| `export_metrics.enable` | Bool | `false` | whether enable export metrics. |
| `export_metrics.write_interval` | String | `30s` | The interval of export metrics. |
| `export_metrics.self_import` | -- | -- | For `standalone` mode, `self_import` is recommend to collect metrics generated by itself |
| `export_metrics.self_import` | -- | -- | For `standalone` mode, `self_import` is recommend to collect metrics generated by itself<br/>You must create the database by yourself. |
| `export_metrics.self_import.db` | String | `None` | -- |
| `export_metrics.remote_write` | -- | -- | -- |
| `export_metrics.remote_write.url` | String | `""` | The url the metrics send to. The url example can be: `http://127.0.0.1:4000/v1/prometheus/write?db=information_schema`. |
Expand Down Expand Up @@ -299,7 +300,7 @@
| `export_metrics` | -- | -- | The datanode can export its metrics and send to Prometheus compatible service (e.g. send to `greptimedb` itself) from remote-write API.<br/>This is only used for `greptimedb` to export its own metrics internally. It's different from prometheus scrape. |
| `export_metrics.enable` | Bool | `false` | whether enable export metrics. |
| `export_metrics.write_interval` | String | `30s` | The interval of export metrics. |
| `export_metrics.self_import` | -- | -- | For `standalone` mode, `self_import` is recommend to collect metrics generated by itself |
| `export_metrics.self_import` | -- | -- | For `standalone` mode, `self_import` is recommend to collect metrics generated by itself<br/>You must create the database by yourself. |
| `export_metrics.self_import.db` | String | `None` | -- |
| `export_metrics.remote_write` | -- | -- | -- |
| `export_metrics.remote_write.url` | String | `""` | The url the metrics send to. The url example can be: `http://127.0.0.1:4000/v1/prometheus/write?db=information_schema`. |
Expand All @@ -323,6 +324,10 @@
| `rpc_runtime_size` | Integer | `None` | Deprecated, use `grpc.runtime_size` instead. |
| `rpc_max_recv_message_size` | String | `None` | Deprecated, use `grpc.rpc_max_recv_message_size` instead. |
| `rpc_max_send_message_size` | String | `None` | Deprecated, use `grpc.rpc_max_send_message_size` instead. |
| `http` | -- | -- | The HTTP server options. |
| `http.addr` | String | `127.0.0.1:4000` | The address to bind the HTTP server. |
| `http.timeout` | String | `30s` | HTTP request timeout. Set to 0 to disable timeout. |
| `http.body_limit` | String | `64MB` | HTTP request body limit.<br/>The following units are supported: `B`, `KB`, `KiB`, `MB`, `MiB`, `GB`, `GiB`, `TB`, `TiB`, `PB`, `PiB`.<br/>Set to 0 to disable limit. |
| `grpc` | -- | -- | The gRPC server options. |
| `grpc.addr` | String | `127.0.0.1:3001` | The address to bind the gRPC server. |
| `grpc.hostname` | String | `127.0.0.1` | The hostname advertised to the metasrv,<br/>and used for connections from outside the host |
Expand Down Expand Up @@ -395,11 +400,12 @@
| `region_engine.mito.compress_manifest` | Bool | `false` | Whether to compress manifest and checkpoint file by gzip (default false). |
| `region_engine.mito.max_background_jobs` | Integer | `4` | Max number of running background jobs |
| `region_engine.mito.auto_flush_interval` | String | `1h` | Interval to auto flush a region if it has not flushed yet. |
| `region_engine.mito.global_write_buffer_size` | String | `1GB` | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB. |
| `region_engine.mito.global_write_buffer_reject_size` | String | `2GB` | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size` |
| `region_engine.mito.sst_meta_cache_size` | String | `128MB` | Cache size for SST metadata. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/32 of OS memory with a max limitation of 128MB. |
| `region_engine.mito.vector_cache_size` | String | `512MB` | Cache size for vectors and arrow arrays. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/16 of OS memory with a max limitation of 512MB. |
| `region_engine.mito.page_cache_size` | String | `512MB` | Cache size for pages of SST row groups. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/16 of OS memory with a max limitation of 512MB. |
| `region_engine.mito.global_write_buffer_size` | String | `auto` | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB.<br/>- `auto`: automatically determine the value based on the system memory size (default)<br/>- `[size]` e.g. `1GB`: fixed size |
| `region_engine.mito.global_write_buffer_reject_size` | String | `auto` | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size`<br/>- `auto`: automatically determine the value based on the system memory size (default)<br/>- `[size]` e.g. `2GB`: fixed size |
| `region_engine.mito.sst_meta_cache_size` | String | `auto` | Cache size for SST metadata. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/32 of OS memory with a max limitation of 128MB.<br/>- `auto`: automatically determine the value based on the system memory size (default)<br/>- `[size]` e.g. `128MB`: fixed size |
| `region_engine.mito.vector_cache_size` | String | `auto` | Cache size for vectors and arrow arrays. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/16 of OS memory with a max limitation of 512MB.<br/>- `auto`: automatically determine the value based on the system memory size (default)<br/>- `[size]` e.g. `512MB`: fixed size |
| `region_engine.mito.page_cache_size` | String | `auto` | Cache size for pages of SST row groups. Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/8 of OS memory.<br/>- `auto`: automatically determine the value based on the system memory size (default)<br/>- `[size]` e.g. `1GB`: fixed size |
| `region_engine.mito.selector_result_cache_size` | String | `auto` | Cache size for time series selector (e.g. `last_value()`). Setting it to 0 to disable the cache.<br/>If not set, it's default to 1/16 of OS memory with a max limitation of 512MB.<br/>- `auto`: automatically determine the value based on the system memory size (default)<br/>- `[size]` e.g. `512MB`: fixed size |
| `region_engine.mito.enable_experimental_write_cache` | Bool | `false` | Whether to enable the experimental write cache. |
| `region_engine.mito.experimental_write_cache_path` | String | `""` | File system path for write cache, defaults to `{data_home}/write_cache`. |
| `region_engine.mito.experimental_write_cache_size` | String | `512MB` | Capacity for write cache. |
Expand Down Expand Up @@ -439,7 +445,7 @@
| `export_metrics` | -- | -- | The datanode can export its metrics and send to Prometheus compatible service (e.g. send to `greptimedb` itself) from remote-write API.<br/>This is only used for `greptimedb` to export its own metrics internally. It's different from prometheus scrape. |
| `export_metrics.enable` | Bool | `false` | whether enable export metrics. |
| `export_metrics.write_interval` | String | `30s` | The interval of export metrics. |
| `export_metrics.self_import` | -- | -- | For `standalone` mode, `self_import` is recommend to collect metrics generated by itself |
| `export_metrics.self_import` | -- | -- | For `standalone` mode, `self_import` is recommend to collect metrics generated by itself<br/>You must create the database by yourself. |
| `export_metrics.self_import.db` | String | `None` | -- |
| `export_metrics.remote_write` | -- | -- | -- |
| `export_metrics.remote_write.url` | String | `""` | The url the metrics send to. The url example can be: `http://127.0.0.1:4000/v1/prometheus/write?db=information_schema`. |
Expand Down
40 changes: 34 additions & 6 deletions config/datanode.example.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,17 @@ rpc_max_recv_message_size = "512MB"
## +toml2docs:none-default
rpc_max_send_message_size = "512MB"

## The HTTP server options.
[http]
## The address to bind the HTTP server.
addr = "127.0.0.1:4000"
## HTTP request timeout. Set to 0 to disable timeout.
timeout = "30s"
## HTTP request body limit.
## The following units are supported: `B`, `KB`, `KiB`, `MB`, `MiB`, `GB`, `GiB`, `TB`, `TiB`, `PB`, `PiB`.
## Set to 0 to disable limit.
body_limit = "64MB"

## The gRPC server options.
[grpc]
## The address to bind the gRPC server.
Expand Down Expand Up @@ -348,22 +359,38 @@ max_background_jobs = 4
auto_flush_interval = "1h"

## Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB.
global_write_buffer_size = "1GB"
## - `auto`: automatically determine the value based on the system memory size (default)
## - `[size]` e.g. `1GB`: fixed size
global_write_buffer_size = "auto"

## Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size`
global_write_buffer_reject_size = "2GB"
## - `auto`: automatically determine the value based on the system memory size (default)
## - `[size]` e.g. `2GB`: fixed size
global_write_buffer_reject_size = "auto"

## Cache size for SST metadata. Setting it to 0 to disable the cache.
## If not set, it's default to 1/32 of OS memory with a max limitation of 128MB.
sst_meta_cache_size = "128MB"
## - `auto`: automatically determine the value based on the system memory size (default)
## - `[size]` e.g. `128MB`: fixed size
sst_meta_cache_size = "auto"

## Cache size for vectors and arrow arrays. Setting it to 0 to disable the cache.
## If not set, it's default to 1/16 of OS memory with a max limitation of 512MB.
vector_cache_size = "512MB"
## - `auto`: automatically determine the value based on the system memory size (default)
## - `[size]` e.g. `512MB`: fixed size
vector_cache_size = "auto"

## Cache size for pages of SST row groups. Setting it to 0 to disable the cache.
## If not set, it's default to 1/8 of OS memory.
## - `auto`: automatically determine the value based on the system memory size (default)
## - `[size]` e.g. `1GB`: fixed size
page_cache_size = "auto"

## Cache size for time series selector (e.g. `last_value()`). Setting it to 0 to disable the cache.
## If not set, it's default to 1/16 of OS memory with a max limitation of 512MB.
page_cache_size = "512MB"
## - `auto`: automatically determine the value based on the system memory size (default)
## - `[size]` e.g. `512MB`: fixed size
selector_result_cache_size = "auto"

## Whether to enable the experimental write cache.
enable_experimental_write_cache = false
Expand Down Expand Up @@ -516,9 +543,10 @@ enable = false
write_interval = "30s"

## For `standalone` mode, `self_import` is recommend to collect metrics generated by itself
## You must create the database by yourself.
[export_metrics.self_import]
## +toml2docs:none-default
db = "information_schema"
db = "greptime_metrics"

[export_metrics.remote_write]
## The url the metrics send to. The url example can be: `http://127.0.0.1:4000/v1/prometheus/write?db=information_schema`.
Expand Down
33 changes: 17 additions & 16 deletions config/frontend.example.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,6 @@
## +toml2docs:none-default
default_timezone = "UTC"

## The runtime options.
[runtime]
## The number of threads to execute the runtime for global read operations.
global_rt_size = 8
## The number of threads to execute the runtime for global write operations.
compact_rt_size = 4

## The heartbeat options.
[heartbeat]
## Interval for sending heartbeat messages to the metasrv.
interval = "18s"

## Interval for retrying to send heartbeat messages to the metasrv.
retry_interval = "3s"

## The HTTP server options.
[http]
## The address to bind the HTTP server.
Expand Down Expand Up @@ -128,6 +113,21 @@ enable = true
## Whether to store the data from Prometheus remote write in metric engine.
with_metric_engine = true

## The runtime options.
[runtime]
## The number of threads to execute the runtime for global read operations.
global_rt_size = 8
## The number of threads to execute the runtime for global write operations.
compact_rt_size = 4
Comment on lines +116 to +121
Copy link
Contributor

@evenyag evenyag Aug 6, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zyy17 Maybe toml2docs can provide a special marker so that we can comment out some options. e.g.

Suggested change
## The runtime options.
[runtime]
## The number of threads to execute the runtime for global read operations.
global_rt_size = 8
## The number of threads to execute the runtime for global write operations.
compact_rt_size = 4
## toml2docs:example-start
## The runtime options.
## [runtime]
## ## The number of threads to execute the runtime for global read operations.
## global_rt_size = 8
## ## The number of threads to execute the runtime for global write operations.
## compact_rt_size = 4
## toml2docs:example-end

We can also choose a special marker like ###, ##!.

##! global_rt_size = 8
##! example_option = "test"

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeap, it's one of the solution that I'm plan to do.


## The heartbeat options.
[heartbeat]
## Interval for sending heartbeat messages to the metasrv.
interval = "18s"

## Interval for retrying to send heartbeat messages to the metasrv.
retry_interval = "3s"

## The metasrv client options.
[meta_client]
## The addresses of the metasrv.
Expand Down Expand Up @@ -200,9 +200,10 @@ enable = false
write_interval = "30s"

## For `standalone` mode, `self_import` is recommend to collect metrics generated by itself
## You must create the database by yourself.
[export_metrics.self_import]
## +toml2docs:none-default
db = "information_schema"
db = "greptime_metrics"

[export_metrics.remote_write]
## The url the metrics send to. The url example can be: `http://127.0.0.1:4000/v1/prometheus/write?db=information_schema`.
Expand Down
Loading