Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Boltdb shipper doc fixes #3265

Merged
merged 3 commits into from
Feb 26, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/sources/configuration/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -1357,6 +1357,36 @@ filesystem:
# CLI flag: -local.chunk-directory
directory: <string>

# Configures storing index in an Object Store(GCS/S3/Azure/Swift/Filesystem) in the form of boltdb files.
# Required fields only required when boltdb-shipper is defined in config.
boltdb_shipper:
# Directory where ingesters would write boltdb files which would then be
# uploaded by shipper to configured storage
# CLI flag: -boltdb.shipper.active-index-directory
[active_index_directory: <string> | default = ""]

# Shared store for keeping boltdb files. Supported types: gcs, s3, azure,
# filesystem
# CLI flag: -boltdb.shipper.shared-store
[shared_store: <string> | default = ""]

# Cache location for restoring boltDB files for queries
# CLI flag: -boltdb.shipper.cache-location
[cache_location: <string> | default = ""]

# TTL for boltDB files restored in cache for queries
# CLI flag: -boltdb.shipper.cache-ttl
[cache_ttl: <duration> | default = 24h]

# Resync downloaded files with the storage
# CLI flag: -boltdb.shipper.resync-interval
[resync_interval: <duration> | default = 5m]

# Number of days of index to be kept downloaded for queries. Works only with
# tables created with 24h period.
# CLI flag: -boltdb.shipper.query-ready-num-days
[query_ready_num_days: <int> | default = 0]

# Cache validity for active index entries. Should be no higher than
# the chunk_idle_period in the ingester settings.
# CLI flag: -store.index-cache-validity
Expand Down
1 change: 0 additions & 1 deletion docs/sources/configuration/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,6 @@ storage_config:
boltdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index_cache
resync_interval: 5s
shared_store: s3

aws:
Expand Down
2 changes: 1 addition & 1 deletion docs/sources/operations/storage/boltdb-shipper.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ For all the queries which require chunks to be read from the store, Queriers als

Queriers lazily loads BoltDB files from shared object store to configured `cache_location`.
When a querier receives a read request, the query range from the request is resolved to period numbers and all the files for those period numbers are downloaded to `cache_location`, if not already.
Once we have downloaded files for a period we keep looking for updates in shared object store and download them every 15 Minutes by default.
Once we have downloaded files for a period we keep looking for updates in shared object store and download them every 5 Minutes by default.
Frequency for checking updates can be configured with `resync_interval` config.

To avoid keeping downloaded index files forever there is a ttl for them which defaults to 24 hours, which means if index files for a period are not used for 24 hours they would be removed from cache location.
Expand Down