Deploy: add docker-compose cluster deployment file #2740
Conversation
|
|
||
| http_prefix: | ||
|
|
||
| server: |
There was a problem hiding this comment.
I'd like suggest better ingester config:
ingester:
chunk_block_size: 262144
chunk_encoding: snappy
chunk_idle_period: 2h
chunk_retain_period: 6m
chunk_target_size: 1.572864e+06
There was a problem hiding this comment.
I just realized you have 2 configs but you could have only one and use -target=all or -target=query-frontend
| align_queries_with_step: true | ||
| max_retries: 5 | ||
| # parallelize queries in 15min intervals | ||
| split_queries_by_interval: 15m |
There was a problem hiding this comment.
I feel like we could also // by shard using parallelise_shardable_queries: true
There was a problem hiding this comment.
I have no idea how you learned all of this but wow ⭐ this is awesome work.
My only suggestion here is related to config. I think we can make this even more simple using a single config.
I'm super excited about this cause this is a great example of how to run Loki and without Kubernetes.
|
Hi, @cyriltovena thanks for your encouraging words (at the beginning of the work I was not sure how it will be acceptable to deploy loki without k8s). I'll update the comment to address your comments. |
There was a problem hiding this comment.
Awesome work @huikang 🚀 🎉
@cyriltovena Let me introduce you to @huikang are newest member in OpenShift Logging Explore from IBM. He is the member I mentioned during the last bug scrub willing to imrove our loki-benchmarks and in turn local loki deployments w/o k8s.
|
|
||
| server { | ||
| listen 80; | ||
| proxy_set_header X-Scope-OrgID 1; |
There was a problem hiding this comment.
nit. I suggest to use a meaningful name as OrgID. These IDs are also represented on the block storage layer as folder names as they represent the tenants.
|
|
||
| } | ||
|
|
||
| server { |
There was a problem hiding this comment.
What is the purpose for this extra server here?
There was a problem hiding this comment.
This is for load-balancing the requests from query-front end to the queriers. To avoid conflict with line 53, the nginx for the queriers listens on a different port, which is this server.
| - Together they form a cluster ring based on memberlist mechanism (if using consul/etcd, modules can be separate for further separate read/write workloads) | ||
| - Index data are stored and replicated through botldb-shipper | ||
| - Replication_factor=2: the receiving distributor sends log data to 2 ingesters based on consistent hashing | ||
| - Chunk storage is a shared directory mounted from the same host directory (to simulate S3 or gcs) |
There was a problem hiding this comment.
@cyriltovena Do we safely support this kind of operations on a local filesystem?
huikang
force-pushed
the
2103-local-docker-deploy
branch
from
5f51299
to
98754ba
Compare
production/docker/config/loki-docker-memberlist-boltdb-shipper.yaml
Outdated
Show resolved
Hide resolved
| chunk_idle_period: 30m | ||
| max_chunk_age: 2h |
There was a problem hiding this comment.
for consistency with what we just made the other defaults a few days ago, can you set both of these to 1h?
|
This is really neat @huikang I added a few comments for the loki config based on changes we just made earlier this week, this would put this config inline with those changes! |
- To help beginners learning how to launch a cluster of loki on a local machine - Use single loki config for both front-end and other nodes - Fix query lookback and add /ring to gateway Signed-off-by: Hui Kang <[email protected]>
huikang
force-pushed
the
2103-local-docker-deploy
branch
from
98754ba
to
2898ce7
Compare
@slim-bean Thanks for your comments. The PR is updated based on your suggestions. |
|
Loving this additional information to build a local clustered Loki. In order to get around some of the errors on startup:
I needed to use:
Also - after startup, I'm not able to get to port 80/3100 externally to the loki docker network. Running on CentOS 7-8.2003.0.el7.centos.x86_64 with Docker version 19.03.13, docker-compose version 1.26.2 I am seeing the logs getting collected by Promtail and can browse those through the Grafana instance. Will continue to diagnose my access issue. Also - any thoughts on why some of the Loki versions in the docker-compose are specific to 1.6 instead of :latest? Thanks again!! Love it!! |
I believe this version mismatch is a hindsight, because the PR was developed close to 2.x release. Would you mind to open a PR to fix this? |
|
To insert events out of order would be great and much wanted since we can't use Loki in our organization until that is resolved. |
loki on a local machine
Signed-off-by: Hui Kang [email protected]
What this PR does / why we need it:
Add a local deployment of loki cluster to ease the experiment of loki cluster.
Which issue(s) this PR fixes:
Fixes #2103
Special notes for your reviewer:
@periklis
Checklist