logs: bump jsonnet dep to start using ConfigMap and Services from loki.mixins #450

JoaoBraveCoding · 2023-04-06T11:33:07Z

See:

Refactors loki.libsonnet to use loki mixins to generate services observatorium/observatorium#508
logs: refactors ConfigMap generation to use loki.mixins observatorium/observatorium#511

JoaoBraveCoding · 2023-04-06T12:34:06Z

periklis

Looks good to me just one entry in the config that should stay as is namely the index_queries_cache and a couple of nits.

periklis · 2023-04-11T07:38:42Z

resources/services/observatorium-logs-template.yaml

@@ -750,7 +737,6 @@ objects:
      "limits_config":
        "cardinality_limit": 100000
        "creation_grace_period": "10m"
-        "deletion_mode": "disabled"


Although disabled is the default, I suggest to keep this here because the Deletion API has caused numerous issues upstream.

Re-added and left a not to move this to observatorium/observatorium since I think it makes more sense to live there

periklis · 2023-04-11T07:39:19Z

resources/services/observatorium-logs-template.yaml

-        "index_queries_cache_config":
-          "memcached":
-            "batch_size": 100
-            "parallelism": 100
-          "memcached_client":
-            "addresses": "dns+observatorium-loki-index-query-cache.${NAMESPACE}.svc.cluster.local:11211"
-            "consistent_hash": true


This should stay here.

Re-added this, only defaults were removed

services/observatorium-logs-template-overwrites.libsonnet

JoaoBraveCoding · 2023-04-13T17:36:04Z

Needs observatorium/observatorium#520

periklis

LGTM let's ensure we have a clear path to overcome the two rollout issues we found on rhobs-testing and this is good to go:

How to rollout the serviceName change on StatefulSets on app-interface?
How to keep the old http service referenced in observatorium-template.yaml in tact to minimize rollout dependencies.

JoaoBraveCoding · 2023-04-17T10:15:26Z

How to rollout the serviceName change on StatefulSets on app-interface?

We should be able to leverage app-sre/qontract-reconcile#1627

How to keep the old http service referenced in observatorium-template.yaml intact to minimize rollout dependencies.

To solve this from my POV we would:

Include in this PR the old services referenced in observatorium-template.yaml
Deploy
Create a new PR where we update the services in observatorium-template.yaml
Deploy
Create a new PR where we remove the old services
Deploy

loki.mixins Signed-off-by: JoaoBraveCoding <[email protected]>

Signed-off-by: JoaoBraveCoding <[email protected]>

periklis · 2023-04-17T10:48:52Z

How to rollout the serviceName change on StatefulSets on app-interface?

We should be able to leverage app-sre/qontract-reconcile#1627

ACK

How to keep the old http service referenced in observatorium-template.yaml intact to minimize rollout dependencies.

To solve this from my POV we would:
1. Include in this PR the old services referenced in `observatorium-template.yaml`

2. Deploy

3. Create a new PR where we update the services in `observatorium-template.yaml`

4. Deploy

5. Create a new PR where we remove the old services

6. Deploy

ACK

JoaoBraveCoding · 2023-04-17T11:00:30Z

Once we have observatorium/observatorium#521 merged we will be able to progress on this

deployment. First we will deploy the new services while keeping the old Then we will update observatorium api to change to the new services Finally we will then remove the old services Signed-off-by: JoaoBraveCoding <[email protected]>

Relates with #450 Signed-off-by: Joao Marcal <[email protected]>

relates to #450 (comment) Signed-off-by: Joao Marcal <[email protected]>

JoaoBraveCoding force-pushed the bump-loki-logs branch from 048f734 to 8296081 Compare April 6, 2023 12:22

periklis requested changes Apr 11, 2023

View reviewed changes

JoaoBraveCoding force-pushed the bump-loki-logs branch 4 times, most recently from 6114895 to 8d91b31 Compare April 11, 2023 09:48

JoaoBraveCoding requested a review from periklis April 11, 2023 10:09

JoaoBraveCoding force-pushed the bump-loki-logs branch 3 times, most recently from c6587ad to 67abc19 Compare April 13, 2023 17:35

JoaoBraveCoding force-pushed the bump-loki-logs branch 3 times, most recently from 81779f6 to 3c6a9aa Compare April 14, 2023 08:34

periklis reviewed Apr 16, 2023

View reviewed changes

JoaoBraveCoding added 2 commits April 17, 2023 11:16

logs: bump jsonnet dep to start using ConfigMap and Services from

0474b7a

loki.mixins Signed-off-by: JoaoBraveCoding <[email protected]>

logs: moved loki CLI args to config

68b0975

Signed-off-by: JoaoBraveCoding <[email protected]>

JoaoBraveCoding mentioned this pull request Apr 17, 2023

Re-introduce http services to minimize downtime when updating to new services observatorium/observatorium#521

Merged

Re-introduced old http service to allow for phased

044afd6

deployment. First we will deploy the new services while keeping the old Then we will update observatorium api to change to the new services Finally we will then remove the old services Signed-off-by: JoaoBraveCoding <[email protected]>

JoaoBraveCoding force-pushed the bump-loki-logs branch from 3c6a9aa to 044afd6 Compare April 17, 2023 14:05

periklis approved these changes Apr 17, 2023

View reviewed changes

periklis merged commit 8d6ebd7 into rhobs:main Apr 17, 2023

JoaoBraveCoding mentioned this pull request Apr 18, 2023

logs: fixes grafana dashboards to start referencing the new services #460

Merged

JoaoBraveCoding added a commit that referenced this pull request Apr 18, 2023

logs: fixes grafana dashboards to start referencing the new services

4c59031

Relates with #450 Signed-off-by: Joao Marcal <[email protected]>

JoaoBraveCoding added a commit that referenced this pull request Apr 19, 2023

logs: moves observatorium api to start using the new services

0978223

relates to #450 (comment) Signed-off-by: Joao Marcal <[email protected]>

JoaoBraveCoding added a commit that referenced this pull request Apr 19, 2023

logs: moves observatorium api to start using the new services

80e9e28

relates to #450 (comment) Signed-off-by: Joao Marcal <[email protected]>

This was referenced Apr 19, 2023

logs: moves observatorium api to start using the new services #461

Draft

logs: removes the old logs svcs and moves obs api to new svcs observatorium/observatorium#524

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

logs: bump jsonnet dep to start using ConfigMap and Services from loki.mixins #450

logs: bump jsonnet dep to start using ConfigMap and Services from loki.mixins #450

JoaoBraveCoding commented Apr 6, 2023

JoaoBraveCoding commented Apr 6, 2023

periklis left a comment

periklis Apr 11, 2023

JoaoBraveCoding Apr 11, 2023

periklis Apr 11, 2023

JoaoBraveCoding Apr 11, 2023

JoaoBraveCoding commented Apr 13, 2023

periklis left a comment

JoaoBraveCoding commented Apr 17, 2023 •

edited

Loading

periklis commented Apr 17, 2023

JoaoBraveCoding commented Apr 17, 2023

logs: bump jsonnet dep to start using ConfigMap and Services from loki.mixins #450

logs: bump jsonnet dep to start using ConfigMap and Services from loki.mixins #450

Conversation

JoaoBraveCoding commented Apr 6, 2023

JoaoBraveCoding commented Apr 6, 2023

periklis left a comment

Choose a reason for hiding this comment

periklis Apr 11, 2023

Choose a reason for hiding this comment

JoaoBraveCoding Apr 11, 2023

Choose a reason for hiding this comment

periklis Apr 11, 2023

Choose a reason for hiding this comment

JoaoBraveCoding Apr 11, 2023

Choose a reason for hiding this comment

JoaoBraveCoding commented Apr 13, 2023

periklis left a comment

Choose a reason for hiding this comment

JoaoBraveCoding commented Apr 17, 2023 • edited Loading

periklis commented Apr 17, 2023

JoaoBraveCoding commented Apr 17, 2023

JoaoBraveCoding commented Apr 17, 2023 •

edited

Loading