Distribution: 3.17 Tracking issue

[slimsag]

Status

This tracking issue is being prepared.

Workload

@unassigned

• Add an option to force plaintext (no syntax highlighting) for debugging #8433 3d

@bobheadxi: 19.00d

• Script to flag GCP and AWS resources that are unexpected / wasteful #10116 4d
• monitoring for CPU and memory usage in Kubernetes #9791 2d
• observability: Add monitoring to signal whether instance is underprovisioned or not #7529 4d
• Prevent admins from saving site config if it doesn't pass validation #10883 1d
• resources-report: add whitelist functionality for certain resources #11005 0.5d
• src config commands fail #10419 0.5d 🐛
• Make conf.DeployType() differentiate between k8s and docker compose #10399 0.5d 🐛
• observability: Alerting should be easily configured through a file or site configuration #10641 4d
• monitoring: container monitoring should exclude jaeger side-car containers #11270 0.5d 🐛
• Automate creation of the Go third-party licenses CSV #9973 2d
• Setup new Grafana alerting on k8s.sgdev.org and sourcegraph.com to go to Slack #7527 1d

@davejrt

• move away from Prometheus alertmanager on Sourcegraph.com specifically #5370

@daxmc99: 4.00d

• Warn users when Sourcegraph is out of date #9708 0.5d
• Cannot start on docker for windows (prometheus crashed) #6577
• Address golangci-lint complaints about monitoring/generator.go #10915 0.5d
• jaeger should be accessible via app proxy #10491 1d
• k8s: Deploy jaeger-agent as a Daemonset instead of sidecar container #10052 1d
• observability: server/dev should always run latest grafana/prometheus #9983 1d
• add terraform configuration for sourcegraph.com's infrastructure #10455 2d
• Add monitoring for syntax highlighting vs. gitserver perf #10049 0.5d
• observability: Add ping performance stats to frontend dashboard #9867 0.5d
• observability: Improve ping logging #9866 0.5d 🐛

@ggilmore: 3.50d

• Set ephemeral storage resource requests and limits in deploy-sourcegraph #9604 1d 👩
• sourcegraph/security-issues #68 0.5d
• RFC 141: spike: Implementation Round 2: Dhall #10936 3d 🕵️
• RFC 141: spike: Implementation Round 2: Cue #10937 3d 🕵️

@slimsag

• Deploy and release: Replace Renovate with GitHub Actions #10011 0.5d
• verify "syntax highlighting sometimes doesn't work" is fixed on customer instance #9557 1d 🐛
• release: Docker Compose in-band #10486
• Alert when repository updates are failing #9782 0.5d
• RFC 128: Investigate and propose a solution for how the Slack /release command could be implemented behind the scenes #9252 3d
• patch release process requires high context switching #7148
• observability: improve/restructure admin documentation #9773 0.5d

@uwedeportivo

• Move e2e tests into regression test suite #9814 2d
• deploy-sourcegraph: include k8s secret that's mounted into gitserver #9722 1d
• support $CUSTOMER #10942 8d

Legend

• 👩 Customer issue
• 🐛 Bug
• 🧶 Technical debt
• 🛠️ Roadmap
• 🕵️ Spike
• 🔒 Security issue
• Pull Request

[slimsag]

End of week update

This week we did the following:

• Uwe:
  • 🎉Released v3.16.0
  • 💪Working with Dave on $CUSTOMER GHE replica, great progress so far
  • 🏃Now focused on: $CUSTOMER replication
• Dave:
  • 💪Working with Uwe on $CUSTOMER GHE replica, great progress so far
  • 💬Researching on the side "switch away from site24x7 or find a way to make it more reliable Improve reliability of Sourcegraph.com site24x7 ping alerts #10742"
  • 🏃Now focused on: $CUSTOMER replication
• Robert:
  • 🎉Merged "siteAdmin: include alerts data in bug report page siteAdmin: include alerts data in bug report page #10704"
  • 💪Working on "distribution: resources report distribution: resources report #10784" and making excellent progress despite terrible GCP/AWS APIs
  • 💪Adding cadvisor (cpu/memory metrics) to kubernetes deployment add cadvisor to deployment deploy-sourcegraph#707
  • 💪Looking into https://github.com/sourcegraph/security-issues/issues/68
  • 🏃Now focused on: above
• Geoffrey:
  • 💬Giving ghe.sgdev.org guidance to Uwe/Dave
  • 💬Guiding Robert on https://github.com/sourcegraph/security-issues/issues/68
  • 🎉Merged PR to update-docker-tags so we can rollout RFC 141 prototype to k8s.sgdev.org
  • 💪Planned next steps for RFC 141: https://github.com/sourcegraph/sourcegraph/issues/10937 and https://github.com/sourcegraph/sourcegraph/issues/10936
  • 🏃Now focused on: Pulling trigger to deploy RFC 141 prototype on k8s.sgdev.org
• Dax:
  • 💪Continued work on "Warn users when Sourcegraph is out of date Warn users when Sourcegraph is out of date #9708" (turned out to be much more involved than Stephen initially thought, worked with Stephen, Joe, and others to find path forward)
  • 🎉Resolved "Cannot start on docker for windows (prometheus crashed) Cannot start on docker for windows (prometheus crashed) #6577"
  • 🏃Now focused on: Finishing up "Warn users when Sourcegraph is out of date Warn users when Sourcegraph is out of date #9708"
• Stephen:
  • 📆 We made a first draft of our planned work for 3.17 (to be finalized by EOD Wed).
  • 💬Planning $CUSTOMER deployment and replica
  • 💬Joining $CUSTOMER2 deployment call and providing scaling advice
  • 💬Interviewed ~3 candidates
  • 💪Reviewing ~25 PRs and helping with ongoing-work for Robert, Dax, Dave, Uwe, Geoffrey, Rijnard, Eric, Keegan, Quinn.
  • 🎉Released Docker Compose v3.16.0
  • 🏃Now focused on: Finalizing 3.17 planning, planning RFC 141 w/Geoffrey, giving direction to Robert & Dax

Legend:

• 💬 Discussion & advice
• 💪Progress
• 🎉Completed
• 🏃Working towards

cc @christinaforney @nicksnyder

[slimsag]

End of week update

This week we did the following:

• Uwe & Dave:
  • 🎉Identified issues in (1) indexing (2) repository status page (3) postgres resource advice (4) indexer (fixed by Tomas, thank you!); see bigdata and rfc151 issue label
  • 🎉New ghe-feeder tool and handbook entry
  • 💪$CUSTOMER GHE and Sourcegraph replicas set up. 178k of 210k repos cloned (1k+ orgs)
  • 💪Assisting $CUSTOMER in loading repos
  • 🏃Now focused on: above
• Robert:
  • 🎉 Resource reporting bot (see #distribution slack) distribution: resources report #10784
  • 🎉 Resource reporting handbook docs
  • 🎉Improvements to resource reporting (working on more)
  • 🎉Cadvisor to Kubernetes deployments (container CPU/memory metrics) add cadvisor to deployment deploy-sourcegraph#707
  • 💪Mostly finished https://github.com/sourcegraph/security-issues/issues/68
  • 🔎Updating dot-com deployment: https://github.com/sourcegraph/deploy-sourcegraph-dot-com/pull/2642
  • 🔎Adding over/under-provisioning alerts/monitoring to all deployments: https://github.com/sourcegraph/sourcegraph/pull/11082
  • 🔎Fixed src config: migrate Configuration usage to Settings src-cli#211
  • 🏃Now focused on: https://github.com/sourcegraph/sourcegraph/issues/10641 and https://github.com/sourcegraph/sourcegraph/issues/10399
• Geoffrey:
  • 🎉Set up automated builds and homebrew for update-docker-tags.
  • 🎉Added generation checks to k8s.sgdev.org CI pipeline
  • 🎉Deployed RFC 141 prototype to k8s.sgdev.org for dogfooding/comparison
  • 💪Began work on "spike: RFC 141 Implementation Round 2: Cue RFC 141: spike: Implementation Round 2: Cue #10937"
  • 🏃Now focused on: RFC 141
• Dax:
  • 🎉Exorcised golanci-lint errors Exorcise golanci-lint errors #10949
  • 🎉Made Jaeger easier for customers to access (no more port-forwarding) Jaeger reverse proxy working  #11002
  • 💬Investigated moving Jaeger to daemonset deployment model and discussed tradeoffs with team https://github.com/sourcegraph/sourcegraph/issues/10052
  • 🔎Waiting for Stephen's final review on "Warn users when Sourcegraph is out of date Warn users when Sourcegraph is out of date #9708"
  • 🏃Now focused on: server/dev should always run latest grafana/prometheus Prevent releasing server without latest grafana/prometheus changes #9983
• Stephen (not working Mon/Tue):
  • 🎉 Finalized our planned work for 3.17.
  • 🎉 Finalized RFC 141 planning w/Geoffrey
  • 🎉 Reviewed managed instances approach
  • 💪 Investigating https://github.com/sourcegraph/customer/issues/60
  • 💬 Discussed resource reporting goals with Robert
  • 💬 Discussed over-under-provisioning monitoring approach with Robert
  • 💬 Discussed Jaeger operators with Dax/Beyang
  • 💬 Answering resource estimation questions
  • 💪 Various Interviews, debriefs, syncs, and other long conversations.
  • 💪 Working with $CUSTOMER on resource advice & HA deployment model across multiple AZs
  • 💪 Assisting Uwe/Dave with $CUSTOMER when needed
  • 🏃Now focused on: Reviewing changes by Robert & Dax (Warn users when Sourcegraph is out of date #9708); planned work (sourcegraph-intellij changes for $CUSTOMER)

Legend:

• 💬Discussion & advice
• 💪Progress
• 🎉Completed
• 🏃Working towards
• 🔎In review

cc @christinaforney @nicksnyder

[slimsag]

https://github.com/sourcegraph/sourcegraph/issues/11122

[slimsag]

(deferred) End of week update

Last week we did the following:

• Uwe & Dave:
  • 🎉$CUSTOMER Sourcegraph replica has ~400k repos cloned/indexed.
  • 💪Assisting $CUSTOMER in loading repos, minikube issues, collecting feedback and filing issues (timeouts, global config issues, feature requests, etc.)
  • 💪 Uwe: Investigated Zoekt issues on replica with Stephen, networking issues with Dave and Dax.
  • 💪Uwe: Providing support to multiple other customers.
  • 💪Dave: Looking at stretch issues, supporting Robert with dot-com.
  • 🏃Now focused on: above
• Robert:
  • 🎉Concluded https://github.com/sourcegraph/security-issues/issues/68
  • 🎉Landed dot-com deployment update https://github.com/sourcegraph/deploy-sourcegraph-dot-com/pull/2642
  • 🎉Updated documentation for updating dot-com deployments: update instructions for merging deploy-sourcegraph about#1020
  • 🎉Landed resource allow-listing https://github.com/sourcegraph/sourcegraph/pull/11107 so CI noise is removed
  • 🎉Tweaked cadvisor to prevent Prometheus OOMs in k8s
  • 🎉Landed fixes for src config: migrate Configuration usage to Settings src-cli#211
  • 💪Planned solution and started "observability: Alerting should be easily configured through a file or site configuration observability: Alerting should be easily configured through a file or site configuration #10641"
  • 🔎Sent PR "resources-report: write each report to new page resources-report: write each report to new page #11316"
  • 🔎Sent PR for unifying license generation: https://github.com/sourcegraph/sourcegraph/pull/11321
  • 💬Discussing approach to "Add over/under-provisioning alerts/monitoring to all deployments: https://github.com/sourcegraph/sourcegraph/pull/11082"
  • 🏃Now focused on: https://github.com/sourcegraph/sourcegraph/issues/10641 and https://github.com/sourcegraph/sourcegraph/issues/10399
• Geoffrey:
  • 🎉Concluded "spike: RFC 141 Implementation Round 2: Cue RFC 141: spike: Implementation Round 2: Cue #10937"
  • 🎉Concluded https://github.com/sourcegraph/security-issues/issues/68
  • 💪Began work on customer-outage issue ( https://github.com/sourcegraph/sourcegraph/issues/9604 )
  • 💪Began work on "spike: RFC 141 Implementation Round 2: Dhall reduce k8s upgrade merge conflicts: RFC 141: Dhall investigation #10936"
  • 🏃Now focused on: RFC 141
• Dax:
  • 🎉Concluded investigation into moving Jaeger to daemonset deployment model https://github.com/sourcegraph/sourcegraph/issues/10052
  • 🎉Landed final configuration changes for the jaeger reverse-proxy in k8s/docker-compose
  • 💪Started work on "add terraform configuration for sourcegraph.com's infrastructure add terraform configuration for sourcegraph.com's infrastructure  #10455"
  • 💪Looking into "observability: server/dev should always run latest grafana/prometheus Prevent releasing server without latest grafana/prometheus changes #9983" and supporting tooling Add new regex for matching with all docker tags slimsag/update-docker-tags#4 (trickier than expected)
  • 🏃Now focused on: above; reviewing Stephen's changes to Warn users when Sourcegraph is out of date #9708
• Stephen (not working Mon/Tue):
  • 🎉Concluded investigation of https://github.com/sourcegraph/customer/issues/60
  • 🎉$CUSTOMER HA deployment set up successfully.
  • 🎉Merged "unify update site alerts + remove animated background from alerts unify update site alerts + remove animated background from alerts #11326"
  • 💬Many Interviews/debriefs/syncs/discussions (cloud security, $CUSTOMER deployment with Uwe/Dave/Dan, etc.) and code reviews.
  • 💪Extensive review & changes to "Warn users when Sourcegraph is out of date Warn users when Sourcegraph is out of date #9708"
  • 💪Started the tedious process of writing a license report for syntect_server: https://github.com/sourcegraph/sourcegraph/issues/11269
  • 💪Started work on broader distribution planning / prioritization for giving context to Gonzalo
  • 🏃Now focused on: Reviewing changes by Robert & Dax; sourcegraph-intellij changes for $CUSTOMER, planned work

Legend:

• 💬Discussion & advice
• 💪Progress
• 🎉Completed
• 🏃Working towards
• 🔎In review

cc @christinaforney @nicksnyder

[uwedeportivo]

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.17 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly.
When in doubt, reach out!

Thank you

[bobheadxi]

Robert update, week of jun 18

• 🎉 Landed automated Grafana notifier deployment (https://github.com/sourcegraph/sourcegraph/pull/11483)
• 🎉 Landed provisioning monitoring (https://github.com/sourcegraph/sourcegraph/pull/11082)
• 🎉 Landed new licenses workflow (https://github.com/sourcegraph/sourcegraph/pull/11321, add third-party license documentation about#1067)
• 🎉 Misc fixes on dev tooling (https://github.com/sourcegraph/sourcegraph/pull/11486, https://github.com/sourcegraph/sourcegraph/pull/11540)
• 💬 Planned some follow-up improvements for Grafana alerts
• 🔍 PR: Fixes for provisioning monitoring (https://github.com/sourcegraph/sourcegraph/pull/11542)
• 🔍 PR: Support for Grafana SMTP configuration and email notifiers (https://github.com/sourcegraph/sourcegraph/pull/11554)
• 🏃 Now focused on above PRs, Grafana/alerting improvements (including Grafana v7: https://github.com/sourcegraph/sourcegraph/pull/11583)

[ggilmore]

Geoffrey update:

• Completed PoC of https://github.com/sourcegraph/sourcegraph/issues/10936, we saw enough to continue investigating it
• Moved Dhall RFC 141 PoC into https://github.com/sourcegraph/deploy-sourcegraph-dhall
• Added more services to https://github.com/sourcegraph/deploy-sourcegraph-dhall to get a better idea of what the initial performance will look like
• Will continue to add services and ask the Dhall community for advice for improving performance
• Next milestone - replace subset of services on k8s.sgdev.org with Dhall customization pipeline

[daxmc99]

Dax update:

• Completed https://github.com/sourcegraph/sourcegraph/issues/10049, we can now observe and determine the time to highlight vs fetch a file.
• Completed https://github.com/sourcegraph/sourcegraph/issues/9866, add a prefix to ping logs to make the debugging experience cleaner
• Finished https://github.com/sourcegraph/sourcegraph/issues/9867, I added a dashboard and clarified the metrics being sent back to Sourcegraph. I left the issue open though because I believe these metrics should be removed after they are used to determine which methods take the most time. These metrics have a high level of cardinality.
• Fixed https://github.com/sourcegraph/sourcegraph/pull/11536, docker build should work on Mac again 😄
• Terraform infrastructure for Sourcegraph.com https://github.com/sourcegraph/sourcegraph/issues/10455. The code is ready to merge but we still need to coordinate with the cloud team to perform the migration to ensure the Sourcegraph.com infrastructure is captured in the state file.
• Next steps: Focusing on Sourcegraph.com cloud HA Postgres as part of https://github.com/sourcegraph/sourcegraph/issues/11496