-
Notifications
You must be signed in to change notification settings - Fork 205
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feed elasticsearch cluster #10193
feed elasticsearch cluster #10193
Conversation
This PR adds a machine that will, every 5 minutes, look at the GCS bucket that stores Bazel metrics and push whatever it finds to ElasticSearch. A huge part of this commit is based on @aherrmann-da's work. You can assume that all the good bits are his. CHANGELOG_BEGIN CHANGELOG_END
Because there is no great way to develop & test this kind of work without actually deploying all of it, it's all already deployed and currently running. We have about 3k jobs to ingest and it takes a few seconds per job, so hopefully by tomorrow morning we should have most of the data loaded. 🤞 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks sensible, I admit I can’t claim to fully grasp the details and some of this seems like a "if it works it works type of thing” but don’t see anything off. Thank you!
scopes = [ | ||
# Required for cloud logging | ||
"cloud-platform", | ||
"logging-write", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is that less or more permissions or just different and the old one didn’t work?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's less, but it's also unchanged. GCP has two different permission systems for instances: at one level, you specify "scopes", which is what we're doing here. This is the "old", deprecated-but-you-still-have-to-use-it way. The official recommendation (from GCP) here is to set it to "cloud-platform", which means "access to everything".
The "current" way of specifying permissions is via the service account associated with the instance.
In order for an action to succeed it needs to be permitted by both the scopes and the service account (which makes debugging permission issues a bit painful at times). So this is restricting the scope, but we were already (and still are) fully restricted to the absolute minimum set of rights by the service account, which is why "less, but actually the same".
This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @Robin-da is in charge of this release. Commit log: ``` 0d881f5 Improvements to the documentation with regards to offsets (#10180) 1d5ba4f feed elasticsearch cluster (#10193) e2bdca6 Use PartialFunction for more concise code. (#10191) fe98c48 release 1.14.1-snapshot.20210706.7133.0.8ec16e94 (#10186) 4fe626a Drop logging on CompletionSource (#10190) 582aa5f Fix a typo in an exceptions example. (#10188) 8750c0c reduce noise in Scala bindings (#10187) 98b5ffe Add divulgence warning and test in script service. (#10179) 8578e56 Tests for transaction validation (#10167) 05a72a3 update compat versions for 1.15.0-snapshot.20210705.7286.0.62aabcc4 (#10184) 7b93923 [In-memory fan-out] Performance optimizations [DPP-470] (#10127) 6c49619 Document how to run the Java codegen from the Daml assistant (#10181) 61aa774 Release SDK 1.15 RC (#10182) ``` Changelog: ``` [Docs] Improvements to the documentation with regards to offsets [Docs] Document how to use the Java codegen using the Daml assistant ``` CHANGELOG_BEGIN CHANGELOG_END
This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @akshayshirahatti-da is in charge of this release. Commit log: ``` 1f35db1 ledger-on-sql: Use random log entry ID allocation so we don't depend on `SeedService`. [KVL-1002] (#10255) b8e2198 Separate traces from warnings in engine. (#10253) 0a7f2b1 [JSON-API] Small refactoring to start using akka http routes internally (#10252) 6e8ec1d LF: Drop old depreated code (#10251) de7a08f [In-memory fan-out] Handle `null` submitters in getTransactionLogUpdates (#10248) 584169a Participant state v1-to-v2 adaptors [KVL-1002] (#10246) e4d7cd7 LF: Deprecate com.daml.lf.data.TryOps.Bracket (#10249) b59f365 [JSON-API] Correctly extract the request source URL/IP (#10244) edaf541 Adapt `ledger-api-bench-tool` to work with LR [DPP-424] (#10171) 6fe6ae0 LF: Make DarReader ZipEntries immulatble (#10243) 05e5218 Consolidating EventStorageBackend (#10224) 3fd3abf Instrument buffer after contract state events stream (#10209) 643a2de v2 participant state API draft [KVL-998] (#10210) 5abcd04 Adds Unit tests for parallel-ingestion [DPP-455] (#10238) 3cdedcf kvutils: Extract validators from TransactionCommitter [KVL-1015] (#10235) 00d622f Make @UNTIL-LF exclusive, add @UNTIL-LF-FEATURE. (#10240) 2bcbd4e es: switch to persistent nodes (#10236) 7f4bc2a update compat versions for 1.14.2 (#10233) 4eba4b0 Ledger: decouple BD value serialization library from archive library. (#10218) 729afa8 Write a few lines on how to run tests against an Oracle database (#10230) 2b67ebb tf: refactor appr var (#10232) 202b7f7 add akshay to appr team (#10229) 2c5410c Release SDK 1.14.2 (#10226) 89d87ad update compat versions for 1.14.2-snapshot.20210708.7135.0.aa297840 (#10223) 4839344 TODO Cleanup: QueryNonPruned [DPP-459] (#10225) 0bea5e3 Allow retriable lookupMaximumLedgerTime on contention (#10211) ca2a9e9 Release 1.14.2 snapshot (#10222) 999577a tweak ES cluster (#10219) 6680d36 Warn on DA.Exception import (#10201) f19f5b0 LF: Simplify DarReader (#10217) 8f1a8f2 Release DAML SDK 1.14.1 (#10204) ebb76dc LF: reorganize errors in com.daml.lf.archive (#10213) 6964645 Improvements for validation tests (#10214) 5f124c3 Avoid collision in execution log postfix (#10205) 9e27ae0 Reshuffle cocreature in daml-lf codeowners (#10215) 41b8448 LF: Simplify archive reader. (#10208) 38734f0 es-feed: ignore invalid files (#10207) 6586dde Move exercise context computation to the client side (#10199) c92c678 Mark Daml profiler as stable (#10203) cb1f4ec ci/windows: disable spool (#10200) 2326d42 Publish execution logs on unix platforms (#10194) 4269345 Drop cleanup of node_modules (#10195) c8faed8 Deduplicate Java codegen running instructions (#10185) e4c8c39 update compat versions for 1.14.1-snapshot.20210706.7133.0.8ec16e94 (#10196) 0d881f5 Improvements to the documentation with regards to offsets (#10180) 1d5ba4f feed elasticsearch cluster (#10193) e2bdca6 Use PartialFunction for more concise code. (#10191) fe98c48 release 1.14.1-snapshot.20210706.7133.0.8ec16e94 (#10186) 4fe626a Drop logging on CompletionSource (#10190) 582aa5f Fix a typo in an exceptions example. (#10188) 8750c0c reduce noise in Scala bindings (#10187) 98b5ffe Add divulgence warning and test in script service. (#10179) 8578e56 Tests for transaction validation (#10167) 05a72a3 update compat versions for 1.15.0-snapshot.20210705.7286.0.62aabcc4 (#10184) 7b93923 [In-memory fan-out] Performance optimizations [DPP-470] (#10127) 6c49619 Document how to run the Java codegen from the Daml assistant (#10181) 61aa774 Release SDK 1.15 RC (#10182) ``` Changelog: ``` - [JSON-API] If the service is put behind a proxy filling either of these headers X-Forwarded-For & X-Real-Ip then these will now be respected for logging the request source ip/url - [Daml Compiler] Importing DA.Exception on LF < 1.14 now produces a warning that exceptions require Daml-LF >= 1.14. - [Daml Profiler] The Daml profiler is now a stable feature. [Docs] Improvements to the documentation with regards to offsets [Docs] Document how to use the Java codegen using the Daml assistant ``` CHANGELOG_BEGIN CHANGELOG_END
* release 1.15.0-snapshot.20210713.7343.0.1f35db17 This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @akshayshirahatti-da is in charge of this release. Commit log: ``` 1f35db1 ledger-on-sql: Use random log entry ID allocation so we don't depend on `SeedService`. [KVL-1002] (#10255) b8e2198 Separate traces from warnings in engine. (#10253) 0a7f2b1 [JSON-API] Small refactoring to start using akka http routes internally (#10252) 6e8ec1d LF: Drop old depreated code (#10251) de7a08f [In-memory fan-out] Handle `null` submitters in getTransactionLogUpdates (#10248) 584169a Participant state v1-to-v2 adaptors [KVL-1002] (#10246) e4d7cd7 LF: Deprecate com.daml.lf.data.TryOps.Bracket (#10249) b59f365 [JSON-API] Correctly extract the request source URL/IP (#10244) edaf541 Adapt `ledger-api-bench-tool` to work with LR [DPP-424] (#10171) 6fe6ae0 LF: Make DarReader ZipEntries immulatble (#10243) 05e5218 Consolidating EventStorageBackend (#10224) 3fd3abf Instrument buffer after contract state events stream (#10209) 643a2de v2 participant state API draft [KVL-998] (#10210) 5abcd04 Adds Unit tests for parallel-ingestion [DPP-455] (#10238) 3cdedcf kvutils: Extract validators from TransactionCommitter [KVL-1015] (#10235) 00d622f Make @UNTIL-LF exclusive, add @UNTIL-LF-FEATURE. (#10240) 2bcbd4e es: switch to persistent nodes (#10236) 7f4bc2a update compat versions for 1.14.2 (#10233) 4eba4b0 Ledger: decouple BD value serialization library from archive library. (#10218) 729afa8 Write a few lines on how to run tests against an Oracle database (#10230) 2b67ebb tf: refactor appr var (#10232) 202b7f7 add akshay to appr team (#10229) 2c5410c Release SDK 1.14.2 (#10226) 89d87ad update compat versions for 1.14.2-snapshot.20210708.7135.0.aa297840 (#10223) 4839344 TODO Cleanup: QueryNonPruned [DPP-459] (#10225) 0bea5e3 Allow retriable lookupMaximumLedgerTime on contention (#10211) ca2a9e9 Release 1.14.2 snapshot (#10222) 999577a tweak ES cluster (#10219) 6680d36 Warn on DA.Exception import (#10201) f19f5b0 LF: Simplify DarReader (#10217) 8f1a8f2 Release DAML SDK 1.14.1 (#10204) ebb76dc LF: reorganize errors in com.daml.lf.archive (#10213) 6964645 Improvements for validation tests (#10214) 5f124c3 Avoid collision in execution log postfix (#10205) 9e27ae0 Reshuffle cocreature in daml-lf codeowners (#10215) 41b8448 LF: Simplify archive reader. (#10208) 38734f0 es-feed: ignore invalid files (#10207) 6586dde Move exercise context computation to the client side (#10199) c92c678 Mark Daml profiler as stable (#10203) cb1f4ec ci/windows: disable spool (#10200) 2326d42 Publish execution logs on unix platforms (#10194) 4269345 Drop cleanup of node_modules (#10195) c8faed8 Deduplicate Java codegen running instructions (#10185) e4c8c39 update compat versions for 1.14.1-snapshot.20210706.7133.0.8ec16e94 (#10196) 0d881f5 Improvements to the documentation with regards to offsets (#10180) 1d5ba4f feed elasticsearch cluster (#10193) e2bdca6 Use PartialFunction for more concise code. (#10191) fe98c48 release 1.14.1-snapshot.20210706.7133.0.8ec16e94 (#10186) 4fe626a Drop logging on CompletionSource (#10190) 582aa5f Fix a typo in an exceptions example. (#10188) 8750c0c reduce noise in Scala bindings (#10187) 98b5ffe Add divulgence warning and test in script service. (#10179) 8578e56 Tests for transaction validation (#10167) 05a72a3 update compat versions for 1.15.0-snapshot.20210705.7286.0.62aabcc4 (#10184) 7b93923 [In-memory fan-out] Performance optimizations [DPP-470] (#10127) 6c49619 Document how to run the Java codegen from the Daml assistant (#10181) 61aa774 Release SDK 1.15 RC (#10182) ``` Changelog: ``` - [JSON-API] If the service is put behind a proxy filling either of these headers X-Forwarded-For & X-Real-Ip then these will now be respected for logging the request source ip/url - [Daml Compiler] Importing DA.Exception on LF < 1.14 now produces a warning that exceptions require Daml-LF >= 1.14. - [Daml Profiler] The Daml profiler is now a stable feature. [Docs] Improvements to the documentation with regards to offsets [Docs] Document how to use the Java codegen using the Daml assistant ``` CHANGELOG_BEGIN CHANGELOG_END * 1.16 not 1.15 changelog_begin changelog_end Co-authored-by: Azure Pipelines DAML Build <[email protected]> Co-authored-by: Moritz Kiefer <[email protected]>
This PR adds a machine that will, every 5 minutes, look at the GCS
bucket that stores Bazel metrics and push whatever it finds to
ElasticSearch.
A huge part of this commit is based on @aherrmann-da's work. You can
assume that all the good bits are his.
CHANGELOG_BEGIN
CHANGELOG_END