Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

oracle debug #11386

Merged
merged 1 commit into from
Nov 1, 2021
Merged

oracle debug #11386

merged 1 commit into from
Nov 1, 2021

Conversation

garyverhaegen-da
Copy link
Contributor

Collect Oracle logs for further debugging.

CHANGELOG_BEGIN
CHANGELOG_END

cocreature
cocreature reviewed Oct 25, 2021
View reviewed changes
ci/build.yml Outdated
@@ -315,6 +315,8 @@ jobs:
//triggers/service:test-oracle \
//ledger/participant-integration-api:participant-integration-api-tests-oracle \
//ledger/ledger-on-sql:conformance-test-oracle
mkdir $(Build.StagingDirectory)/oracle-debug
docker exec oracle bash -c '/opt/oracle/diag/rdbms/.../*.trc' > $(Build.StagingDirectory)/oracle-debug/logs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn’t work does it? If bash runs in the container you cannot access Build.StagingDirectory. You need to copy from inside the container to outside the container.

Copy link
Contributor

@S11001001 S11001001 Oct 25, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is ... supported in this context? There's no one path, there are identifiers included in the paths, and more than one file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn’t work does it? If bash runs in the container you cannot access Build.StagingDirectory. You need to copy from inside the container to outside the container.

This was not meant to work per se, as I was waiting for the "real" path from Stephen. I did not know there were multiple paths. Note that the redirection itself should work, as it's happening outside of the container. (The in-container Bash command is missing a cat, though.)

@S11001001 S11001001 mentioned this pull request Oct 25, 2021
Merged
7 tasks
@garyverhaegen-da
Copy link
Contributor Author

@S11001001 on #11391 seems to say this is not going to be useful (emphasis mine):

This should also hide the cause from #11386, preventing us from determining the exact deadlock that occurred.

so I'm closing for now. @S11001001 please reopen if I misinterpreted that statement, and I'll make this work for saving the entire /opt/oracle/diag/rdbms/ folder.

@garyverhaegen-da garyverhaegen-da deleted the oracle-debug branch October 26, 2021 11:20
@S11001001 S11001001 restored the oracle-debug branch October 26, 2021 15:17
@S11001001
Copy link
Contributor

@garyverhaegen-da

However, [the deadlock] will still manifest, yielding an error and hopefully useful logs, should it be serious enough as to exhaust all retries.

In other words, it's far more improbable than it was before, and we'll miss it should it happen again.

It's generally useful for diagnosing any Oracle problem. For example, when we were getting segfaults with search indices, the trc files were the only place to learn what was happening.

(I don't know how big the rest of the tree is; the only important information AFAICT is in the trc files.)

@S11001001 S11001001 reopened this Oct 26, 2021
@garyverhaegen-da garyverhaegen-da force-pushed the oracle-debug branch 2 times, most recently from ef4849e to 95f5976 Compare October 28, 2021 16:11
@garyverhaegen-da
Copy link
Contributor Author

@S11001001 Is this useful? Would you prefer a single tarball?

S11001001
S11001001 approved these changes Oct 29, 2021
View reviewed changes
Copy link
Contributor

@S11001001 S11001001 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@garyverhaegen-da The logs look good in this form; thanks.

ci/build.yml Outdated
@@ -331,6 +339,13 @@ jobs:
pathtoPublish: 'bazel-testlogs/'
artifactName: 'Test logs Oracle'

- task: PublishBuildArtifacts@1
#condition: failed()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uncomment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, thanks for flagging! I had to do that for the logs to be saved as an example, and couldn't switch it back until you'd taken a look, as Azure would then delete the old build.

Which means I'd totally forgotten about it by now.

stefanobaghino-da
stefanobaghino-da reviewed Nov 1, 2021
View reviewed changes
Copy link
Contributor

@stefanobaghino-da stefanobaghino-da left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worked in "Big Data" on amounts of data that are probably smaller than the logging we produce at every CI run. ❤️ It's great to make it easier for us to inspect errors on CI, thanks for working on this. 🙇🏻

@garyverhaegen-da
oracle debug
0552210 
Collect Oracle logs for further debugging.

CHANGELOG_BEGIN
CHANGELOG_END
@garyverhaegen-da garyverhaegen-da merged commit 79505b5 into main Nov 1, 2021
@garyverhaegen-da garyverhaegen-da deleted the oracle-debug branch November 1, 2021 11:19
azure-pipelines bot pushed a commit that referenced this pull request Nov 3, 2021
release 1.18.0-snapshot.20211102.8257.0.7391a3cd
1c21d8d 
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.

@Robin-da is in charge of this release.

Commit log:
```
7391a3c Migrate language-support codegen tests to sandbox (#11508)
da76e28 iface: check for interface implementations precond (#11494)
9e045c1 Remove a leftover from `check-protobuf-stability.sh` (#11513)
17c2fa9 ErrorCodeDocumentationGenerator uses multiple prefixes for scanning classpaths (#11504)
a8ae4d6 LF: Rename GenNode to Node (#11469)
73b6f8b Fixed ledger-api-bench-tool random distribution in submission phase. (#11507)
92495b0 Upgrade nixpkgs (#11490)
48afb7b interface: Add inherited choice names in scala AST (#11503)
9f8b039 [Trigger-Service] Migrate tests to sandbox (#11501)
6bafeb0 DPP-662 Support archivals in bench tool (#11474)
63d855f Pass `damlc build` logs through daml logger (#11485)
565dfac Bump rules_nodejs to 4.4.2 (#11495)
9214956 Enable experimental nix command (#11498)
f81d880 LF: drop com.daml.lf.transaction.Transaction.Transaction type alias. (#11470)
505a7f5 Extend debug logging around dependencies/data-dependencies (#11488)
55d35e5 Drop TLA+ from dev-env (#11492)
3a5cb18 [Self-service error codes] Adapt all grpc assertions to additionally take self-service error codes [DPP-680] (#11437)
98e7461 LF: ContractInstance should be a CidContainer (#11487)
9afb3b3 Add and wire ledger-end-cache [DPP-703] (#11480)
cdd2acc Explicitly import Java runtime from Nix/dadew (#11411)
b5540a9 Evidence for contract-key testing (#11479)
78735a8 Less logging on //docs:generate-error-codes-json (#11486)
d8898b8 Check fixed choices in name collision checker (#11481)
7a0198b daml-react: allow for React 17 (#11463)
c415d9d Migrate trigger tests from sandbox-classic to sandbox (#11478)
f9ec12b Add cocreature as a codeowner of the docs (#11450)
a79cc08 DbDispatcher handles SQL exceptions with self-service error codes [DPP-690] (#11435)
164cdbb update compat versions for 1.18.0-snapshot.20211026.8179.0.e474b2d1 (#11426)
741a6e7 interfaces: Add list of fixed choices in TemplateImplements structure. (#11364)
79505b5 oracle debug (#11386)
257d536 Switch from sandbox to sandbox-classic in daml test-script (#11476)
e91ced5 KV: generate self-service error docs [KVL-1148] (#11431)
8ffd835 Migrate daml script tests to Sandbox next (#11472)
df65d02 Exclude dedup tests affected by current changes (#11471)
45beaf7 interfaces: typecheck for precond, haskell side. (#11468)
e445fe1 LfEngineToApi conversions do not have to deal with version. (#11414)
d371274 Sandbox-classic: contracts sent to the engine do not need enrichment (#11416)
05312f1 Lift out constraint tuples to synonyms in dfun contexts (#11467)
a492a6c [DPP-636] Update docs on MacOS firewall issue when running tests (#11460)
5a85881 rotate release duty after 1.18.0-snapshot.20211026.8179.0.e474b2d1 (#11422)
1096f91 Adds default keepalive support for postgreSQL [DPP-697] (#11453)
3799963 Fix akka-streams shutdown at LedgerConfigurationSubscriptionFromIndex (#11462)
e89da7c Move non-repudiation proxy conformance tests to their own package (#11464)
b98a3ad interfaces: precondition, scala ast, decoder/encoder (#11452)
fd973e6 Drop assistant integration tests for relative/absolute DAML_PROJECT (#11461)
05505e3 Multiplex multiple virtual query streams through a single web socket (#10221)
3098b70 pretty print contract IDs properly (#11359)
bb4f4c5 DPP-665 Make in-flight commands configurable (#11456)
bf00956 replace OneAnd party Sets with NonEmpty Set (#11420)
570160b Add LoggingValue instance for Time.Timestamp (#11444)
bad50a5 Type reexports explicitly use 'type' namespace (#11451)
d3b9c84 ifaces: preconditions, proto changes (#11448)
73718a2 Mark script export tests as flaky (#11449)
22e9e6b Multi-observer command submission in the ledger-api-bench-tool [DPP-660] (#11413)
3eacbf9 Bazel cache retry on connection reset by peer (#11445)
fcc7e58 Make stable packages dependent on supported LF versions (#11382)
6c45f09 Remove TODO (#11446)
d8b92b7 Drop flaky marker from lsp-tests (#11442)
68f4432 Improve evidence of testing (#11428)
6126fc2 Remove append only prefix from conformance tests [KVL-1152] (#11424)
926655d Drop daml ledger timeout tests from compat tests (#11441)
15cb840 Switch back to binary execution logs (#11440)
ac28e61 Update ScalaPB to the latest version (0.11.6) (#11409)
a71a8f9 Prevent overflow in DeduplicationPeriod (#11432)
6c88e55 Pass submission id as correlation id to error codes in Ledger API (#11381)
ba10687 Replace 1.14 min version conformance test by custom test (#11433)
a600f75 Bump bazel memory limit to 3gb (#11436)
ebe7420 fix es ingest: file too large (#11415)
d86eca0 Add documentation annotations to KV self-service errors [KVL-1144] (#11385)
46d31d5 [DPP-676][Self-service error codes] Adapt ApiConfigManagementService (#11402)
927b237 Make Oracle tests exclusive (#11408)
68d7f83 Switch bazel exec logs to json format (#11430)
12e782b ifaces: update ghc-parser (#11383)
503e391 [Short] Typo (#11410)
3031190 [DPP-676][Self-service error codes] Adapt request validators (#11379)
b412f71 Fix MonotonicRecordTimeIT for pruned participants (#11423)
500e5b0 release 1.18.0-snapshot.20211026.8179.0.e474b2d1 (#11421)
d678a40 LF: remove type parameter from ContractInstance (#11419)
```
Changelog:
```

- [@daml/react] Our React bindings now request the `react` library as a
  peer dependency, and will accept React 17 in addition to the current
  React 16. For new projects, the `create-daml-app` template now uses
  React 17.

- [Integration Kit] Document KV-specific self-service errors

[TS bindings] When using daml-react and daml-ledger, all streaming request to the query endpoint will be multiplexed through a single web socket.
[daml LF] - Add LoggingValue instance for Time.Timestamp
- [Integration Kit] - ledger-api-bench-tool can generate synthetic contracts with multiple observers
[api-test-tool] - remove AppendOnly prefix from test suites as the append-only schema is the only one left
```

CHANGELOG_BEGIN
CHANGELOG_END
garyverhaegen-da pushed a commit that referenced this pull request Nov 3, 2021
@azure-pipelines
release 1.18.0-snapshot.20211102.8257.0.7391a3cd (#11521)
d6fb67c 
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.

@Robin-da is in charge of this release.

Commit log:
```
7391a3c Migrate language-support codegen tests to sandbox (#11508)
da76e28 iface: check for interface implementations precond (#11494)
9e045c1 Remove a leftover from `check-protobuf-stability.sh` (#11513)
17c2fa9 ErrorCodeDocumentationGenerator uses multiple prefixes for scanning classpaths (#11504)
a8ae4d6 LF: Rename GenNode to Node (#11469)
73b6f8b Fixed ledger-api-bench-tool random distribution in submission phase. (#11507)
92495b0 Upgrade nixpkgs (#11490)
48afb7b interface: Add inherited choice names in scala AST (#11503)
9f8b039 [Trigger-Service] Migrate tests to sandbox (#11501)
6bafeb0 DPP-662 Support archivals in bench tool (#11474)
63d855f Pass `damlc build` logs through daml logger (#11485)
565dfac Bump rules_nodejs to 4.4.2 (#11495)
9214956 Enable experimental nix command (#11498)
f81d880 LF: drop com.daml.lf.transaction.Transaction.Transaction type alias. (#11470)
505a7f5 Extend debug logging around dependencies/data-dependencies (#11488)
55d35e5 Drop TLA+ from dev-env (#11492)
3a5cb18 [Self-service error codes] Adapt all grpc assertions to additionally take self-service error codes [DPP-680] (#11437)
98e7461 LF: ContractInstance should be a CidContainer (#11487)
9afb3b3 Add and wire ledger-end-cache [DPP-703] (#11480)
cdd2acc Explicitly import Java runtime from Nix/dadew (#11411)
b5540a9 Evidence for contract-key testing (#11479)
78735a8 Less logging on //docs:generate-error-codes-json (#11486)
d8898b8 Check fixed choices in name collision checker (#11481)
7a0198b daml-react: allow for React 17 (#11463)
c415d9d Migrate trigger tests from sandbox-classic to sandbox (#11478)
f9ec12b Add cocreature as a codeowner of the docs (#11450)
a79cc08 DbDispatcher handles SQL exceptions with self-service error codes [DPP-690] (#11435)
164cdbb update compat versions for 1.18.0-snapshot.20211026.8179.0.e474b2d1 (#11426)
741a6e7 interfaces: Add list of fixed choices in TemplateImplements structure. (#11364)
79505b5 oracle debug (#11386)
257d536 Switch from sandbox to sandbox-classic in daml test-script (#11476)
e91ced5 KV: generate self-service error docs [KVL-1148] (#11431)
8ffd835 Migrate daml script tests to Sandbox next (#11472)
df65d02 Exclude dedup tests affected by current changes (#11471)
45beaf7 interfaces: typecheck for precond, haskell side. (#11468)
e445fe1 LfEngineToApi conversions do not have to deal with version. (#11414)
d371274 Sandbox-classic: contracts sent to the engine do not need enrichment (#11416)
05312f1 Lift out constraint tuples to synonyms in dfun contexts (#11467)
a492a6c [DPP-636] Update docs on MacOS firewall issue when running tests (#11460)
5a85881 rotate release duty after 1.18.0-snapshot.20211026.8179.0.e474b2d1 (#11422)
1096f91 Adds default keepalive support for postgreSQL [DPP-697] (#11453)
3799963 Fix akka-streams shutdown at LedgerConfigurationSubscriptionFromIndex (#11462)
e89da7c Move non-repudiation proxy conformance tests to their own package (#11464)
b98a3ad interfaces: precondition, scala ast, decoder/encoder (#11452)
fd973e6 Drop assistant integration tests for relative/absolute DAML_PROJECT (#11461)
05505e3 Multiplex multiple virtual query streams through a single web socket (#10221)
3098b70 pretty print contract IDs properly (#11359)
bb4f4c5 DPP-665 Make in-flight commands configurable (#11456)
bf00956 replace OneAnd party Sets with NonEmpty Set (#11420)
570160b Add LoggingValue instance for Time.Timestamp (#11444)
bad50a5 Type reexports explicitly use 'type' namespace (#11451)
d3b9c84 ifaces: preconditions, proto changes (#11448)
73718a2 Mark script export tests as flaky (#11449)
22e9e6b Multi-observer command submission in the ledger-api-bench-tool [DPP-660] (#11413)
3eacbf9 Bazel cache retry on connection reset by peer (#11445)
fcc7e58 Make stable packages dependent on supported LF versions (#11382)
6c45f09 Remove TODO (#11446)
d8b92b7 Drop flaky marker from lsp-tests (#11442)
68f4432 Improve evidence of testing (#11428)
6126fc2 Remove append only prefix from conformance tests [KVL-1152] (#11424)
926655d Drop daml ledger timeout tests from compat tests (#11441)
15cb840 Switch back to binary execution logs (#11440)
ac28e61 Update ScalaPB to the latest version (0.11.6) (#11409)
a71a8f9 Prevent overflow in DeduplicationPeriod (#11432)
6c88e55 Pass submission id as correlation id to error codes in Ledger API (#11381)
ba10687 Replace 1.14 min version conformance test by custom test (#11433)
a600f75 Bump bazel memory limit to 3gb (#11436)
ebe7420 fix es ingest: file too large (#11415)
d86eca0 Add documentation annotations to KV self-service errors [KVL-1144] (#11385)
46d31d5 [DPP-676][Self-service error codes] Adapt ApiConfigManagementService (#11402)
927b237 Make Oracle tests exclusive (#11408)
68d7f83 Switch bazel exec logs to json format (#11430)
12e782b ifaces: update ghc-parser (#11383)
503e391 [Short] Typo (#11410)
3031190 [DPP-676][Self-service error codes] Adapt request validators (#11379)
b412f71 Fix MonotonicRecordTimeIT for pruned participants (#11423)
500e5b0 release 1.18.0-snapshot.20211026.8179.0.e474b2d1 (#11421)
d678a40 LF: remove type parameter from ContractInstance (#11419)
```
Changelog:
```

- [@daml/react] Our React bindings now request the `react` library as a
  peer dependency, and will accept React 17 in addition to the current
  React 16. For new projects, the `create-daml-app` template now uses
  React 17.

- [Integration Kit] Document KV-specific self-service errors

[TS bindings] When using daml-react and daml-ledger, all streaming request to the query endpoint will be multiplexed through a single web socket.
[daml LF] - Add LoggingValue instance for Time.Timestamp
- [Integration Kit] - ledger-api-bench-tool can generate synthetic contracts with multiple observers
[api-test-tool] - remove AppendOnly prefix from test suites as the append-only schema is the only one left
```

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines Daml Build <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stefanobaghino-da stefanobaghino-da stefanobaghino-da left review comments

@cocreature cocreature cocreature left review comments

@S11001001 S11001001 S11001001 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@garyverhaegen-da @S11001001 @cocreature @stefanobaghino-da