Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Example export #9756

Merged
merged 12 commits into from
May 21, 2021
Merged

Example export #9756

merged 12 commits into from
May 21, 2021

Conversation

aherrmann-da
Copy link
Contributor

Pretty print the generated daml.yaml file as suggested in #9732 (comment).

Adds the generated daml.yaml to the golden test. In this case we need to normalize the data-dependencies by replacing the path to the output directory by a placeholder and by replacing the package-id hashes by a placeholder before comparison to the golden file, so that it doesn't have to be updated whenever a stdlib package-id changes.

Sorts the list of data-dependencies in the generated daml.yaml file to improve reproducibility.

Pull Request Checklist

  • Read and understand the contribution guidelines
  • Include appropriate tests
  • Set a descriptive title and thorough description
  • Add a reference to the issue this PR will solve, if appropriate
  • Include changelog additions in one or more commit message bodies between the CHANGELOG_BEGIN and CHANGELOG_END tags
  • Normal production system change, include purpose of change in description

NOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with /AzurePipelines run to
trigger the build.

@aherrmann-da aherrmann-da added this to the Daml Script dumps milestone May 20, 2021
@aherrmann-da aherrmann-da requested a review from cocreature May 20, 2021 10:12
aherrmann-da
aherrmann-da commented May 20, 2021
View reviewed changes
daml-script/export/BUILD.bazel
Comment on lines +155 to +158
server_args = [
"--port",
"0",
],
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was an oversight in #9732. I forgot to configure dynamic port selection there.

cocreature
cocreature approved these changes May 20, 2021
View reviewed changes
Copy link
Contributor

@cocreature cocreature left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thank you!

aherrmann added 2 commits May 20, 2021 13:50
@aherrmann
Factor out deleteRecursively
22df85c 
changelog_begin
changelog_end
@aherrmann
Generate ledger export in temp-dir
fb8e47a 
Build actions are not sandboxed in Windows. The ledger export will also
created a directory `deps` to store all the DALFs in. On Windows this
directory may persist across Bazel builds and may cause errors due to
attempts to overwrite an existing directory.

To avoid these issues the ledger export is generated into a temporary
directory and only the wanted files are moved to the expected output
locations in the bazel execroot afterwards.
@aherrmann-da aherrmann-da requested a review from a user May 20, 2021 15:29
@aherrmann-da
Copy link
Contributor Author

Windows was failing with

java.nio.file.FileAlreadyExistsException: bazel-out\x64_windows-opt\bin\daml-script\export\example-export\deps

because the ledger export was generated directly into the Bazel execroot. This is problematic on Windows due to the lack of sandboxing where undeclared outputs may persist across builds and then cause conflicts like the above with later builds.
To avoid this I've changed the code to generated the ledger export into a temp-dir and then only move the exected outputs to the execroot.

aherrmann added 3 commits May 20, 2021 17:47
@aherrmann
Close source
03d3457
@aherrmann
Use replace instead of replaceFirst
8672013 
The latter matches regex pattern instead of just substring matching
which fails with Windows paths due to `\` being interpreted as an escape
character.
@aherrmann
Normalize Windows \ to Unix /
b346271
@aherrmann-da aherrmann-da merged commit fa0815b into main May 21, 2021
@aherrmann-da aherrmann-da deleted the example-export branch May 21, 2021 07:36
azure-pipelines bot pushed a commit that referenced this pull request May 26, 2021
release 1.14.0-snapshot.20210525.7017.0.2710fad0
aca9db1 
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.

@aherrmann-da is in charge of this release.

Commit log:
```
2710fad Normalize rollback nodes when a transaction is constructed (#9712)
646c956 new windows signing (#9786)
cae4292 Upgrade rules-nodes to version 3.5.0 (#9635)
a336363 ghc-lib: update (#9742)
8ef9361 Scenario: Test ArithemticError (#9757)
ad529e9 http-json-perf: querying larger ACSes with different data patterns (#9530)
0b24655 Fix archive calls in ExceptionsIT (#9787)
abb7142 Update LF spec for exceptions. (#9784)
d208979 Make succ/pred throw an ArithmeticError on overflow. (#9783)
cd2ed07 Rename ContractError to PreconditionFailed (#9782)
1b428be Add ledger API test tool tests for rollback projections (#9778)
fb6f72c debug signtool on main (#9780)
459de1c update NOTICES file (#9781)
61293c3 maybe fix azuresigntool invocation (#9779)
66b8f19 DPP-406 split append only migration (#9777)
45bca6e test_windows_signing: install for u (#9776)
ed36caa Make use of nameOf for naming telemetry traces in participant-integration-api (#9762)
6f20d78 Makes sandbox-classic compression configurable (#9770)
b8afa02 test_windows_signing: add quotes (#9773)
4af6608 fix signing machine (#9772)
5717356 fix test_windows_signing job name (#9771)
5855a99 LF: Drop CONTRACT_ERROR (#9754)
f5c5b63 prepare for EV Windows signing (#9758)
88c83ec Mark //daml-assistant/daml-helper:test-deployment as flaky (#9769)
fa0815b Example export (#9756)
3b33397 update NOTICES file (#9767)
4f97a4d Avoid test log collisions across platforms (#9763)
6f631da Use munit for diffs in ledger-api-test-tool (#9765)
92568eb Make hikari connection pool timeout configurable (#9750)
38cb8b4 Fix flaky CMConcurrentSetConflicting conformance test (#9766)
d2bce16 DPP-335 Implement data migration for the append-only schema (#9481)
765d7e3 Throw ContractError in template precondition field (#9760)
f5a03b3 Stop leaking Sdk installations from assistant integration tests (#9751)
a277560 LF: change format of ArithmeticError message (#9761)
90180eb Make sure we don’t start leaking directories in compat tests (#9759)
beb2d13 Add cli option & system property to enable json only logging for http json service (#9725)
3827f82 Fix consumedBy in rollback nodes for scenario service (#9746)
2ba5ca7 Make DA.Assert throw AssertionFailed instead of GeneralError (#9747)
b9c36c9 Disable per commit windows compat tests (#9755)
b1ca310 ledger: Damlification of Scala files (#9667)
40cf26f Log improvements for parallel indexer (#9752)
3bf6531 ledger-service, navigator: Damlification of Scala files (#9669)
f947c02 Extend ConfigManagementServiceIT with tests on generation checking (#9753)
f5c84a2 Daml-LF: Damlification of Scala/Haskell files (#9666)
0d931b2 Speedy: implement Arithmetic builtin exceptions (#9653)
5f954da Fix flakiness in hot reload tests (#9748)
40b2381 Compiler: clean convertion builtin name (#9724)
7b1a776 Dpp 384 todo cleanup postgres storage backend (#9727)
2e9bb21 Engine: clean conversion builtin names (#9726)
8efdde6 FutureResourceOwner --> ResourceOwner.forTry (#9720)
1794a6f Refactor GenerateStablePackages (#9744)
14cfe99 update compat versions for 1.14.0-snapshot.20210518.6953.0.a6c7b86a (#9741)
83d60f0 more frequent Windows cache cleanup (#9743)
10177d2 Add golden test for example Daml ledger export (#9732)
bb5dd4c Fix locking of envScenarioContexts (#9736)
fcbba1c LF: rename convertion builtins to be more obvious (#9716)
b55613b Parallelize the data-dependencies test. (#9733)
3209188 Add a ledger model page for exceptions. (#9396)
6245e34 Drop damlc integration test todo (#9735)
ae523c7 `ledger-api-bench-tool` - support for multiple streams [DPP-372] (#9691)
0f6f654 rotate release duty after 1.14.0-snapshot.20210518.6953.0.a6c7b86a (#9738)
123e142 release 1.14.0-snapshot.20210518.6953.0.a6c7b86a (#9737)
2dbfe43 Fix conversion of TryContextInfo in scenario service (#9731)
e375fef update NOTICES file (#9739)
```
Changelog:
```
[jdbc ledger] increase default hikari connection pool timeout to 2s from 250ms
- [Ledger HTTP Json Service] Logging output can now be in JSON either via providing the cli option `--log-encoder json` or via setting the env var `LOG_FORMAT_JSON=true`
- [Daml Standard Library] `assert`, `(===)`, and other assertion functions (see DA.Assert) now use a new `CanAssert` typeclass constraint instead of `CanAbort`, in preparation for exceptions support.
- [docs] The Daml ledger model has been updated to describe the changes introduced by the new exceptions feature. See  here: https://docs.daml.com/concepts/ledger-model/ledger-exceptions.html
- [Integration Kit] - Created the ledger-api-bench-tool prototype for benchmarking ledger transaction streaming capabilities
- [Integration Kit] - added metrics to the ledger-api-bench-tool
- [Integration Kit] - support for multiple streams in the ledger-api-bench-tool
```

CHANGELOG_BEGIN
CHANGELOG_END
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cocreature cocreature cocreature approved these changes

@bame-da bame-da Awaiting requested review from bame-da

@nemanja-da nemanja-da Awaiting requested review from nemanja-da

@stefanobaghino-da stefanobaghino-da Awaiting requested review from stefanobaghino-da

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Daml Script dumps
Development

Successfully merging this pull request may close these issues.
3 participants
@aherrmann-da @cocreature @aherrmann