Add golden test for example Daml ledger export #9732

aherrmann-da · 2021-05-18T16:34:09Z

Adds an example Daml ledger export to the docs tree to be used as an example in the Daml ledger export documentation.
Incorporating it in the actual documentation is left for a follow-up PR.
Adds a Bazel build target to generate a Daml ledger export using client_server_build.
Adds a golden test that compares the example export in the docs to the one generated by the build target.
Some minor changes in support of the above:
- Expose an overload of the Daml script runner's main function that takes a config object
- Extend client_server_build to support multiple output files.
- Extend sh_inline_test to support toolchains for make variable substitution.

Pull Request Checklist

Read and understand the contribution guidelines
Include appropriate tests
Set a descriptive title and thorough description
Add a reference to the issue this PR will solve, if appropriate
Include changelog additions in one or more commit message bodies between the CHANGELOG_BEGIN and CHANGELOG_END tags
Normal production system change, include purpose of change in description

NOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with /AzurePipelines run to
trigger the build.

Let the user define a list of output files on client_server_build. changelog_begin changelog_end

changelog_begin changelog_end

Support toolchain arguments to sh_inline_test. Usefuly for make variable expansion, e.g. for the POSIX toolchain.

aherrmann-da · 2021-05-18T16:35:06Z

daml-script/runner/src/main/scala/com/digitalasset/daml/lf/engine/script/RunnerMain.scala

@@ -29,108 +29,109 @@ import com.daml.auth.TokenHolder
 object RunnerMain {


This is best reviewed ignoring white-space changes.

cocreature

Nice, thank you!

cocreature · 2021-05-19T07:54:44Z

docs/source/tools/export/output-root/daml.yaml

+script-options: [--input-file, args.json]
+parties: [Bank,Alice,Bob]
+dependencies: [daml-stdlib, daml-prim, daml-script]
+data-dependencies: [EXPORT_OUT/deps/daml-stdlib-0.0.0-68a8ef91f10c283e1a5fc914bbf0addd41a089ad3625641966df2415e6010e2e.dalf,EXPORT_OUT/deps/script-test-0.0.1-87967f7d2b53db9652bd343a24db7cbe371c4b105005d57eed853bd309bac477.dalf,EXPORT_OUT/deps/c1f1f00558799eec139fb4f4c76f95fb52fa1837a5dd29600baa1c8ed1bdccfd.dalf,EXPORT_OUT/deps/733e38d36a2759688a4b2c4cec69d48e7b55ecc8dedc8067b815926c917a182a.dalf,EXPORT_OUT/deps/bfcd37bd6b84768e86e432f5f6c33e25d9e7724a9d42e33875ff74f6348e733f.dalf,EXPORT_OUT/deps/518032f41fd0175461b35ae0c9691e08b4aea55e62915f8360af2cc7a1f2ba6c.dalf,EXPORT_OUT/deps/cc348d369011362a5190fe96dd1f0dfbc697fdfd10e382b9e9666f0da05961b7.dalf,EXPORT_OUT/deps/6839a6d3d430c569b2425e9391717b44ca324b88ba621d597778811b2d05031d.dalf,EXPORT_OUT/deps/99a2705ed38c1c26cbb8fe7acf36bbf626668e167a33335de932599219e0a235.dalf,EXPORT_OUT/deps/76bf0fd12bd945762a01f8fc5bbcdfa4d0ff20f8762af490f8f41d6237c6524f.dalf,EXPORT_OUT/deps/e22bce619ae24ca3b8e6519281cb5a33b64b3190cc763248b4c3f9ad5087a92c.dalf,EXPORT_OUT/deps/d58cf9939847921b2aab78eaa7b427dc4c649d25e6bee3c749ace4c3f52f5c97.dalf,EXPORT_OUT/deps/6c2c0667393c5f92f1885163068cd31800d2264eb088eb6fc740e11241b2bf06.dalf,EXPORT_OUT/deps/d14e08374fc7197d6a0de468c968ae8ba3aadbf9315476fd39071831f5923662.dalf,EXPORT_OUT/deps/057eed1fd48c238491b8ea06b9b5bf85a5d4c9275dd3f6183e0e6b01730cc2ba.dalf,EXPORT_OUT/deps/e491352788e56ca4603acc411ffe1a49fefd76ed8b163af86cf5ee5f4c38645b.dalf,EXPORT_OUT/deps/daml-prim-0.0.0-b76d13799ea5488f281440b63072ad3e3d48d72936144e7d19209c9cf115aa9d.dalf,EXPORT_OUT/deps/40f452260bef3f29dede136108fc08a88d5a5250310281067087da6f0baddff7.dalf,EXPORT_OUT/deps/daml-stdlib-DA-Set-Types-1.0.0-97b883cd8a2b7f49f90d5d39c981cf6e110cf1f1c64427a28a6d58ec88c43657.dalf,EXPORT_OUT/deps/daml-script-0.0.0-024f7a831e04595b36c668121dce9209d63e5a6a46f01a4533fca5e4c521e7e4.dalf,EXPORT_OUT/deps/8a7806365bbd98d88b4c13832ebfa305f6abaeaf32cfa2b7dd25c4fa489b79fb.dalf]


maybe worth changing our generated code to split this over multiple lines. this is getting out of hand

Good idea. I'll address it in a separate PR as it seems out of scope for this one.

stefanobaghino-da

LGTM, just a couple of notes.

stefanobaghino-da · 2021-05-19T07:48:29Z

daml-script/export/BUILD.bazel

+# Compare the generated Daml ledger export to the example export used in the
+# documentation. This functions as both a golden test on ledger exports and to
+# make sure that the documentation stays up-to-date.


stefanobaghino-da · 2021-05-19T07:51:43Z

docs/source/tools/export/output-root/Export.daml

+-- Copyright (c) 2021 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
+-- SPDX-License-Identifier: Apache-2.0


Is this file the golden master? Does the comparison somehow skip comments? Should we add this file to the license checking exclusion list?

Answering my own question: yes, the comparison skips comments. To make sure we don't change the golden master every year, it would probably make sense to remove the license header.

I think we update the headers in all our docs every year so including the golden master in that seems probably more consistent 🤷

No strong opinion on my side. I'm happy to keep it this way for consistency.

stefanobaghino-da · 2021-05-19T07:53:29Z

daml-script/export/BUILD.bazel

+# Normalize the expected file by removing the copyright header and any documentation import markers.
+# Normalize the actual output by adding a newline to the last line if missing.


Should these comments go on top of the test rule? Spotting comments on inlined scripts can be tricky. Duplicating the comment is probably fine.

That's a good point. I've expanded the top comment and kept the comment in the inline code.

The JSON formatting of the args file uses Windows line endings on Windows. Therefore, the args.json output technically differs from the expected output. We're happy to ignore the difference in CRLF here.

@aherrmann-da

This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @aherrmann-da is in charge of this release. Commit log: ``` 2710fad Normalize rollback nodes when a transaction is constructed (#9712) 646c956 new windows signing (#9786) cae4292 Upgrade rules-nodes to version 3.5.0 (#9635) a336363 ghc-lib: update (#9742) 8ef9361 Scenario: Test ArithemticError (#9757) ad529e9 http-json-perf: querying larger ACSes with different data patterns (#9530) 0b24655 Fix archive calls in ExceptionsIT (#9787) abb7142 Update LF spec for exceptions. (#9784) d208979 Make succ/pred throw an ArithmeticError on overflow. (#9783) cd2ed07 Rename ContractError to PreconditionFailed (#9782) 1b428be Add ledger API test tool tests for rollback projections (#9778) fb6f72c debug signtool on main (#9780) 459de1c update NOTICES file (#9781) 61293c3 maybe fix azuresigntool invocation (#9779) 66b8f19 DPP-406 split append only migration (#9777) 45bca6e test_windows_signing: install for u (#9776) ed36caa Make use of nameOf for naming telemetry traces in participant-integration-api (#9762) 6f20d78 Makes sandbox-classic compression configurable (#9770) b8afa02 test_windows_signing: add quotes (#9773) 4af6608 fix signing machine (#9772) 5717356 fix test_windows_signing job name (#9771) 5855a99 LF: Drop CONTRACT_ERROR (#9754) f5c5b63 prepare for EV Windows signing (#9758) 88c83ec Mark //daml-assistant/daml-helper:test-deployment as flaky (#9769) fa0815b Example export (#9756) 3b33397 update NOTICES file (#9767) 4f97a4d Avoid test log collisions across platforms (#9763) 6f631da Use munit for diffs in ledger-api-test-tool (#9765) 92568eb Make hikari connection pool timeout configurable (#9750) 38cb8b4 Fix flaky CMConcurrentSetConflicting conformance test (#9766) d2bce16 DPP-335 Implement data migration for the append-only schema (#9481) 765d7e3 Throw ContractError in template precondition field (#9760) f5a03b3 Stop leaking Sdk installations from assistant integration tests (#9751) a277560 LF: change format of ArithmeticError message (#9761) 90180eb Make sure we don’t start leaking directories in compat tests (#9759) beb2d13 Add cli option & system property to enable json only logging for http json service (#9725) 3827f82 Fix consumedBy in rollback nodes for scenario service (#9746) 2ba5ca7 Make DA.Assert throw AssertionFailed instead of GeneralError (#9747) b9c36c9 Disable per commit windows compat tests (#9755) b1ca310 ledger: Damlification of Scala files (#9667) 40cf26f Log improvements for parallel indexer (#9752) 3bf6531 ledger-service, navigator: Damlification of Scala files (#9669) f947c02 Extend ConfigManagementServiceIT with tests on generation checking (#9753) f5c84a2 Daml-LF: Damlification of Scala/Haskell files (#9666) 0d931b2 Speedy: implement Arithmetic builtin exceptions (#9653) 5f954da Fix flakiness in hot reload tests (#9748) 40b2381 Compiler: clean convertion builtin name (#9724) 7b1a776 Dpp 384 todo cleanup postgres storage backend (#9727) 2e9bb21 Engine: clean conversion builtin names (#9726) 8efdde6 FutureResourceOwner --> ResourceOwner.forTry (#9720) 1794a6f Refactor GenerateStablePackages (#9744) 14cfe99 update compat versions for 1.14.0-snapshot.20210518.6953.0.a6c7b86a (#9741) 83d60f0 more frequent Windows cache cleanup (#9743) 10177d2 Add golden test for example Daml ledger export (#9732) bb5dd4c Fix locking of envScenarioContexts (#9736) fcbba1c LF: rename convertion builtins to be more obvious (#9716) b55613b Parallelize the data-dependencies test. (#9733) 3209188 Add a ledger model page for exceptions. (#9396) 6245e34 Drop damlc integration test todo (#9735) ae523c7 `ledger-api-bench-tool` - support for multiple streams [DPP-372] (#9691) 0f6f654 rotate release duty after 1.14.0-snapshot.20210518.6953.0.a6c7b86a (#9738) 123e142 release 1.14.0-snapshot.20210518.6953.0.a6c7b86a (#9737) 2dbfe43 Fix conversion of TryContextInfo in scenario service (#9731) e375fef update NOTICES file (#9739) ``` Changelog: ``` [jdbc ledger] increase default hikari connection pool timeout to 2s from 250ms - [Ledger HTTP Json Service] Logging output can now be in JSON either via providing the cli option `--log-encoder json` or via setting the env var `LOG_FORMAT_JSON=true` - [Daml Standard Library] `assert`, `(===)`, and other assertion functions (see DA.Assert) now use a new `CanAssert` typeclass constraint instead of `CanAbort`, in preparation for exceptions support. - [docs] The Daml ledger model has been updated to describe the changes introduced by the new exceptions feature. See here: https://docs.daml.com/concepts/ledger-model/ledger-exceptions.html - [Integration Kit] - Created the ledger-api-bench-tool prototype for benchmarking ledger transaction streaming capabilities - [Integration Kit] - added metrics to the ledger-api-bench-tool - [Integration Kit] - support for multiple streams in the ledger-api-bench-tool ``` CHANGELOG_BEGIN CHANGELOG_END

aherrmann added 7 commits May 18, 2021 18:26

client_server_build - user defined outputs

937c7d7

Let the user define a list of output files on client_server_build. changelog_begin changelog_end

daml-script runner expose main(config)

226f3e1

changelog_begin changelog_end

Add an example Daml ledger export to the docs

4c35dfe

Build Daml ledger export

c4a0dbe

changelog_begin changelog_end

sh_inline_test

cd14a21

Support toolchain arguments to sh_inline_test. Usefuly for make variable expansion, e.g. for the POSIX toolchain.

Add a golden test for Daml ledger export

06db16c

Test args files as well

8679d5f

aherrmann-da added this to the Daml Script dumps milestone May 18, 2021

aherrmann-da requested a review from cocreature May 18, 2021 16:34

aherrmann-da requested review from bame-da, fabiotudone-da, gerolf-da, miklos-da and nemanja-da as code owners May 18, 2021 16:34

aherrmann-da requested a review from a user May 18, 2021 16:34

aherrmann-da requested a review from stefanobaghino-da as a code owner May 18, 2021 16:34

aherrmann-da commented May 18, 2021

View reviewed changes

Use sed from POSIX toolchain

3f46d8f

cocreature approved these changes May 19, 2021

View reviewed changes

stefanobaghino-da approved these changes May 19, 2021

View reviewed changes

aherrmann added 2 commits May 19, 2021 13:30

Add normalization comment to top-level comment

1d089d9

Ignore trailing CR on Windows

0641916

The JSON formatting of the args file uses Windows line endings on Windows. Therefore, the args.json output technically differs from the expected output. We're happy to ignore the difference in CRLF here.

aherrmann-da added the automerge label May 19, 2021

mergify bot merged commit 10177d2 into main May 19, 2021

mergify bot deleted the example-export branch May 19, 2021 12:02

aherrmann-da mentioned this pull request May 20, 2021

Example export #9756

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add golden test for example Daml ledger export #9732

Add golden test for example Daml ledger export #9732

aherrmann-da commented May 18, 2021

aherrmann-da May 18, 2021

cocreature left a comment

cocreature May 19, 2021

aherrmann-da May 19, 2021

stefanobaghino-da left a comment

stefanobaghino-da May 19, 2021

stefanobaghino-da May 19, 2021

stefanobaghino-da May 19, 2021

cocreature May 19, 2021

aherrmann-da May 19, 2021

stefanobaghino-da May 19, 2021

aherrmann-da May 19, 2021

		@@ -29,108 +29,109 @@ import com.daml.auth.TokenHolder
		object RunnerMain {

		-- Copyright (c) 2021 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
		-- SPDX-License-Identifier: Apache-2.0

		# Normalize the expected file by removing the copyright header and any documentation import markers.
		# Normalize the actual output by adding a newline to the last line if missing.

Add golden test for example Daml ledger export #9732

Add golden test for example Daml ledger export #9732

Conversation

aherrmann-da commented May 18, 2021

Pull Request Checklist

Choose a reason for hiding this comment

cocreature left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefanobaghino-da left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment