-
Notifications
You must be signed in to change notification settings - Fork 204
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Data dependencies support for reexports #11147
Conversation
e76eb75
to
71f1c40
Compare
] | ||
] | ||
|
||
, dataDependenciesTest "Using reexported values" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
strictly speaking this is a different issue than the one described in #10772, but I think they can both be taken care of with the same solution, namely using the md_exports
field of ModDetails
, which includes all values, types and classes exported by a module regardless of their origin.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we support more than just full module reexports that’s even better 👍
@@ -488,7 +488,14 @@ convertModule lfVersion pkgMap stablePackages isGenerated file x depOrphanModule | |||
templates <- convertTemplateDefs env | |||
exceptions <- convertExceptionDefs env | |||
interfaces <- convertInterfaces env (eltsUFM (cm_types x)) | |||
pure (LF.moduleFromDefinitions lfModName (Just $ fromNormalizedFilePath file) flags (types ++ templates ++ exceptions ++ definitions ++ interfaces ++ depOrphanModules)) | |||
let defs = | |||
types |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I got a bit tired of the super long line, so I factored this out. Please let me know if this is the style you'd prefer for splitting the expression into lines, or if you prefer operators at the end or something different.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don’t think we have any strong guidelines here. This looks perfectly reasonable.
compiler/damlc/daml-lf-conversion/src/DA/Daml/LFConversion/MetadataEncoding.hs
Outdated
Show resolved
Hide resolved
71f1c40
to
4db8fe7
Compare
835def6
to
4dda6a7
Compare
Please note that a few tests are failing, weirdly saying that a type or class defined in that module is not in scope, for example,
|
The error with |
That silenced some errors (and some warnings), but I'm still getting quite a few errors such as this one:
|
c22e09d
to
70b2ac3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work, thank you!
@@ -488,7 +488,14 @@ convertModule lfVersion pkgMap stablePackages isGenerated file x depOrphanModule | |||
templates <- convertTemplateDefs env | |||
exceptions <- convertExceptionDefs env | |||
interfaces <- convertInterfaces env (eltsUFM (cm_types x)) | |||
pure (LF.moduleFromDefinitions lfModName (Just $ fromNormalizedFilePath file) flags (types ++ templates ++ exceptions ++ definitions ++ interfaces ++ depOrphanModules)) | |||
let defs = | |||
types |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don’t think we have any strong guidelines here. This looks perfectly reasonable.
] | ||
] | ||
|
||
, dataDependenciesTest "Using reexported values" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we support more than just full module reexports that’s even better 👍
@@ -20,24 +20,68 @@ def get_text(pkg): .text_interned_str | resolve_interned_string(pkg); | |||
|
|||
def norm_ty(pkg): if has("interned") then pkg.interned_types[.interned] else . end; | |||
|
|||
# @SINCE-LF 1.7 | |||
def norm_qualified_module(pkg; f): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we could factor out a generic struct normalization or something but not quite sure that works and definitely does not have to happen in this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah! I've wished for that way too many times while working on this
. (\(LFC.QualName q) -> LF.qualModule q) | ||
. exportName | ||
where | ||
forbiddenModules = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is that the same list targeted by
isInternal (GHC.moduleNameString -> x) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
aha! So far it is, thanks for sharing this!
availInfoToExportInfo = \case | ||
GHC.Avail name -> ExportInfoVal | ||
<$> convertQualName name | ||
GHC.AvailTC name pieces fields -> ExportInfoTC |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there some way to detect a full module reexport? I’m slightly worried about the size of this expression exploding if you reexport a hole module that defines lots of identifiers. Probably not a huge issue but if we could detect those somehow and store them as a single "reexport this module" part instead of this that seems potentially much nicer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn’t have to be in this PR but probably worth thinking about.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not with md_exports
:( I'd have to look for a different API, probably something closer to the parser
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm yeah looks like we can only get it from hsmodExports
. Another option to avoid large expressions from blowing up could be to emit multiple export expressions at the extreme end one per export but we could also do some form of batching like 5 exports max per item or whatever a sensible number is.
A large number of expressions isn’t as problematic as a single very large expression because the latter will be handled relatively poorly by some of our code.
I iteratively added a few modules to skip, for now,
Now most data-dependencies tests pass, but this feels extremely ad-hoc and prone to break :/ A few tests still fail, and they mention missing values from modules such as |
which test specifically is failing? |
These two:
EDIT: which might make sense if the missing functions from |
Thanks will take a look later today or early tomorrow. Your idea sounds like it might be on the right track but I haven’t thought about it too deeply. |
I’ve looked into it and I think I understand the failure: data-dependencies for LF < 1.8 are very limited. Specifically, typeclasses and typeclass-constrained methods are not available here (they depend on LF type synonyms which only got introduced in 1.8). So I think there are two options:
2 seems harder to implement and at this point nobody should be on LF < 1.8 (it’s more than 2 years old iirc) so I’d suggest to go with 1. |
thanks for looking into it @cocreature! I'll go with option 1 |
04617bc
to
9ccda71
Compare
|
ab2323e
to
d7a32cc
Compare
@cocreature, how would I find a sensible number for the size of these batches? If that's too much trouble, I could go ahead with one expression per |
I don’t think there is a supersystematic way of coming up with these expressions unfortunately. I’d probably use 10 or so as a start but it’s definitely somewhat arbitrary. As for naming, yeah just appending a number sounds perfectly fine. |
This includes all the values exposed by the module.
changelog_begin changelog_end
d3f991c
to
ef7e273
Compare
ef7e273
to
22db0a0
Compare
This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @sofiafaro-da is in charge of this release. Commit log: ``` 8ff347d Expand type synonyms lazily (#11282) 2fd200f KV: support contextual logging in KeyValueConsumption [KVL-1143] (#11288) f4df1cc converting server errors to proper client errors (#11184) 525e4ce Add GrpcStatus.toProto(JavaStatus) -> ScalaStatus conversion [KVL-1143] (#11292) 9b00a1a Rotate release rotation (#11291) dd09c38 Upgrade rules-nodejs (#11290) 87f1418 [Short] Remove unnecessary traits from ApiSubmissionServiceSpec (#11254) 6b65a72 Merge npm install steps in create-daml-app tests (#11287) a9f6afb kvutils: Rename `VersionedOffset` to `KVOffset`. (#11286) 98cf8d8 KV: introduce v2 error codes behind a CLI switch [KVL-1140] (#11224) 46f6877 Increase time model skew limits (#11273) 8f94cff kvutils: Use `VersionedOffsetBuilder` where possible, and introduce `VersionedOffsetMutator`. [KVL-1154] (#11277) 81fde97 Bazel: Call `_wrap_rule` directly when building the Scala REPL rule. (#11279) edb2b04 Document scaladoc usage in Bazel (#11278) 6255837 Data dependencies support for reexports (#11147) 2fc7490 [Self-service error codes] Adapt error factories [DPP-656] (#11270) b1a6b11 ledger-api-test-tool: Add static time awareness [KVL-1156] (#11266) 243c120 Add an LF typechecking benchmark (#11276) 2db8c13 Test authorization within Exercise choice. (#11246) ec58ed6 Typecheck nested LF type applications more efficiently (#11253) a940016 Add debugRaw corresponding to traceM in Haskell (#11259) a988579 Fix Scala repl targets (#11272) e4808f6 Extract UtilLF module from daml-lf-conversion to its own library (#11263) 38712c0 sandbox-next - Disable participant command deduplication [KVL-1156] (#11264) 82fa229 Add daml-util-ghc lib (#11260) 68a93cf kvutils - Use the ledger configured time for command dedup [KVL-1149] (#11239) c8cede0 Bump startup time in create-daml-app tests (#11261) 4fac87f Bump the schema version for the JSON API (#11252) 5507670 update NOTICES file (#11256) 0063b10 Retry faster during startup (#11255) 04feb40 Improve reset service tests (#11240) b3375fd update compat versions for 1.18.0-snapshot.20211013.8071.0.514e8b50 (#11237) 9ed1eb3 Address security notice on `ansi-regex` (#11243) 387d0e8 Make compatibility migrations `eternal` (#11242) be899b3 Extract rejection_reason.proto from daml_kvutils.proto [KVL-1090] (#11235) a1d94e1 LF: Create special case class for VersionedContractInstance (#11127) b738988 Release 1.18 snapshot (#11234) 3c26852 LF: Use template Id in exercise node of fixed choice. (#11229) 139b6f3 CI: Set `PROJ_DIR` inside the bash lib, not outside. (#11236) ce64cb2 interfaces: Do some TODOs (#11231) 7a88c7d trigger-service: dev-mode-unsafe flag (#11233) 909a1bf [DPP-417][DDP-612] Adapt ApiSubmissionService to support V2 error codes (#11052) ``` Changelog: ``` - [JSON API] Several kinds of gRPC server errors are now reported with associated HTTP statuses; for example, a Daml-LF interpreter error now returns a 400 instead of a 500, and an exercise on an archived contract returns a 409 Conflict instead of a 500. Errors internal to JSON API (e.g. internal assertion failures) are no longer detailed in the HTTP response; their details are only logged. See `issue #11184 <https://github.com/digital-asset/daml/pull/11184>`__. - [Daml Stdlib] Add `debugRaw` as a convenience wrapper around `traceRaw` when used inside a do-block. `debugRaw` compares to `debug` like `traceRaw` compares to `trace` meaning it expects a `Text` instead of calling `show` on an expression. sandbox-next - Disable participant command deduplication kvutils - Command deduplication uses the configured time model (static/wall) and not running always using wall-clock [JSON API] Solving a bug that could cause the JSON API to return correct result if a contract with the same key is observed twice required a schema change. The JSON API data needs to be dropped and the query store needs to reset. If you are migrating from a previous version, either reset your database manually or start the HTTP JSON API with one of the options that regenerate the schema (`create-only`, `create-if-needed-and-start`, `create-and-start`). ``` CHANGELOG_BEGIN CHANGELOG_END
This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging. @sofiafaro-da is in charge of this release. Commit log: ``` 8ff347d Expand type synonyms lazily (#11282) 2fd200f KV: support contextual logging in KeyValueConsumption [KVL-1143] (#11288) f4df1cc converting server errors to proper client errors (#11184) 525e4ce Add GrpcStatus.toProto(JavaStatus) -> ScalaStatus conversion [KVL-1143] (#11292) 9b00a1a Rotate release rotation (#11291) dd09c38 Upgrade rules-nodejs (#11290) 87f1418 [Short] Remove unnecessary traits from ApiSubmissionServiceSpec (#11254) 6b65a72 Merge npm install steps in create-daml-app tests (#11287) a9f6afb kvutils: Rename `VersionedOffset` to `KVOffset`. (#11286) 98cf8d8 KV: introduce v2 error codes behind a CLI switch [KVL-1140] (#11224) 46f6877 Increase time model skew limits (#11273) 8f94cff kvutils: Use `VersionedOffsetBuilder` where possible, and introduce `VersionedOffsetMutator`. [KVL-1154] (#11277) 81fde97 Bazel: Call `_wrap_rule` directly when building the Scala REPL rule. (#11279) edb2b04 Document scaladoc usage in Bazel (#11278) 6255837 Data dependencies support for reexports (#11147) 2fc7490 [Self-service error codes] Adapt error factories [DPP-656] (#11270) b1a6b11 ledger-api-test-tool: Add static time awareness [KVL-1156] (#11266) 243c120 Add an LF typechecking benchmark (#11276) 2db8c13 Test authorization within Exercise choice. (#11246) ec58ed6 Typecheck nested LF type applications more efficiently (#11253) a940016 Add debugRaw corresponding to traceM in Haskell (#11259) a988579 Fix Scala repl targets (#11272) e4808f6 Extract UtilLF module from daml-lf-conversion to its own library (#11263) 38712c0 sandbox-next - Disable participant command deduplication [KVL-1156] (#11264) 82fa229 Add daml-util-ghc lib (#11260) 68a93cf kvutils - Use the ledger configured time for command dedup [KVL-1149] (#11239) c8cede0 Bump startup time in create-daml-app tests (#11261) 4fac87f Bump the schema version for the JSON API (#11252) 5507670 update NOTICES file (#11256) 0063b10 Retry faster during startup (#11255) 04feb40 Improve reset service tests (#11240) b3375fd update compat versions for 1.18.0-snapshot.20211013.8071.0.514e8b50 (#11237) 9ed1eb3 Address security notice on `ansi-regex` (#11243) 387d0e8 Make compatibility migrations `eternal` (#11242) be899b3 Extract rejection_reason.proto from daml_kvutils.proto [KVL-1090] (#11235) a1d94e1 LF: Create special case class for VersionedContractInstance (#11127) b738988 Release 1.18 snapshot (#11234) 3c26852 LF: Use template Id in exercise node of fixed choice. (#11229) 139b6f3 CI: Set `PROJ_DIR` inside the bash lib, not outside. (#11236) ce64cb2 interfaces: Do some TODOs (#11231) 7a88c7d trigger-service: dev-mode-unsafe flag (#11233) 909a1bf [DPP-417][DDP-612] Adapt ApiSubmissionService to support V2 error codes (#11052) ``` Changelog: ``` - [JSON API] Several kinds of gRPC server errors are now reported with associated HTTP statuses; for example, a Daml-LF interpreter error now returns a 400 instead of a 500, and an exercise on an archived contract returns a 409 Conflict instead of a 500. Errors internal to JSON API (e.g. internal assertion failures) are no longer detailed in the HTTP response; their details are only logged. See `issue #11184 <https://github.com/digital-asset/daml/pull/11184>`__. - [Daml Stdlib] Add `debugRaw` as a convenience wrapper around `traceRaw` when used inside a do-block. `debugRaw` compares to `debug` like `traceRaw` compares to `trace` meaning it expects a `Text` instead of calling `show` on an expression. sandbox-next - Disable participant command deduplication kvutils - Command deduplication uses the configured time model (static/wall) and not running always using wall-clock [JSON API] Solving a bug that could cause the JSON API to return correct result if a contract with the same key is observed twice required a schema change. The JSON API data needs to be dropped and the query store needs to reset. If you are migrating from a previous version, either reset your database manually or start the HTTP JSON API with one of the options that regenerate the schema (`create-only`, `create-if-needed-and-start`, `create-and-start`). ``` CHANGELOG_BEGIN CHANGELOG_END Co-authored-by: Azure Pipelines Daml Build <[email protected]>
Pull Request Checklist
CHANGELOG_BEGIN
andCHANGELOG_END
tagsNOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with
/AzurePipelines run
totrigger the build.