Mapping file generation super-powers

[prescod]

1. Allow users to inject load declarations into generated mapping file.
2. Allow users to order mapping file declarations
3. Auto-inject "after:" declarations where appropriate.

(the description below is the vision...not every detail is implemented yet)

Still needs CCI side, documentation and tests.

Snowfakery and Mapping Files

CumulusCI's bulk loader depends on a mapping file to determine the load order of data.

Snowfakery generates this mapping file automatically in the generate_and_load_from_yaml job. Occasionally you may need to inject some additional information into the load. For example, you might want to change which API or processing mode to use (REST API, BULK API, Bulk Serial, Bulk Parallel). Or you might want to enforce a certain load order.

You can now do that by specifying a "recipe.load.yml" file to go with your recipe file. The --load-yml command line argument allows that and there is also a file naming convention which does it automatically (TBD).

Here are the declarations you may put into the "recipe.load.yml".

• sf_object: Contact
  bulk_mode: serial # same as specifying bulk_mode in mapping file

• sf_object: Contact
  load_after: Opportunity # load contacts after opportunities for some reasoon

• sf_object: Account # same as specifying api in mapping file
  api: rest

[davisagli]

I think the feature makes sense, but why a separate file instead of including these hints somewhere in the main recipe that the mapping generator knows to look for? Do you anticipate a need to use different sets of hints with the same recipe?

[prescod]

I think the feature makes sense, but why a separate file instead of including these hints somewhere in the main recipe that the mapping generator knows to look for? Do you anticipate a need to use different sets of hints with the same recipe?

Yes, I do anticipate that e.g. the standard NPSP recipe might work on your org but automation you've installed might make the load order unique to your org.

Or conversely (and perhaps more commonly!) the standard NPSP load file might work but you want to add a custom field to the recipe and therefore create your own.

In the not too distant future I could also imagine you sharing the files between Snowfakery data loads and CCI SQL loads instead of using the succinct Snowfakery format in one place and the verbose and unwieldy mapping file format in the other.

And then in the distant future, perhaps there is some centralized management of them analogous to TypeShed, DefinitelyTyped, SchemaStore.org etc. (or maybe that won't be useful enough)

Also: any syntax for adding CCI-specific information to recipe files would need to be either CCI-specfic syntax or a pragma syntax. I'd rather not muddy the recipe format with that.

[TheBitShepherd]

TIL: pysnooper!

[TheBitShepherd]

I took a look at the changes since we reviewed this together. All looks good. I really appreciated the addition of the docstrings!

[davisagli]

Still needs docs, right?

[prescod]

I'm planning to put the docs in the CCI repo.

…

On Fri, Mar 26, 2021 at 9:58 AM David Glick ***@***.***> wrote: Still needs docs, right? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#290 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFMS6N3GJD36C7LG2FY63TFS4LNANCNFSM4ZLLPTIQ> .