Add record flattening feature from SDK

[aaronsteers]

In theory, this should be a simple addition, since the feature should be available in the underlying SDK.

The SDK source code for this feature's config is here:

https://github.com/meltano/sdk/blob/3998ac14918b6b2338851b8bb428485a9378ff49/singer_sdk/helpers/capabilities.py#L29-L43

In theory, we may be able to simply extend this repo's config with those additional options.

Related: docs improvement for SDK

Making this slightly more complex, it seems flattening is also not well documented in the SDK docs.

I didn't see any reference to those options in the SDK docs, so I'll log a separate issue on the SDK to properly document those.

UPDATE: now logged here:

• Add docs for flattening_enabled and flattening_max_depth meltano/sdk#717

[teohm]

My team is looking for this flattening feature!

I'm curious about at which level we can control the flattening? e.g. can we specify a specific stream, or a specific property to be flattened?

[edgarrmondragon]

My team is looking for this flattening feature!

I'm curious about at which level we can control the flattening? e.g. can we specify a specific stream, or a specific property to be flattened?

Hi @teohm! As it currently stands, flattening is applied at a configurable level to all streams, and properties if they're at the right level.

What you ask would be an interesting enhancement, though. Would you mind opening an issue if the feature is a hard requirement for you to use flattening?

[ashish-atidiv]

Hi @edgarrmondragon

What is the progress on this feature request?

[edgarrmondragon]

@ashish-atidiv Record flattening is supported. You can see an example in https://github.com/MeltanoLabs/meltano-map-transform/pull/103/files#diff-ad41fb697ad72b7175f50fa33964f04d2535d1fe18a4502cb1e0f32b228971c5.

Note that flattening_max_depth has a bug (meltano/sdk#1400) so it's better to set it to a large enough value to flatten all objects.

[ashish-atidiv]

HI @edgarrmondragon
Thanks for the PR. I am excited to try this soon. Looking forward to have this merged quickly.

[ashish-atidiv]

Also @edgarrmondragon , can you tell how will the flattening logic treat arrays and list of dicts?

[edgarrmondragon]

meltano/sdk#1400 should be fixed by meltano/sdk#1939

[edgarrmondragon]

Also @edgarrmondragon , can you tell how will the flattening logic treat arrays and list of dicts?

@ashish-atidiv You can take a look at the snapshot in https://github.com/meltano/sdk/blob/0b2fe9e1cb740e5310692025812b8779b097b955/tests/snapshots/mapped_stream/flatten_all.jsonl, for example

[edgarrmondragon]

Closed by #103