bug: property-level aliases/renaming doesn't appear to work

[kcphila]

Meltano Version

3.0.0

Python Version

3.10

Bug scope

CLI (options, error messages, logging, etc.)

Operating System

Linux - 20.04

Description

Multiple core docs specify that properties can be renamed with mappers:

• The meltano mappers guide documentation indicates, in the Property (Column) Level, section, that Aliasing: properties can be renamed in the resulting stream e.g. altering a column name from id -> customer_id
• The Meltano SDK stream_maps documentation, in the Property-Level Mapping Applications section, indicate that stream_maps support Property-level aliasing: properties can be renamed in the resulting stream.

However, in v3.0.0, either this no longer works or all documentation that provides examples have been lost, as I searched through the docs as well as the source code for both the SDK and the meltano-map-transform plugin.

For example, for a config file that looks like :

...
  mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+https://github.com/MeltanoLabs/meltano-map-transform.git
    mappings:
    - name: lower-case-schema
      config:
        stream_maps:
          {STREAM_NAME}:
            __alias__: {new-table-name}
            {COLUMN-PROPERTY}:
              __alias__: {new-property-name}
...

The stream/table name is renamed but the property/column is not. There is a partial work around to make a copy of the column and then delete the old column.

      config:
        stream_maps:
          {STREAM_NAME}:
            __alias__: {new-table-name}
            {new-property-name}: {COLUMN-PROPERTY}
            {COLUMN-PROPERTY}: __NULL__

Sorry if this isn't quite a "bug" or if issues with the mapper belong somewhere else - it was not clear where communication / contribution streams are or go (and Slack is unfortunately blocked at my workplace).

Code

...
  mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+https://github.com/MeltanoLabs/meltano-map-transform.git
    mappings:
    - name: lower-case-schema
      config:
        stream_maps:
          {STREAM_NAME}:
            __alias__: {new-table-name}
            {COLUMN-PROPERTY}:
              __alias__: {new-property-name}
...

[edgarrmondragon]

Hi @kcphila! Thanks for logging.

This format, with nested column properties is not actually supported

  mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+https://github.com/MeltanoLabs/meltano-map-transform.git
    mappings:
    - name: lower-case-schema
      config:
        stream_maps:
          {STREAM_NAME}:
            __alias__: {new-table-name}
            {COLUMN-PROPERTY}:
              __alias__: {new-property-name}

so I'm curious if you found an example like it in the docs, or whether you were following similar patterns for other Meltano settings.

---

There is a partial work around to make a copy of the column and then delete the old column.

Is the workaround not good enough for your use case? If so, what are you missing that's not supported by the "copy and delete" approach?

[kcphila]

This format, with nested column properties is not actually supported

Hi @edgarrmondragon,

Thanks for the quick response.

There weren't code examples of the nested column format (aside from some "this would be nice" in feature requests), but the two main mapper doc pages both list changing property names as an existing feature of mappers (meltano mappers guide documentation and Meltano SDK stream_maps documentation). So, if property renaming isn't actually an existing (or soon to be coming) feature, it should be removed from the docs.

The copy/delete is doable (especially since it looks like __else__: __NULL__ will clear out the original column names) - but the real need is to be able to handle renames/reformats of the property schema in bulk - Is there some other way to do inline property/schema changes? The meltano/singer Oracle tap pulls in all uppercase, and so what I'm really looking to do as a starting point is something like (and it does seem like it would require a special key to refer to the stream metadata definition):

      config:
        stream_maps:
          {STREAM_NAME}:
            __meta__[col['name'].lower()]: col

[kinghuang]

There weren't code examples of the nested column format (aside from some "this would be nice" in feature requests), but the two main mapper doc pages both list changing property names as an existing feature of mappers (meltano mappers guide documentation and Meltano SDK stream_maps documentation). So, if property renaming isn't actually an existing (or soon to be coming) feature, it should be removed from the docs.

I'm just encountering this myself, too. Property-level aliasing is listed as a feature, but I don't see how it can be configured. I am using the "copy and delete" approach to make things work.

[edgarrmondragon]

I think the request in #2545 (comment) is now supported with a combination of "copy-and-delete" and glob expressions (7eebef9).

To document:

• Added a test case to the mapper repo: MeltanoLabs/meltano-map-transform@ba466cb
• Added docs here in the SDK:
  • https://sdk.meltano.com/en/latest/stream_maps.html#aliasing-properties
  • https://sdk.meltano.com/en/latest/stream_maps.html#applying-a-mapping-across-two-or-more-streams