🎉 Source Shopify: Add metafield streams

[artem1205]

What

Resolving #5493

How

Add metafield streams

Recommended reading order

1. x.java
2. y.python

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub by running the /publish command described here
• After the connector is published, connector added to connector index as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

[bazarnov]

Minor changes are required. Please refer to the comments.

Also, let's try to cache the Products stream, since we have it re-used a lot for sub-sequent streams. To do so, simply add use_cache = True attribute to the Products stream class, the cache will be created and should be re-used for all other subsequent stream classes where possible (if possible), this will also reduce the sync time dramatically, I think. WDYT?

[artem1205]

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3252588598
❌ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3252588598
🐛 https://gradle.com/s/eanicrrk2ds6a

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/unit_test.py::test_privileges_validation - AssertionError: ...
	 �[31m================== �[31m�[1m1 failed�[0m, �[32m23 passed�[0m, �[33m92 warnings�[0m�[31m in 0.65s�[0m�[31m ===================�[0m

[artem1205]

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3252691468
❌ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3252691468
🐛 https://gradle.com/s/3tddte24fal4o

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs1] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs2] - so...
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
================== 4 failed, 47 passed in 1341.51s (0:22:21) ===================

[artem1205]

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3264128887
❌ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3264128887
🐛 https://gradle.com/s/ickzt6lg4buve

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs1] - so...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs2] - so...
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
======================== 8 failed, 43 passed in 48.10s =========================

[artem1205]

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3264428114
❌ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3264428114
🐛 https://gradle.com/s/nh7u6thvdsvjw

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [3] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:235: Backward compatibility tests are disabled for version 0.1.38.
============= 1 failed, 47 passed, 3 skipped in 1337.53s (0:22:17) =============

[artem1205]

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3264945674
❌ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3264945674
🐛 https://gradle.com/s/xit4hgrmcx5bo

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_incremental.py::TestIncremental::test_state_with_abnormally_large_values[inputs0]
SKIPPED [3] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:235: Backward compatibility tests are disabled for version 0.1.38.
============= 1 failed, 47 passed, 3 skipped in 1291.03s (0:21:31) =============

[artem1205]

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3265970253
❌ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3265970253
🐛 https://gradle.com/s/stf52pgshqtsa

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
SKIPPED [3] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:235: Backward compatibility tests are disabled for version 0.1.38.
============= 1 failed, 47 passed, 3 skipped in 1587.67s (0:26:27) =============

[artem1205]

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3266658108
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3266658108
Python tests coverage:

Name                          Stmts   Miss  Cover
-------------------------------------------------
source_shopify/__init__.py        2      0   100%
source_shopify/transform.py      58      3    95%
source_shopify/utils.py          58      6    90%
source_shopify/auth.py           20      4    80%
source_shopify/source.py        402    122    70%
-------------------------------------------------
TOTAL                           540    135    75%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [3] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:235: Backward compatibility tests are disabled for version 0.1.38.
================== 48 passed, 3 skipped in 1366.08s (0:22:46) ==================

[bazarnov]

/test connector=connectors/source-shopify

🕑 connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3267559648
✅ connectors/source-shopify https://github.com/airbytehq/airbyte/actions/runs/3267559648
Python tests coverage:

Name                          Stmts   Miss  Cover
-------------------------------------------------
source_shopify/__init__.py        2      0   100%
source_shopify/transform.py      58      3    95%
source_shopify/utils.py          58      6    90%
source_shopify/auth.py           20      4    80%
source_shopify/source.py        403    123    69%
-------------------------------------------------
TOTAL                           541    136    75%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       145     20    86%   21-23, 29-31, 36-43, 48-61, 224
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     10    87%   15-16, 24-30, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1351    453    66%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [3] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:235: Backward compatibility tests are disabled for version 0.1.38.
================== 48 passed, 3 skipped in 1754.60s (0:29:14) ==================

[bazarnov]

Nice work.

[artem1205]

/publish connector=connectors/source-shopify

🕑 Publishing the following connectors:
connectors/source-shopify
https://github.com/airbytehq/airbyte/actions/runs/3268171033

Connector	Did it publish?	Were definitions generated?
connectors/source-shopify	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[loris]

Minor changes are required. Please refer to the comments.

Also, let's try to cache the Products stream, since we have it re-used a lot for sub-sequent streams. To do so, simply add use_cache = True attribute to the Products stream class, the cache will be created and should be re-used for all other subsequent stream classes where possible (if possible), this will also reduce the sync time dramatically, I think. WDYT?

Hi, @bazarnov I am looking at the source-shopify connector performance lately, especially to speed up syncs of metafields. I am wondering why you only enabled cache for the products stream here? Did you leave others (like customers) without cache on purpose?

[bazarnov]

@loris My best guess here, we didn't realize the need.

@artem1205 are there any difficulties with caching the metadata-* related streams to improve the performance?

As for me, this will not add more speed to the connector, since these streams are not re-used but standalone based on the parent stream they need the ids from. But we can try to research this anyway.

Did you create the issue for this? @loris

[loris]

Hi @bazarnov. Thanks for the reply, I will create a new issue regarding Shopify metafields, but first I have some question you can answer to help me understand some Airbyte design choices to handle metafields with substreams.

Considering the way metafields are modelized and exposed in Shopify:

• You can only fetch the metafields of a single parent resource (ie, /admin/products/{product_id}/metafields.json)
• If you change the metafields of a parent resource (add, update or remove any metafields), then the parent resource document will have its updated_at property updated

I don't understand why metafields (and substream in general) are designed the way they are in Airbyte right now as separate standalone streams, with their own state, and which needs to use the parent stream query to fetch the resource IDs (hopefully, some queries can be cached, but it comes with memory challenge). My understanding, is that it was designed this way to make it possible in an Airbyte connection to sync only the substream (ie, the metafields) without syncing the parent stream (ie, the products), but I don't get why someone would want to do that.

It would be easier to consider metafields as dependant streams of the parent resource, meaning:

• that you cannot enable it if you have not enabled the parent resource stream
• that they don't have their own state, this is the parent stream which use its own state to fetch the updated parent resources (ie, /admin/products.json?updated_at_min={state}) then fetch the metafields for each parent resources and emit all the metafields returned by Shopify in the destination (this is only way to handle deletions of metafields)

WDYT? Are you aware of other Airbyte connectors which have these "required" streams (ie, you cannot enable stream A if you don't have stream B enabled)

[bazarnov]

@loris Regarding your question:

My understanding, is that it was designed this way to make it possible in an Airbyte connection to sync only the substream (ie, the metafields) without syncing the parent stream (ie, the products), but I don't get why someone would want to do that.

me either, but there are Customers who need just 1 or 2 streams fetched independently, but in a nutshell - the metafields are dependent on their parent streams even though it looks independent in the UI, the backend fetches the parent stream first, then the substream using the parrent_id, like you've described earlier here:

that they don't have their own state, this is the parent stream which use its own state to fetch the updated parent resources (ie, /admin/products.json?updated_at_min={state}) then fetch the metafields for each parent resources and emit all the metafields returned by Shopify in the destination (this is only way to handle deletions of metafields)

As for:

WDYT? Are you aware of other Airbyte connectors which have these "required" streams (ie, you cannot enable stream A if you don't have stream B enabled)

The Shopify connector is one among the few others that use this technique to fetch the streams explicitly independent providing the parent stream state and the substream state to fetch the substream based on the parent stream to provide the incremental functionality. As has already been said - they look independent in the UI but on the backend, they are synced based on the parent updates. If not, we will consider all the issues around the metafields streams during the planned maintenance we are already working on.

[loris]

@bazarnov Thanks for the detailed answer.
However I'm convinced the substream has its own state which is not the same as the parent one. I opened up an issue a while ago (#27355) where one of our shopify store had the following state at some point:

[
  {
    "streamDescriptor": {
      "name": "customers"
    },
    "streamState": {
      "updated_at": "2023-06-14T09:40:41+02:00"
    }
  },
  {
    "streamDescriptor": {
      "name": "metafield_customers"
    },
    "streamState": {
      "customers": {
        "updated_at": "2023-03-30T05:32:13+02:00"
      },
      "updated_at": "2021-09-01T17:57:41+02:00"
    }
  }
]

As you can see, the metafield_customers stream has its own states (updated_at for metafield_customers and updated_at for customers, that value being the most recent customer updated_at value found in the most recent metafield_customers value).And the bug in that case is that the sync will:

• correctly synchronize the customers stream (using updated_at_min=2023-06-14T09:40:41+02:00)
• but will use the 2023-03-30T05:32:13+02:00 timestamp to fetch all customer IDs, and then all metafields for those customers (instead of just using the 2023-06-14T09:40:41+02:00 timestamp from the parent stream)

[bazarnov]

@loris have you noticed that the metafields could be added and regardless the actual customer object for instance? like the metafields field "name" could be updated without the update for the actual customer id directly using API? which means there is a possibility someone will update the properties of the metafields instead of updating the actual record id. that's why the metafields have their "updated_at" because they are not stateless.
Making the direct dependency to the parent object state in this case could cause missing metafileds records for some users, if the parent record state would have a bigger value than the metafiled state.

[bazarnov]

I'll double-check the actual logic how the metafields streams work, once i work on this issue in a few days, thanks you for your findings

[loris]

@bazarnov Just did some tests on a shopify store we own to validate which state we should use:

Initial state

customer has an updated_at=2023-07-20T10:13:58+02:00 and no metafields

GET /admin/customers/7150840250556.json?fields=id,updated_at
{
    "customer": {
        "id": 7150840250556,
        "updated_at": "2023-07-20T10:13:58+02:00"
    }
}

GET /admin/customers/7150840250556/metafields.json?fields=id,created_at,updated_at
{
    "metafields": []
}

Adding a metafield

customer state got properly updated as we add a new metafield

POST /admin/customers/7150840250556/metafields.json {"metafield":{"namespace":"foo","key":"bar","type":"single_line_text_field","value":"baz"}}

GET /admin/customers/7150840250556.json?fields=id,updated_at
{
    "customer": {
        "id": 7150840250556,
        "updated_at": "2023-07-25T10:22:09+02:00"
    }
}

GET /admin/customers/7150840250556/metafields.json?fields=id,created_at,updated_at
{
    "metafields": [
        {
            "id": 37437136044220,
            "created_at": "2023-07-25T10:22:09+02:00",
            "updated_at": "2023-07-25T10:22:09+02:00"
        }
    ]
}

Updating a metafield

customer state got properly updated as we update an existing metafield

PUT /admin/customers/7150840250556/metafields/37437136044220.json {"metafield":{"value":"baz2"}}

GET /admin/customers/7150840250556.json?fields=id,updated_at
{
    "customer": {
        "id": 7150840250556,
        "updated_at": "2023-07-25T10:25:58+02:00"
    }
}

GET /admin/customers/7150840250556/metafields.json?fields=id,created_at,updated_at
{
    "metafields": [
        {
            "id": 37437136044220,
            "created_at": "2023-07-25T10:22:09+02:00",
            "updated_at": "2023-07-25T10:25:58+02:00"
        }
    ]
}

Deleting a metafield

customer state got also updated as we delete an existing metafield

DELETE /admin/customers/7150840250556/metafields/37437136044220.json

GET /admin/customers/7150840250556.json?fields=id,updated_at
{
    "customer": {
        "id": 7150840250556,
        "updated_at": "2023-07-25T10:27:27+02:00"
    }
}

GET /admin/customers/7150840250556/metafields.json?fields=id,created_at,updated_at
{
    "metafields": []
}

As you can see, the only proper way to synchronize the metafields of a Shopify resource (and handle all operations: creation, modification and deletion of metafields) is to do a full refresh of the metafields of a given parent resource when the parent resource has its updated_at value changed. That's why I proposed (but not sure how it goes in the opposite direction of how Airbyte handles child stream) to sync metafields dependently of the parent streams sync: we only store the updated_at state of the parent stream, fetch updated parent resource, and for each fetched parent records fetch the associated metafields (no need for any metafields state). WDYT?

[bazarnov]

@loris
I've just double-checked the connector's logic, against this statement:

to sync metafields dependently of the parent streams sync: we only store the updated_at state of the parent stream, fetch updated parent resource, and for each fetched parent records fetch the associated metafields (no need for any metafields state). WDYT?

It turns out that we're indeed doing like you propose:

We sync the parent incrementally and get a few records, then we pass the ids we fetched for parent to fetch metafields from the acquired ids from parent (Incremental parent to Incremental substream) if state is available.

See the logic here: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-shopify/source_shopify/source.py#L257

This method produces the slices with ids, let's say we use metafields Customers stream for instance here.

The ids are taken and passed to fetch all metafields from the incrementally fetched customers ids, which are updated after the state value.

Initial sync sample:

Parent stream

Customer slice example: {'id': 123, 'updated_at': '2023-04-24T06:53:48-07:00'}

Substream:

// Metafields Customer

[
  {
        "id": 1111,
        "namespace": "custom",
        "value": "Test\n",
        "description": null,
        "owner_id": 123,
        "created_at": "2023-04-13T04:50:10-07:00",
        "updated_at": "2023-04-13T04:50:10-07:00",
        ....
  },
 {
        "id": 222,
        "namespace": "custom",
        "value": "Test15",
        "description": null,
        "owner_id": 123,
        "created_at": "2023-04-13T04:50:10-07:00",
        "updated_at": "2023-04-14T05:31:11-07:00",
        ....
   }
]

State

the state, in this case, for the metafield_customers stream would be:

{
   "state": {
            "metafield_customers": {
                "updated_at": "2023-04-14T05:31:11-07:00",
                "customers": {
                    "updated_at": "2023-04-24T06:53:48-07:00"
                }
            }
        }
  }

Because the parent record was updated and the substream for this parent record - no. It was updated days ago manually, after this the parent record was updated with the other information, non-related to the metafields.
There are 2 records for our sandbox listed above for this parent one, because it was added at the same time twice.

Now, when we run the subsequent sync for the same stream metafields_customer we now have the latest state available:

Subsequent Incremental sync

Parent stream

Customer slice example: {'id': 123, 'updated_at': '2023-04-24T06:53:48-07:00'}

The updated_at was not changed because it's a static value for this test.

We fetch the same customer again but we expect a different number of values, smaller ones.

Substream:

{
        "id": 222,
        "namespace": "custom",
        "value": "Test15",
        "description": null,
        "owner_id": 123,
        "created_at": "2023-04-13T04:50:10-07:00",
        "updated_at": "2023-04-14T05:31:11-07:00",
        ....
   }

State

{
   "state": {
            "metafield_customers": {
                "updated_at": "2023-04-14T05:31:11-07:00",
                "customers": {
                    "updated_at": "2023-04-24T06:53:48-07:00"
                }
            }
        }
  }

The reason the metafields are designed like this, and I totally agree with what you've said earlier - is because the parent object could be updated with non-metafields information, leaving the related metafields behind, if they are not explicitly updated manually or in another way.

if we fetch the records like you propose, this record will be omitted for the substream leading us to the lost records and lots of issues in the future, we'd like to avoid unnecessary complications by making this logic transparent and understandable for the end-user.

[loris]

@bazarnov Thanks for the investigation, but how the current design (in code and that you described) can handle deletions of metafields?

Is this a known issue / limitations that Airbyte will not propagate deletions of metafields (and thus we should warn customers and give them workarounds, like doing full refresh)? Or did I missed something?

I still do understand the only way to handle deletions is to do a full refresh (aka, fetch all metafields for a given parent, and emit them all to the destination) for any parent we get to sync

As you can see, the only proper way to synchronize the metafields of a Shopify resource (and handle all operations: creation, modification and deletion of metafields) is to do a full refresh of the metafields of a given parent resource when the parent resource has its updated_at value changed.