Description:

Some cohorts have enough members that the imported Airbyte connector OOMs as it paginates through them. This commit makes it so the connector flushes its buffer of cohort_members records after every 100 records are read.

cohort_members behaves like a full refresh stream, but Airbyte's incremental tools are used to force records to be flushed periodically, helping prevent OOMs on these large cohorts. cohort_members was moved over to use the newer state property for state management to pre-position for future improvements.

Potential future improvements include:

• Making cohort_members incremental by updating the state to contain cursors for each unique cohort.
• Aligning engage with the changes to cohort_members (assuming these changes work well).

Workflow steps:

(How does one use this feature, and how has it changed)

Documentation links affected:

(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)

Notes for reviewers:

Tested on a local stack. Confirmed:

• cohort_members acts like a full refresh stream
• cohort_members checkpoints (i.e. flushes) records when paginating through a large cohort's members.
• Memory % was typically ~30% or less while the connector paginated through results. Due to frequent schema inference updates causing connector restart & Mixpanel's 1 request/minute rate limit, I could only confirm this when paginating up through 6 pages.

Although not entirely necessary, I think all existing tasks should have their cohort_members bindings backfilled to avoid any funky behavior related to state management.

This change is