PDE-5309 feat(core): add support for "buffered creates"

[eliangcs]

Introducing "Buffered Create Actions"...

A Buffered Create allows you to create objects in bulk with a single or fewer API request(s). This is useful when you want to reduce the number of requests made to your server. When enabled, Zapier holds the data until the buffer reaches a size limit or a certain time has passed, then sends the buffered data using the performBuffer function you define.

To implement a Buffered Create, you define a buffer configuration object and a performBuffer function in the operation object. In the buffer config object, you specify how you want to group the buffered data using the groupedBy setting and the maximum number of items to buffer using the limit setting.

Here's an example:

const performBuffer = async (z, bufferedBundle) => {
  // Send a request to create tasks in bulk
  const response = await z.request({
    method: 'POST',
    url: 'https://httpbin.zapier-tooling.com/post',
    body: {
      items: bufferedBundle.buffer,
    },
  });

  const items = response.data.json.items;
  const result = {};

  for (const item of items) {
    const idempotencyId = item.meta.id;
    const itemProps = item.inputData;
    result[idempotencyId] = {
      outputData: {
        id: Date.now(),
        ...itemProps,
      },
    };
  }

  return result;
};

module.exports = {
  key: 'task_buffered',
  noun: 'Task (Buffered)',

  display: {
    label: 'Create Task (Buffered)',
    description: 'Creates a task.',
  },

  operation: {
    performBuffer,
    buffer: {
      groupedBy: ['project_id'],
      limit: 100,
    },

    inputFields: [
      { key: 'project_id', required: true },
      { key: 'title', required: true },
      { key: 'content' },
    ],

    sample: {
      id: 1,
      name: 'Test',
    },

    outputFields: [],
  },
};

[standielpls]

invocation typo

[standielpls]

We seldom use the word write/writing on the schema, I want to double check if this is the intended word to use

[eliangcs]

Good call! I've replaced "write" with "create" in several places I found.

[standielpls]

Is this still relevant if we introduced this?

[eliangcs]

It was written before we added supported for large response payloads. I think for response payloads, it's no longer relevant. But for request payloads, it's still relevant, so I think we should keep it.

[standielpls]

Do these usually include a happy path? 😅

[eliangcs]

Sorry I'm not sure what you mean. Can you clarify?

[standielpls]

Ignore me, I thought I only saw test cases for all the various error paths, but didn't see a happy success path, but I see it now.

[standielpls]

I reviewed bc it was referenced in the backend MR, but noticed this was still in draft, so did a quick pass through, lmk when this is ready!

[standielpls]

Changes look great, thank you.
I wasn't able to test this on an integration (found issues upstream, likely an environment thing) but it looks like you tested at length in the demo, so I'm happy with that.

[eliangcs]

@standielpls thanks for the review! I added some text mentioning the feature is internal-only. Can you re-review?

[standielpls]

I'll approve to unblock, just found a couple typos

[standielpls]

Found a typo here for internal

[standielpls]

internal typo