[10.x] Allow placing a batch on a chain

[khepin]

Context

Where do we use this:

This is a use case we encounter in a couple places at Square where we have a serial process of jobs that need to be handled and one or more of the steps either should be worked on in parallel or are of unknown length when initially triggering the workflow and will create additional jobs, but we need to know when this is finished to ensure we keep the chain going.

Example workflow

An example could be onboarding users from a different, separate system. The steps to do so could be the following:

1. Create user account and base metadata
2. Pull user account additional information / status from remote systems
3. Download e-commerce catalog from service A
    ^ many paginated requests, good use case for Batch.
4. Notify service B or current setup status
5. Precompute indexing data based on data downloaded in step 3
    ^ also a perfect candidate for batching
6. Notify user of account readiness

How it would look like with Laravel today

There is currently not a great way to express something like this in Laravel. The closest we can get to is something like:

// Start with a chain of multiple steps
Bus::chain([
    new AccountInitialize(),
    new GetAccountMetadata(),
    /**
     * This step would make paginated requests to a remote service.
     * Each job would query 1 page and then if given a cursor to a next page,
     * dispatch a job to do the same thing on that next page.
     * 
     * So we're creating a job to start a batch since a batch can dynamically
     * receive new jobs and still track the completion of the entire process. 
     */
    new StartDownloadEcommerceData(), 
])->dispatch();

class StartDownloadEcommerceData extends BaseJob
{
    public function handle()
    {
        Bus::batch([new DownloadEcommerceDataPage(page: 0)]) // if there are more pages, the first job would add another to the batch
            ->then(function () {
                Bus::chain([
                    new NotifyServiceBStatusUpdate(),
                    new StartPrecomputingDownloadedData(),
                ])->dispatch();
            });
    }
}

class StartPrecomputingDownloadedData extends BaseJob
{
    public function handle()
    {
        $b = Bus::batch();
        foreach ($data as $d) {
            $b->add(new PrecomputeData($d));
        }

        $b->then(function () {
            dispatch(new NotifyUserAccountReady());
        });

        $b->dispatch();
    }
}

Proposal

The previous example takes away a lot of clarity from the workflow with different pieces being started / kicked off in multiple places in code. What we would prefer instead is to write something like:

Bus::chain([
    new AccountInitialize(),
    new GetAccountMetadata(),
    Bus::batch([new DownloadEcommerceDataPage(page: 0)]), // if there are more pages, the first job would add another to the batch
    new NotifyServiceBStatusUpdate(),
    Bus::batch([new StartPrecomputingDownloadedData()]), // first job will load the data and add 1 job / item to the batch
    new NotifyUserAccountReady(),
])->dispatch();

This allows us to see in one single place all the actions that will happen as part of the entire workflow. That also makes maintenance much simpler as we don't need to track across many files the downstream dependencies and effects of a given job. Everything is laid out clearly in one central place.

The PR

This PR implements a way to achieve the proposed syntax above. It also allows nesting chains and batches to even deeper levels if needed.

Would love to see if this would make a good addition to Laravel. This is definitely something we'll be using on our side.

[themsaid]

Nice one.

One comment I have is that attaching the chain continuity to the batch's then callbacks means that a batch that allows failure will break the chain even though failure is allowed.

I suggest switching to a finally callback and then check if the batch is cancelled. If so, the chain is discarded, otherwise the chain continues even if the batch has failures.

[khepin]

@themsaid I like that. Updated accordingly, thanks for noticing that.