Feature proposal - Fetch paginated results automatically

[Jeremie-Kiwik]

Hello,

when calling results with pagination, we only receive the first fragment, and must call the API for each subsequent paginated result.
It's not immediately apparent that the results we have are truncated.

So I propose to alter the Response object to detect if there is a pagination, and if yes, fetch the remaining results. However, this would be a breaking change.

So:

• In your opinion, is this a good idea, or should the responsibility lie with the software using your client rather than the client itself?
• If it is a breaking change, could this be implemented as an option in the Config value, for example?

I would appreciate your opinion. If everyone agrees, I can try to make the PR :)

[tgeorgel]

I am not really pleased by the proposition, as it means you'll query the API multiples times until there are no more pages available, completely breaking the whole essence of pagination.

The Response object actually contains tools to work with the Pagination :

$response = $api->index(...);
$pagination = $response->pagination();

while ($pagination->total > ($pagination->offset + $pagination->count)) {
   // more pages available...
}

You could iterate over all the results this way :

$api = new CompaniesApi();
$companies = new EntityCollection();

$perPage = 20;
$offset = 0;

do {
    $response = $api->index([ 'limit' => $perPage, 'offset' => $offset ]);
    $pagination = $response->pagination();

    $companies = $companies->merge(
        $responses->entities()
    );

    $offset += $pagination->count;
    $hasMoreResults = $pagination->total > ($pagination->offset + $pagination->count);
} while ($hasMoreResults);

I would rather add some helpers on the Pagination object if it can improve developer experience :

$pagination->hasMorePages(): bool;
$pagination->firstPage(): bool;
$pagination->lastPage(): bool;
$pagination->current(): int;
....

What do you think ?

[Jeremie-Kiwik]

OK, so it is more the role of the app than the client, I understand.
I'll use your snippet, it looks great.

It's just that this piece of code would be duplicated many times in big projects, because you will always need to fetch many paginated data.

For now, I'l make a helper function on my own code with the snippet, and it will do the job.
Thanks!!

[Jeremie-Kiwik]

To keep somewhere if you add a helper someday: example of paginated results fetching
(Needs to be improved: the code is duplicated, it's ugly. But it does the job)

EntityCollection

<?php

namespace App\Helper\SellsyClient;

use Bluerock\Sellsy\Entities\Entity;
use Bluerock\Sellsy\Contracts\EntityCollectionContract;
use Spatie\DataTransferObject\DataTransferObjectCollection;


class EntityCollection extends DataTransferObjectCollection implements EntityCollectionContract
{
	public static function create(array $data): EntityCollection
	{
		return new static(Entity::arrayOf($data));
	}

	public function merge(EntityCollectionContract $collection): EntityCollection
	{
		foreach($collection as $entity) {
			$this->iterator->append($entity);
		}
		return $this;
	}
}

SellsyV2Helper

<?php

namespace App\Helper;

class SellsyV2Helper
{
	/**
	 * Does index() on an API, then loop to fetch all paginated results
	 * @param \Bluerock\Sellsy\Api\AbstractApi $api
	 * @param array $query					  $query
	 * @param int                              $perPage
	 * @return SellsyClient\EntityCollection
	 */
	static function indexPaginated(
		\Bluerock\Sellsy\Api\AbstractApi $api,
		array $query = [],
		int $perPage = 25
	):SellsyClient\EntityCollection  {

		$collection = new SellsyClient\EntityCollection();
		$offset = 0;
		do {
			$query['limit'] = $perPage;
			$query['offset'] = $offset;
			$response = $api->index($query);
			$pagination = $response->pagination();

			// Add results to collection
			$collection = $collection->merge(
				$response->entities()
			);

			// No pagination? We already have everything. Bye
			if (null === $pagination)
				break;

			// If yes, call again to get next data
			$offset += $pagination->count;
			$hasMoreResults = $pagination->total > ($pagination->offset + $pagination->count);
			# echo "INDEX offset = $offset, more = $hasMoreResults\n";
		} while ($hasMoreResults);
		return $collection;
	}


	/**
	 * Does search() on an API, then loop to fetch all paginated results
	 * @param \Bluerock\Sellsy\Api\AbstractApi $api
	 * array $query							   $query
	 * array $filters						   $filters
	 * @param int                              $perPage
	 * @return SellsyClient\EntityCollection
	 */
	static function searchPaginated(
		\Bluerock\Sellsy\Api\AbstractApi $api,
		array $query = [],
		array $filters = [],
		int $perPage = 25
	):SellsyClient\EntityCollection  {

		$collection = new SellsyClient\EntityCollection();
		$offset = 0;
		do {
			$query['limit'] = $perPage;
			$query['offset'] = $offset;
			$response = $api->search($query, $filters);
			$pagination = $response->pagination();

			// Add results to collection
			$collection = $collection->merge(
				$response->entities()
			);

			// No pagination? We already have everything. Bye
			if (null === $pagination)
				break;

			// If yes, call again to get next data
			$offset += $pagination->count;
			$hasMoreResults = $pagination->total > ($pagination->offset + $pagination->count);
			# echo "SEARCH offset = $offset, more = $hasMoreResults\n";
		} while ($hasMoreResults);
		return $collection;
	}
}

Usage example:

$cfApi = new CustomFieldsApi();
$response = SellsyV2Helper::indexPaginated($cfApi);
echo "TOTAL = ", count($response->items()), PHP_EOL;