Can't reuse QueryCommand in loop

[clintfoster]

Describe the bug
This is either a bug or a documentation / usability issue. When attempting to reuse a Dynamo QueryCommand in a typical paging loop I get a "Duplicate middleware" exception:

Loop iteration: 1
Loop iteration: 2
12-05T17:40:45, t2: 2021-01-04T17:40:45 Error: Duplicate middleware name 'deserializerMiddleware'
    at Object.add (MiddlewareStack.js:122)
    at Object.applyToStack (serdePlugin.js:16)
    at Object.use (MiddlewareStack.js:141)
    at QueryCommand.resolveMiddleware (QueryCommand.js:80)
    at DynamoDBClient.Client.send (client.js:10)

SDK version number
3.1.0

Is the issue in the browser/Node.js/ReactNative?
Browser

Details of the browser/Node.js/ReactNative version
Paste output of npx envinfo --browsers

Chrome: 87.0.4280.88

To Reproduce (observed behavior)
Below is a typical loop to scan the sort key of an item with partition key fooId = 123:

const dynamoClient = new DynamoDBClient();
const qParams: Record<string, any> = {
  TableName: 'FooData',
  KeyConditionExpression: 'fooId = :fid',
  ExpressionAttributeValues: {':fid': {N: '123'}}
};

let qCmd = new QueryCommand(qParams);  // shouldn't it be possible for this to be const?
let i = 0;
do {
  console.log('Loop iteration:', ++i);
  // qCmd = new QueryCommand(qParams);  // enabling this line avoids the problem
  const qOutput = await dynamoClient.send(qCmd) as QueryOutput;
  qParams.ExclusiveStartKey = qOutput.LastEvaluatedKey;
} while(qParams.ExclusiveStartKey)

My specific question is why QueryCommand must be recreated each time through the loop to avoid the error. Wouldn't it be natural to reuse it, changing only the ExclusiveStartKey as needed?

I guess this is kind of a nit, and what I'm really looking for is the most efficient and idiomatic way to implement a standard paging loop in the v3 API. An example in the documentation would go a long way.

Expected behavior
It should be possible to express a simple paged query (or scan) without inefficient or confusing constructs.

[trivikr]

AWS SDK for JavaScript v3 supports Pagination using Async Iterators.

Can you try paginateQuery operation?

[AllanZhengYP]

The root cause for this issue seems to be duplicated with #1857. We will look further into this.

[AllanZhengYP]

Hi @clintfoster,

Just like it mentioned by @trivikr, you can still work around it by using Pagination using Async Iterators. The SDK handles making consecutive requests for you.

[clintfoster]

@trivikr @AllanZhengYP Thanks for the pointer to async iterators. I tried a trivial paged query, and it was clean and simple. However, I need the ability to exit the pagination loop and later re-enter it where I left off. I think this should be possible by saving QueryOutput.LastEvaluatedKey at the point where I exit the loop. When I need to restart I can set PaginationConfiguration.startingToken and run paginateQuery() again. However, startingToken is a string and LastEvaluatedKey is of type [key: string]: AttributeValue. No doubt there is a way to convert it to the needed string format. But overall the code was getting more complicated than simply using ExclusiveStartKey and LastEvaluatedKey in a less magical do/while loop.

As a temporary workaround I will stick with creating a new QueryCommand each time through the loop unless you advise switching to async iterators for other reasons.

[AllanZhengYP]

Hey @clintfoster,

However, startingToken is a string and LastEvaluatedKey is of type [key: string]: AttributeValue. No doubt there is a way to convert it to the needed string format.

Looks like the startingToken is an any type for now. In fact we can make it more specific using generic. Looking at the implementation, you are able to just supply LastEvaluatedKey to startingToken or input. ExclusiveStartKey.

But overall the code was getting more complicated than simply using ExclusiveStartKey and LastEvaluatedKey in a less magical do/while loop.

You can exist the async iterator and resume it easily in my opinion too, It would look like:

  const iter = paginateScan({client: dynamoClient}, params);
  let res: ScanCommandOutput
  let isEnd = false;
  while(isEnd) {
    const {value, done} = await iter.next();
    isEnd = done;
    res = value;
    if (value.Items[0] === {} /*when you want to exit the pagination*/) {
      break; //exit
    }
  }
  //do something
  //resume paginating; using for loop this time.
  for await (const result of paginateScan(
    {client: dynamoClient},
    {...qParams, ExclusiveStartKey: res.LastEvaluatedKey}
  )) {
    //do something
  }

[AllanZhengYP]

I opened a feature request for adding more types in paginator.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.