'configservice get-resource-config-history' really slow

[adamchainz]

$ time aws configservice get-resource-config-history --resource-type AWS::EC2::Instance --resource-id i-redacted
<output redacted>
aws configservice get-resource-config-history --resource-type  --resource-id   0.61s user 0.08s system 4% cpu 14.075 total

but it's not the service, as in ipython it takes milliseconds:

In [1]: import boto3

In [2]: %timeit boto3.client('config').get_resource_config_history(resourceType='AWS::EC2::Instance', resourceId='i-redacted')
1 loop, best of 3: 122 ms per loop

I reran with the same instance repeatedly in case there's caching in the service too, but it doesn't appear so.

It's only this one command in configservice I've found slow, other things work at normal speed, e.g.

$ time aws s3api list-buckets
<output redacted>
aws s3api list-buckets  0.30s user 0.07s system 59% cpu 0.630 total
$ time aws configservice describe-delivery-channels                       ✭
<output redacted>
aws configservice describe-delivery-channels  0.29s user 0.07s system 74% cpu 0.482 total

Using latest CLI, boto3, botocore, etc.

[adamchainz]

I did some more tracing and it's down to the pagination, doing this is equally as slow:

In [20]: list(paginator.paginate(resourceType='AWS::EC2::Instance', resourceId='i-00835a2530fc90155'))

Config pagination returns a nextToken that then gets used over and over by botocore's paginator whilst returning the same nextToken and an empty configurationItems list. Maybe this is a bug in the service itself, or in botocore? Not sure.

[adamchainz]

Sorry it's not the same nextToken, it does change, I just skimmed a few characters :)

[dstufft]

As you noticed, this is just going to come down to the fact that the CLI fetches all of the results using pagination while your initial snippet is only going to fetch the first page of results. You can control how the CLI does pagination using the pagination options.

[adamchainz]

But isn't there a bug somewhere here? The service is sending hundreds of empty pages.

[dstufft]

Ah, that's what I get for responding to tickets too early in the morning, I missed that the response was empty. Can you include a redacted debug log (add --debug to the aws cli output)?

[adamchainz]

Voila debug.log

154 pages fetched, only the first one had any data :)

[stealthycoin]

It looks like if you do not provide time bounds with (--earlier-time and --later-time) it will have this behavior based on the docs associated with --earlier-time:

       --earlier-time (timestamp)
          The time stamp that indicates an earlier time. If not specified, the
          action  returns  paginated  results that contain configuration items
          that start from when the first configuration item was recorded.

I don't think there is anything we can do from the CLI's perspective since we just paginate through whatever they give us. You can disable pagination by using --no-paginate and you can manually use the pagination tokens that are provided by the raw api or try out using the --earlier-time argument and see if it helps. You could also try contacting the service team on the forums or through customer support

[adamchainz]

OK thanks for looking into it. I'll contact support.

[adamchainz]

Btw I've reported this to support and it's being sent through to the Config team.

[khbvirus]

Hello,
I'm still facing the issue of getting the nextToken in the response with empty configurationItems !
So if you handle the nextToken properly you will a a lot of API calls and takes time for each resource.