Idempotency: Single DynamoDB table for multiple AWS Lambda functions

[Dunedan]

First of all, thanks for another great release. 👍

I got a feature request for the new idempotency feature: As I understand the documentation, the key_attr of the DynamoDB table consists solely of the hashed idempotency key, which is based on the payload. That makes it hard to use a single DynamoDB table for multiple different AWS Lambda functions which expect the same payloads. I guess it'd be possible to somehow ensure that there is some kind of static identifier in event_key_jmespath, but that feels cumbersome and error prone.

It'd be great to have the ability to have something like the function name included in the key_attr so it's possible for multiple AWS Lambda functions to share the same DynamoDB table to avoid having to create a new table for each AWS Lambda function.

[heitorlessa]

Hey @Dunedan - Thanks for raising this feature request, and it's also awesome to hear about the new release.

Could you share an example where different Lambda functions would handle the same payload and idempotence key?

I can see that when you're using an orchestration layer like Step Functions, where the same state could be sent to multiple functions. For example, a booking process where the idempotency key, event_key_jmespath, would be order_id and customer_id the same for multiple functions - Problem here is that the return differs because one might be used for reserve booking, other for reserve payment, and while you could have a slight different idempotence key it could be error prone.

Does that capture the challenge you had in mind?

[Dunedan]

Could you share an example where different Lambda functions would handle the same payload and idempotence key?

The example I had in mind have been multiple AWS Lambda functions triggered by the same SNS topic. However I now believe the payload would be slightly different in this case for each AWS Lambda function, as at least the EventSubscriptionArn key should have a different value per invoked AWS Lambda function.

I can see that when you're using an orchestration layer like Step Functions, where the same state could be sent to multiple functions. For example, a booking process where the idempotency key, event_key_jmespath, would be order_id and customer_id the same for multiple functions - Problem here is that the return differs because one might be used for reserve booking, other for reserve payment, and while you could have a slight different idempotence key it could be error prone.

Does that capture the challenge you had in mind?

Yes, that'd be a similar situation to the one I had in mind with SNS.

I believe in practice the chance to have multiple AWS Lambda functions with the same payload and idempotence key is very low, but nevertheless as far as I can see, there is nothing in the code to prevent unexpected behavior in case it does happen. That makes me a bit uncomfortable using the same DynamoDB table for multiple functions, as I have to remember to explicitly check each AWS Lambda function for possible collisions of the payload and idempotence key.

[heitorlessa]

Perfect, that makes sense - We'll ping you back here once we have an initial implementation to review :)

[michaelbrewer]

@heitorlessa @Dunedan I drafted a PR for this.

[heitorlessa]

it's now merged @Dunedan - Set a release date for 1.12 with this and other minor enhancements like correlation_id to be launched next Wed

[Dunedan]

Awesome. Thanks a lot. 👍

[michaelbrewer]

Awesome. Thanks a lot. 👍

thanks for bringing it up, it was a good catch

[heitorlessa]

Hey @Dunedan this is now available in 1.12.0 ;) Give it a try, and let us know if there's anything else you need before we prepare to go GA in one or two releases: https://github.com/awslabs/aws-lambda-powertools-python/releases/tag/v1.12.0

Thanks again for the idea!