Targets of event rules are duplicated if delete permission is missing

[cloetzi]

I am experimenting with minimal permissions for the IAM user used by zappa. If the permission for deleting a target/rule is missing, the targets in a schedule are duplicated without that an error is raised. This means, the lambda function gets triggered multiple times simultaneously.

The issue can be reproduced by restricting the event permissions of the Zappa AWS user to the policy below and run zappa update.

{
  "Action": [
    "events:PutRule",
    "events:PutTargets",
    "events:ListRules"
  ],
  "Resource": "*",
  "Effect": "Allow"
}

[Miserlou]

Damn, that's a great bug.

Can you confirm that the function is actually executed multiple times concurrently?

I guess the issue is that we should raise a warning when this occurs, or do you expect another behavior?

[cloetzi]

Yes, it seems that it executes concurrently. See Cloudwatch graph below from the Lambda execution, at the end I had 5 targets running every minute.

I'd expect to see the AWS error, for example if you remove the ListRules permission, Zappa nicely tells you that this permission is missing. For the delete targets, this error must be swallowed somehow. I had a brief look in the code but I could not find where it actually happens, will have another look now.

[Miserlou]

Ah, dang. Okay, good bug report. This code got messed with recently. Is this against 0.21.0 or master head?

[cloetzi]

It is against master head.

[cloetzi]

Found the issue: https://github.com/Miserlou/Zappa/blob/master/zappa/zappa.py#L1275
I was missing the permission ListTargetsByRule, this error is swallowed and skips the actual delete call.
A possible solution could be to check if the ClientError is an AccessDeniedException and if so, raise it.

EDIT: Added missing words.

[Miserlou]

Excellent sleuthing! Sounds like a good solution to me if you can properly type damn boto3 errors, want to send a PR? 😄

[cloetzi]

Working on it.