[BUG] flytepropeller gracefully cleanup CRDs missing execution

[ckiosidis]

Describe the bug

When an execution does not exist in flyteadmin but the CRD has been created, Flytepropeller continuously tries to reconcile the flyteworkflow CRD but fails.

example of resource status

  failedAttempts: 15086
  message: 'Workflow[] failed. ErrorRecordingError: failed to publish event, caused
    by: ExecutionNotFound: The execution that the event belongs to does not exist,
    caused by [rpc error: code = NotFound desc = missing entity of type execution
    with identifier project:"flyte-canary" domain:"production" name:"tvy2szcxi7bcnzg36oef"
    ]'
  phase: 0

https://flyte-org.slack.com/archives/C01P3B761A6/p1647350804430329

Expected behavior

Flytepropeller should gracefully fail and clean up the crd of the non-existent execution (at least after exhausting the retries)

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes