[BUG] Fix or enhance failure to catalog errors in propeller

[wild-endeavor]

Describe the bug
Sometimes, we'll see this log line in propeller

Error putting to catalog [failed to create dataset for ID resource_type:TASK project:"asdf" domain:"production" name:"app.workflows.asdfasdf" version:"fdsafsdf" : rpc error: code = InvalidArgument desc = missing artifactData]

(See the internal link in our on-call notes for an example execution.)

Note that the execution succeeded. In fact, there is no indication that it did not cache.

We should

1. Figure out why this error is happening, and fix it if it's something broken on our end.
2. If not, then we should return an error to the user, possibly failing the workflow.
3. Even if we don't fail the entire workflow, we should somehow make it clear to the user that it failed to cache.

Expected behavior
See above.

Flyte component

• Overall
• Flyte Setup and Installation scripts
• Flyte Documentation
• Flyte communication (slack/email etc)
• FlytePropeller
• FlyteIDL (Flyte specification language)
• Flytekit (Python SDK)
• FlyteAdmin (Control Plane service)
• FlytePlugins
• DataCatalog
• FlyteStdlib (common libraries)
• FlyteConsole (UI)
• Other

To Reproduce
Not sure.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment
Flyte component

• Sandbox (local or on one machine)
• Cloud hosted
  • AWS
  • GCP
  • Azure
• Baremetal
• Other

[EngHabu]

I think our window of investigation might have closed... will investigate further if it comes up again..