[BUG] Worker calls Get method on Webapi plugin long after it's completed

[csenel]

Describe the bug
We have observed that Get method on webapi plugin is called by worker long after it's marked as succeeded on Flyte UI. For the particular case, we are using bigquery plugin and added logging to debug a separate issue when observed this behavior.

Restarting the flytepropeller stops the calls made by worker. Although we haven't observed when the calls would be stopped on their own, in one case they happened 10 hours after the task is successfully completed before the restart.

Expected behavior
Worker should remove task from the list once the task is succeeded and we shouldn't observe Get calls afterwards.

Screenshots
Please find attached the logs we have added with timestamps after 3 hours the task is successfully completed.

[welcome]

Thank you for opening your first issue here! 🛠