Add spark.stage details attribute at the end of the stage

[paul-laffon-dd]

What Does This Do

Adds the spark.stage details attribute at the end of the stage, rather than at the beginning.

Motivation

The details attribute contains a large amount of data, including the full stack trace that initiated the stage. When using long-running spans, the span is flushed multiple times, which can significantly increase the ingestion volume.

By adding the details attribute at the end of the stage, we reduce the ingestion volume while still ensuring the information is available once the stage has completed.

Additional Notes

Contributor Checklist

• Format the title according the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any usefull labels
• Squash your commits prior merging or merge using GitHub's Squash and merge
• Don't use close, fix or any linking keywords when referencing an issue.
  Use solves instead, and assign the PR milestone to the issue
• Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]