Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

collect metrics in all operations #1152

Open
roeap opened this issue Feb 17, 2023 · 4 comments
Open

collect metrics in all operations #1152

roeap opened this issue Feb 17, 2023 · 4 comments
Assignees
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@roeap
Copy link
Collaborator

roeap commented Feb 17, 2023

Description

Some operations e.g. vacuum and optimize already collect metrics that are added to the commit history. Other operations do not yet collect these metrics. We should collect the documented metrics on all operations according to the reference.

Use Case

Related Issue(s)

@rtyler
Copy link
Member

rtyler commented Jan 6, 2024

I believe this may be covered by #2032 which @ion-elgreco has proposed

@ion-elgreco
Copy link
Collaborator

Actually this one is partially covered by this one: #2009

I will cross check which ones we still don't track metrics where we should though!

@ion-elgreco
Copy link
Collaborator

Current state:

  • Write: doesn't collect any metrics
  • Update: rewrite_ms is missing
  • FSCK: we give the removed_file_paths, but spark-delta uses num_removed_files @rtyler shall we change this to simply the count of the vec here?
  • Convert: we don't write the correct metrics here since we create operation underneath
  • Optimize: we collect metrics but totally different ones

The ones I didn't mentioned we collect the same metrics for.

@ion-elgreco ion-elgreco added the help wanted Extra attention is needed label Aug 19, 2024
@ion-elgreco ion-elgreco removed their assignment Aug 19, 2024
@gavinmead
Copy link
Contributor

take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

4 participants