Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: expose log writer bytes timeseries metric #104531

Closed
jbowens opened this issue Jun 7, 2023 · 0 comments · Fixed by #104890
Closed

storage: expose log writer bytes timeseries metric #104531

jbowens opened this issue Jun 7, 2023 · 0 comments · Fixed by #104890
Assignees
@raggar
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) E-quick-win Likely to be a quick win for someone experienced. T-storage Storage Team

Comments

@jbowens
Copy link
Collaborator

jbowens commented Jun 7, 2023 •
edited by cockroach-jira-scripts
Loading

I believe the volume of data written to the storage engine WAL is the last missing piece to be able to compute write amplification over an arbitrary interval. We already have existing metrics for flushed bytes, ingested bytes and compacted bytes.

Jira issue: CRDB-28589
@jbowens jbowens added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-storage Relating to our storage engine (Pebble) on-disk storage. T-storage Storage Team labels Jun 7, 2023
@jbowens jbowens added the E-quick-win Likely to be a quick win for someone experienced. label Jun 12, 2023
@raggar raggar self-assigned this Jun 13, 2023
raggar added a commit to raggar/cockroach that referenced this issue Jun 14, 2023
@raggar
Expose WAL.BytesWritten
e68a36c 
This change exposes the `WAL.BytesWritten` metric from Pebble to the
Cockroach Side.

Fixes: cockroachdb#104531
Release note: None
@raggar raggar mentioned this issue Jun 14, 2023
Merged
raggar added a commit to raggar/cockroach that referenced this issue Jun 14, 2023
@raggar
kvserver: Expose WAL logical and total bytes written
2b1fe00 
This change exposes the `WAL.BytesWritten` and `WAL.BytesIn` metric from
Pebble to the Cockroach Side.

Fixes: cockroachdb#104531
Release note: None
raggar added a commit to raggar/cockroach that referenced this issue Jun 14, 2023
@raggar
kvserver: Expose WAL logical and total bytes written
1ba4675 
This change exposes the `WAL.BytesWritten` and `WAL.BytesIn` metric from
Pebble to the Cockroach Side.

Fixes: cockroachdb#104531
Release note: None
raggar added a commit to raggar/cockroach that referenced this issue Jun 15, 2023
@raggar
kvserver: Expose WAL logical and total bytes written
c2ca0c2 
This change exposes the `WAL.BytesWritten` and `WAL.BytesIn` metric from
Pebble to the Cockroach Side.

Fixes: cockroachdb#104531
Release note: None
craig bot pushed a commit that referenced this issue Jun 16, 2023
@raggar
Merge #104890
6e641ff 
104890: kvserver: Expose Bytes Written to WAL r=RahulAggarwal1016 a=RahulAggarwal1016

This change exposes the `WAL.BytesWritten` `WAL.BytesIn` metric from Pebble to the Cockroach Side. Proto changes seemed to have been made previously: https://github.com/cockroachdb/cockroach/blob/e68a36c9c82a6cb321e595c4c9875c58c759a438/pkg/util/log/eventpb/storage_events.proto#L113-L114

Fixes: #104531
Release note: None

Co-authored-by: Rahul Aggarwal <[email protected]>
@craig craig bot closed this as completed in b485fd9 Jun 16, 2023
raggar added a commit to raggar/cockroach that referenced this issue Jun 26, 2023
@raggar
kvserver: Fix WAL Bytes Written Metrics
3e3ef98 
Currently the number of WAL bytes written does not show up. This pr
fixes that by using a consistent metric name.

Informs: cockroachdb#104531
Release note: None
@raggar raggar mentioned this issue Jun 26, 2023
Merged
raggar added a commit to raggar/cockroach that referenced this issue Jun 26, 2023
@raggar
kvserver: Fix WAL Bytes Written Metrics
b166595 
Currently the number of WAL bytes written does not show up. This pr
fixes that by using a consistent metric name.

Informs: cockroachdb#104531
Release note: None
raggar added a commit to raggar/cockroach that referenced this issue Jun 26, 2023
@raggar
kvserver: Fix WAL Bytes Written Metrics
3eb6d6a 
Currently the number of WAL bytes written does not show up. This pr
fixes that by using a consistent metric name.

Informs: cockroachdb#104531
Release note: None
craig bot pushed a commit that referenced this issue Jun 27, 2023
@raggar
Merge #105536
c7965ed 
105536: kvserver: Fix WAL Bytes Written Metrics r=RahulAggarwal1016 a=RahulAggarwal1016

Currently the number of WAL bytes written does not show up. This pr fixes that by using a consistent metric name.

Informs: #104531
Release note: None

Co-authored-by: Rahul Aggarwal <[email protected]>
@jbowens jbowens moved this to Done in [Deprecated] Storage Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@raggar raggar

Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) E-quick-win Likely to be a quick win for someone experienced. T-storage Storage Team
Projects
No open projects
[Deprecated] Storage
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kvserver: Expose Bytes Written to WAL raggar/cockroach
2 participants
@jbowens @raggar