Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[subtask] store and retrieve the Iceberg metrics in Iceberg REST service #453

Closed
Tracked by #321
FANNG1 opened this issue Sep 26, 2023 · 3 comments
Closed
Tracked by #321
Assignees
Labels
improvement Improvements on everything

Comments

@FANNG1
Copy link
Contributor

FANNG1 commented Sep 26, 2023

What would you like to be improved?

now we just add a mock interface and drop the metrics. we need to store the metrics.

parent: #321

@FANNG1
Copy link
Contributor Author

FANNG1 commented Oct 26, 2023

Iceberg metrics include commit metrics and scan metrics.

commit metrics:

{reportType=COMMIT_REPORT, report=CommitReport{tableName=rest.iceberg_rest_dml.test, snapshotId=7084087367671717860, sequenceNumber=0, operation=append, commitMetrics=CommitMetricsResult{totalDuration=TimerResult{timeUnit=NANOSECONDS, totalDuration=PT0.359504583S, count=1}, attempts=CounterResult{unit=COUNT, value=1}, addedDataFiles=CounterResult{unit=COUNT, value=1}, removedDataFiles=null, totalDataFiles=CounterResult{unit=COUNT, value=2}, addedDeleteFiles=null, addedEqualityDeleteFiles=null, addedPositionalDeleteFiles=null, removedDeleteFiles=null, removedEqualityDeleteFiles=null, removedPositionalDeleteFiles=null, totalDeleteFiles=CounterResult{unit=COUNT, value=0}, addedRecords=CounterResult{unit=COUNT, value=2}, removedRecords=null, totalRecords=CounterResult{unit=COUNT, value=4}, addedFilesSizeInBytes=CounterResult{unit=BYTES, value=662}, removedFilesSizeInBytes=null, totalFilesSizeInBytes=CounterResult{unit=BYTES, value=1324}, addedPositionalDeletes=null, removedPositionalDeletes=null, totalPositionalDeletes=CounterResult{unit=COUNT, value=0}, addedEqualityDeletes=null, removedEqualityDeletes=null, totalEqualityDeletes=CounterResult{unit=COUNT, value=0}}, metadata={engine-version=3.4.1, app-id=local-1698288445275, engine-name=spark, iceberg-version=Apache Iceberg 1.3.1 (commit 62c34711c3f22e520db65c51255512f6cfe622c4)}}}

Scan metrics:

{reportType=SCAN_REPORT, report=ScanReport{tableName=rest.iceberg_rest_dml.test, snapshotId=7084087367671717860, filter=true, schemaId=0, projectedFieldIds=[1, 2], projectedFieldNames=[id, data], scanMetrics=ScanMetricsResult{totalPlanningDuration=TimerResult{timeUnit=NANOSECONDS, totalDuration=PT0.145495083S, count=1}, resultDataFiles=CounterResult{unit=COUNT, value=2}, resultDeleteFiles=CounterResult{unit=COUNT, value=0}, totalDataManifests=CounterResult{unit=COUNT, value=2}, totalDeleteManifests=CounterResult{unit=COUNT, value=0}, scannedDataManifests=CounterResult{unit=COUNT, value=2}, skippedDataManifests=CounterResult{unit=COUNT, value=0}, totalFileSizeInBytes=CounterResult{unit=BYTES, value=1324}, totalDeleteFileSizeInBytes=CounterResult{unit=BYTES, value=0}, skippedDataFiles=CounterResult{unit=COUNT, value=0}, skippedDeleteFiles=CounterResult{unit=COUNT, value=0}, scannedDeleteManifests=CounterResult{unit=COUNT, value=0}, skippedDeleteManifests=CounterResult{unit=COUNT, value=0}, indexedDeleteFiles=CounterResult{unit=COUNT, value=0}, equalityDeleteFiles=CounterResult{unit=COUNT, value=0}, positionalDeleteFiles=CounterResult{unit=COUNT, value=0}}, metadata={engine-version=3.4.1, iceberg-version=Apache Iceberg 1.3.1 (commit 62c34711c3f22e520db65c51255512f6cfe622c4), app-id=local-1698288445275, engine-name=spark}}}

@FANNG1 FANNG1 changed the title [Improvement] store and retrieve the Iceberg metrics in Iceberg REST service [subtask] store and retrieve the Iceberg metrics in Iceberg REST service Oct 26, 2023
@jerryshao jerryshao removed this from the Gravitino 0.3.0 milestone Nov 23, 2023
@FANNG1 FANNG1 self-assigned this Nov 24, 2023
@FANNG1 FANNG1 added this to the Gravitino 0.3.0 milestone Nov 24, 2023
@FANNG1
Copy link
Contributor Author

FANNG1 commented Nov 24, 2023

@FANNG1
Copy link
Contributor Author

FANNG1 commented Nov 27, 2023

@jerryshao , please move this to Gravitino 0.4, I'll try to fix the remaining issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improvements on everything
Projects
None yet
Development

No branches or pull requests

2 participants