feat: support using GCS for backup uploader #1130

JeffreyDallas · 2025-01-08T18:37:08Z

Description

This pull request changes the following:

Add new flags for using backupBucket and google credentials
google credentials is loaded to newly created k8 secret
Update test workflow

Related Issues

Closes Add S3 and GCS support into the Backup Uploader in solo-charts and update Solo to provide the secret #1101

Signed-off-by: Jeffrey Tang <[email protected]>

…google-storage

Signed-off-by: Jeffrey Tang <[email protected]>

…google-storage Signed-off-by: Jeffrey Tang <[email protected]> # Conflicts: # version.ts

Signed-off-by: Jeffrey Tang <[email protected]>

github-actions · 2025-01-09T03:49:12Z

E2E Test Report

16 files 121 suites 1h 21m 23s ⏱️
258 tests 258 ✅ 0 💤 0 ❌
261 runs 261 ✅ 0 💤 0 ❌

Results for commit 48d1a0d.

github-actions · 2025-01-09T03:49:13Z

E2E Test Report

16 files ±0 121 suites ±0 1h 24m 14s ⏱️ + 2m 49s
258 tests ±0 258 ✅ ±0 0 💤 ±0 0 ❌ ±0
261 runs ±0 261 ✅ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 21beef7. ± Comparison against base commit ed6527b.

♻️ This comment has been updated with latest results.

codacy-production · 2025-01-09T03:49:27Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ -0.06% (target: -1.00%)	✅ 63.64%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`ed6527b`)	20607	17398	84.43%
Head commit (`21beef7`)	20688 (+81)	17455 (+57)	84.37% (-0.06%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#1130)	55	35	63.64%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

codecov · 2025-01-09T03:50:16Z

Codecov Report

Attention: Patch coverage is 63.63636% with 20 lines in your changes missing coverage. Please review.

Project coverage is 83.59%. Comparing base (ed6527b) to head (21beef7).
Report is 37 commits behind head on main.

Files with missing lines	Patch %	Lines
src/commands/network.ts	31.03%	20 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1130      +/-   ##
==========================================
- Coverage   83.67%   83.59%   -0.08%     
==========================================
  Files          77       77              
  Lines       20634    20688      +54     
  Branches     1397     1869     +472     
==========================================
+ Hits        17265    17295      +30     
+ Misses       3326     3264      -62     
- Partials       43      129      +86

Files with missing lines	Coverage Δ
src/commands/flags.ts	`75.10% <100.00%> (+0.33%)`	⬆️
src/core/constants.ts	`97.34% <100.00%> (+0.01%)`	⬆️
version.ts	`100.00% <100.00%> (ø)`
src/commands/network.ts	`66.12% <31.03%> (-1.07%)`	⬇️

... and 26 files with indirect coverage changes

mishomihov00

Review applies to .github/workflows/flow-gcs-test.yaml

Signed-off-by: Jeffrey Tang <[email protected]>

…backup-uploader Signed-off-by: Jeffrey Tang <[email protected]> # Conflicts: # src/commands/flags.ts # src/commands/network.ts # version.ts

.github/workflows/script/gcs_test.sh

Signed-off-by: Jeffrey Tang <[email protected]>

## [0.33.0](v0.32.0...v0.33.0) (2025-01-13) ### Features * add support for genesis throttle configuration ([#1026](#1026)) ([d8903a8](d8903a8)) * enable Blockstreams and uploader ([#1109](#1109)) ([ed6527b](ed6527b)) * inspect existing cluster resources during cluster setup ([#1094](#1094)) ([808516c](808516c)) * production readiness pass custom mirror node postgresql credentials and server address ([#1128](#1128)) ([f8b7e41](f8b7e41)) * support using GCS for backup uploader ([#1130](#1130)) ([5fe2f4e](5fe2f4e)) * update genesis-network.json to add roster entries and more ([#1122](#1122)) ([c27d1d6](c27d1d6)) * Updating solo to leverage changes after moving acme-cluster-issuer and haproxy-ingress out of mirror-node-explorer ([#1079](#1079)) ([7965894](7965894)) * use gcs for stream file storage ([#994](#994)) ([04d5806](04d5806)) ### Bug Fixes * add missing images, fix debug page images ([#1092](#1092)) ([4073d31](4073d31)) * Bug when running `solo cluster connect` after fresh install ([#1123](#1123)) ([1cd0e28](1cd0e28)) * bump solo-charts to 0.42.2 to pull in blockStreams fix ([#1138](#1138)) ([c482895](c482895)) * bump versions of hedera, json rpc relay, and mirror node ([#1147](#1147)) ([0784b46](0784b46)) * clarify how to use nvm ([#1125](#1125)) ([bdc1873](bdc1873)) * commit message of auto update ([#1106](#1106)) ([a3231f9](a3231f9)) * external link rendering ([#1118](#1118)) ([28af1fe](28af1fe)) * file size empty error when throttle file is provided on command line ([#1140](#1140)) ([5180a76](5180a76)) * fixed release artifact version issue and updated readme link ([#1090](#1090)) ([84e6611](84e6611)) * **lease:** reworks the previous fix addressing renewal issues ([#1105](#1105)) ([5edb9c9](5edb9c9)) * should run `npm install` before build ([#1117](#1117)) ([3398875](3398875)) * solo deployment create should use email address in local config if it is already there instead of prompting the user for it ([#1112](#1112)) ([362df60](362df60)) * started adding a yachli example and taskfile ([#1132](#1132)) ([8753723](8753723)) * update to AccountManager._getNodeClient() to parallel node connection setup ([#1091](#1091)) ([4959a12](4959a12))

swirlds-automation · 2025-01-13T21:31:32Z

🎉 This PR is included in version 0.33.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

swirlds-automation · 2025-01-21T02:02:12Z

🎉 This PR is included in version 0.33.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

JeffreyDallas added 30 commits December 16, 2024 13:36

Add test

81d89c4

Signed-off-by: Jeffrey Tang <[email protected]>

Save

ca46661

Signed-off-by: Jeffrey Tang <[email protected]>

Save

cb79a05

Signed-off-by: Jeffrey Tang <[email protected]>

change prefix

2e84977

Signed-off-by: Jeffrey Tang <[email protected]>

save

5ce7874

Signed-off-by: Jeffrey Tang <[email protected]>

save

105cfbe

Signed-off-by: Jeffrey Tang <[email protected]>

Merge commit '861c579e353be7d452ff52e12c152123f5a84aef' into 00711-D-…

16ac154

…google-storage

update

43ceb61

Signed-off-by: Jeffrey Tang <[email protected]>

save

05174b9

Signed-off-by: Jeffrey Tang <[email protected]>

update bucket name

30b9b94

Signed-off-by: Jeffrey Tang <[email protected]>

fix id token

e878555

Signed-off-by: Jeffrey Tang <[email protected]>

save

a87fc71

Signed-off-by: Jeffrey Tang <[email protected]>

save

d13b2b6

Signed-off-by: Jeffrey Tang <[email protected]>

update service account

ae8ddd2

Signed-off-by: Jeffrey Tang <[email protected]>

save

defd504

Signed-off-by: Jeffrey Tang <[email protected]>

add install solo

0a7eccc

Signed-off-by: Jeffrey Tang <[email protected]>

save

517b91a

Signed-off-by: Jeffrey Tang <[email protected]>

add dump info

083b2aa

Signed-off-by: Jeffrey Tang <[email protected]>

save

a87f48f

Signed-off-by: Jeffrey Tang <[email protected]>

save

17676be

Signed-off-by: Jeffrey Tang <[email protected]>

debug

849efc5

Signed-off-by: Jeffrey Tang <[email protected]>

typo

93981cd

Signed-off-by: Jeffrey Tang <[email protected]>

add test

6ad1d53

Signed-off-by: Jeffrey Tang <[email protected]>

save

c385c9f

Signed-off-by: Jeffrey Tang <[email protected]>

save

9a899af

Signed-off-by: Jeffrey Tang <[email protected]>

setup helm

0548825

Signed-off-by: Jeffrey Tang <[email protected]>

save

6443e75

Signed-off-by: Jeffrey Tang <[email protected]>

save

63c9f37

Signed-off-by: Jeffrey Tang <[email protected]>

stop debug

1a9ebb5

Signed-off-by: Jeffrey Tang <[email protected]>

Merge commit '1b9b7e67eac69aedec695e50d9a93377e381c530' into 00711-D-…

d6b27e8

…google-storage Signed-off-by: Jeffrey Tang <[email protected]> # Conflicts: # version.ts

jeromy-cannon assigned JeffreyDallas Jan 8, 2025

JeffreyDallas added 2 commits January 8, 2025 16:42

save

a72de88

Signed-off-by: Jeffrey Tang <[email protected]>

switch to use secret to save credentials

f7e6177

Signed-off-by: Jeffrey Tang <[email protected]>

JeffreyDallas requested a review from jeromy-cannon January 9, 2025 00:04

JeffreyDallas marked this pull request as ready for review January 9, 2025 00:04

JeffreyDallas and others added 4 commits January 8, 2025 18:04

Merge branch 'main' into 01101-D-backup-uploader

ac7717d

save

ec0fdb2

Signed-off-by: Jeffrey Tang <[email protected]>

save

29f3b0e

Signed-off-by: Jeffrey Tang <[email protected]>

sace

48d1a0d

Signed-off-by: Jeffrey Tang <[email protected]>

mishomihov00 previously approved these changes Jan 9, 2025

View reviewed changes

JeffreyDallas added 2 commits January 9, 2025 09:09

update chart version

6eca938

Signed-off-by: Jeffrey Tang <[email protected]>

Merge commit 'ed6527b1ad9dda474967c6682d45e8de455343b3' into 01101-D-…

759cd04

…backup-uploader Signed-off-by: Jeffrey Tang <[email protected]> # Conflicts: # src/commands/flags.ts # src/commands/network.ts # version.ts

JeffreyDallas dismissed mishomihov00’s stale review via 759cd04 January 9, 2025 15:14

jeromy-cannon previously approved these changes Jan 9, 2025

View reviewed changes

andrewb1269hg reviewed Jan 9, 2025

View reviewed changes

.github/workflows/script/gcs_test.sh Show resolved Hide resolved

Add test logic to check backup uploader logs

35fa101

Signed-off-by: Jeffrey Tang <[email protected]>

JeffreyDallas dismissed jeromy-cannon’s stale review via 35fa101 January 9, 2025 17:17

Save

21beef7

Signed-off-by: Jeffrey Tang <[email protected]>

jeromy-cannon approved these changes Jan 9, 2025

View reviewed changes

JeffreyDallas merged commit 5fe2f4e into main Jan 9, 2025
40 of 41 checks passed

JeffreyDallas deleted the 01101-D-backup-uploader branch January 9, 2025 18:36

swirlds-automation added the released label Jan 13, 2025

swirlds-automation added the released on @0.33.x label Jan 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support using GCS for backup uploader #1130

feat: support using GCS for backup uploader #1130

JeffreyDallas commented Jan 8, 2025 •

edited

Loading

github-actions bot commented Jan 9, 2025

github-actions bot commented Jan 9, 2025 •

edited

Loading

codacy-production bot commented Jan 9, 2025 •

edited

Loading

codecov bot commented Jan 9, 2025 •

edited

Loading

mishomihov00 left a comment

swirlds-automation commented Jan 13, 2025

swirlds-automation commented Jan 21, 2025

feat: support using GCS for backup uploader #1130

feat: support using GCS for backup uploader #1130

Conversation

JeffreyDallas commented Jan 8, 2025 • edited Loading

Description

Related Issues

github-actions bot commented Jan 9, 2025

E2E Test Report

github-actions bot commented Jan 9, 2025 • edited Loading

E2E Test Report

codacy-production bot commented Jan 9, 2025 • edited Loading

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

codecov bot commented Jan 9, 2025 • edited Loading

Codecov Report

mishomihov00 left a comment

Choose a reason for hiding this comment

swirlds-automation commented Jan 13, 2025

swirlds-automation commented Jan 21, 2025

JeffreyDallas commented Jan 8, 2025 •

edited

Loading

github-actions bot commented Jan 9, 2025 •

edited

Loading

codacy-production bot commented Jan 9, 2025 •

edited

Loading

codecov bot commented Jan 9, 2025 •

edited

Loading