Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add feature for copying backups #1153

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
143 commits
Select commit Hold shift + click to select a range
46ce5b8
src test and protos copied over
danieljbruce Jul 29, 2022
576550d
lint fix
danieljbruce Jul 29, 2022
31a1d9a
implemented copy backup
danieljbruce Aug 2, 2022
90e290e
part of test done
danieljbruce Aug 2, 2022
7bce08d
workaround
danieljbruce Aug 22, 2022
29a736f
corrections to get code working
danieljbruce Sep 7, 2022
edc30d3
Add expire time
danieljbruce Sep 7, 2022
c8a16fc
This allows the copy backup endpoint to work
danieljbruce Sep 7, 2022
2d5320c
Working system test
danieljbruce Sep 7, 2022
adabd71
delete try block
danieljbruce Sep 7, 2022
89f742e
tests pass with new refactor
danieljbruce Sep 7, 2022
d4f5d47
Create backup and copy on another cluster
danieljbruce Sep 8, 2022
ea43929
modifications to second test
danieljbruce Sep 8, 2022
bb6e708
copy backup unit test
danieljbruce Sep 9, 2022
c96442a
Revert "lint fix"
danieljbruce Sep 9, 2022
701badc
Revert "src test and protos copied over"
danieljbruce Sep 9, 2022
448e41d
First correction
danieljbruce Sep 9, 2022
7407fa2
Check the list of backups
danieljbruce Sep 15, 2022
4eba7d3
fetched backup
danieljbruce Sep 15, 2022
9c21e29
get name and id
danieljbruce Sep 15, 2022
c87a4d9
Remove only
danieljbruce Sep 16, 2022
a5c1ce2
refactor request mock
danieljbruce Sep 28, 2022
3ee1ec3
Add unit test for copying backup to another projec
danieljbruce Sep 28, 2022
1d80cf4
Merge branch 'main' into copy-backup-2-relevant-changes-only
danieljbruce May 26, 2023
081ddac
feat: add experimental reverse scan for public preview
gcf-owl-bot[bot] Jun 26, 2023
2315c78
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jun 26, 2023
0c07e7c
feat: Increase the maximum retention period for a Cloud Bigtable back…
gcf-owl-bot[bot] Jun 29, 2023
2330a56
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jun 29, 2023
aa5bab3
docs: fix formatting for reversed order field example
gcf-owl-bot[bot] Jul 12, 2023
967bd71
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jul 12, 2023
0075152
feat: add last_scanned_row_key feature
gcf-owl-bot[bot] Jul 26, 2023
c97b89f
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jul 26, 2023
fbdf7d4
fix: fix typings for IAM methods
gcf-owl-bot[bot] Jul 27, 2023
8c685fd
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jul 27, 2023
38d7961
test: disable retry-request for streaming tests
gcf-owl-bot[bot] Aug 8, 2023
e4d2229
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 8, 2023
77d105f
Merge branch 'main' into owl-bot-copy
sofisl Aug 15, 2023
a69c8c6
feat: publish CopyBackup protos to external customers
gcf-owl-bot[bot] Aug 15, 2023
d3387d3
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 15, 2023
452eef0
fix: simplify logic for HTTP/1.1 REST fallback option
gcf-owl-bot[bot] Aug 24, 2023
d7c4e94
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Aug 24, 2023
c142deb
fix: add feature flag for improved mutate rows throttling
gcf-owl-bot[bot] Sep 13, 2023
fbe0b0c
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 13, 2023
11148fb
Merge branch 'main' into owl-bot-copy
sofisl Sep 20, 2023
e055215
run lint
sofisl Sep 20, 2023
463666a
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 20, 2023
3318e84
Merge branch 'main' into copy-backup-2-relevant-changes-only
meredithslota Sep 22, 2023
c932b4c
build: update typescript generator version to publish in dual format …
gcf-owl-bot[bot] Sep 26, 2023
db741f2
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Sep 26, 2023
0b91bb6
feat: Add support for Cloud Bigtable Request Priorities in App Profiles
gcf-owl-bot[bot] Oct 6, 2023
307a8cb
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Oct 6, 2023
ee33f43
Merge branch 'main' into owl-bot-copy
danieljbruce Oct 11, 2023
a3a4d9c
build: update Node.js generator to compile protos
gcf-owl-bot[bot] Nov 15, 2023
e554f92
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Nov 15, 2023
3a1cd27
Merge branch 'main' into copy-backup-2-relevant-changes-only
danieljbruce Nov 20, 2023
3f594b2
Merge branch 'owl-bot-copy' of https://github.com/googleapis/nodejs-b…
danieljbruce Nov 24, 2023
4ffb7a1
Use the destination backup expiry time
danieljbruce Nov 24, 2023
8073653
Use the config instead for copy backup
danieljbruce Nov 27, 2023
e1f2dde
Rename CopyBackupConfig
danieljbruce Dec 6, 2023
f66ee04
Merge branch 'main' into copy-backup-2-relevant-changes-only
danieljbruce Dec 6, 2023
13e128d
Fix signature of copyBackup
danieljbruce Dec 6, 2023
b1af9c1
Merge branch 'copy-backup-2-relevant-changes-only' of https://github.…
danieljbruce Dec 6, 2023
4c0310c
Change the copy signature to require callback
danieljbruce Dec 11, 2023
d61e1cc
Some debugging on the system tests
danieljbruce Dec 11, 2023
e2a8be5
DestinationBackupConfig
danieljbruce Dec 11, 2023
763115b
lint fix
danieljbruce Dec 11, 2023
1827059
Move the config and create a generic callback
danieljbruce Jan 3, 2024
3259115
Eliminate the generic callback data structure
danieljbruce Jan 3, 2024
4534399
Get the first copy backup test working
danieljbruce Jan 3, 2024
0836841
Fix different cluster, different instance test
danieljbruce Jan 4, 2024
49f693f
Add another test for copy backup
danieljbruce Jan 4, 2024
67d5eb0
Add a test case for copying to another project
danieljbruce Jan 5, 2024
2ef42e6
Restructure tests for multiple expire time inputs
danieljbruce Jan 5, 2024
86a0311
Various changes to test interaction with copy back
danieljbruce Jan 8, 2024
f163aba
Finish the restore tests and modify copy backup
danieljbruce Jan 8, 2024
a27f29f
Add comment describing second argument
danieljbruce Jan 8, 2024
80f0154
Remove TODO
danieljbruce Jan 9, 2024
3b95726
Modify copy backup unit test
danieljbruce Jan 9, 2024
ae0e313
Modify the name of the test
danieljbruce Jan 9, 2024
df282b4
Test for gax options in the test
danieljbruce Jan 9, 2024
99e3d8c
TODO is done
danieljbruce Jan 9, 2024
3dddc0b
Change comment slightly
danieljbruce Jan 9, 2024
5a0d07b
Eliminate TODO
danieljbruce Jan 9, 2024
e4d38d4
Fix documentation for source function
danieljbruce Jan 10, 2024
6a26bde
Move copy backup code to proper place
danieljbruce Jan 10, 2024
e7e1eb3
Use cluster instead of parent
danieljbruce Jan 10, 2024
d859a33
Change parent to cluster
danieljbruce Jan 10, 2024
b2c728f
A couple cleanup changes
danieljbruce Jan 10, 2024
377d5bc
Use parent instead of cluster
danieljbruce Jan 10, 2024
1520f22
Eliminate unused time references
danieljbruce Jan 10, 2024
74cbffa
This test should not have changed
danieljbruce Jan 10, 2024
26289af
Eliminate unused import
danieljbruce Jan 11, 2024
c5ba924
Remove assert statement that is not needed
danieljbruce Jan 11, 2024
ffb9588
Move two lines of code to location used
danieljbruce Jan 11, 2024
1fe2ef0
Improve readability
danieljbruce Jan 11, 2024
9c79b43
Inline variables instead of using them explicitly
danieljbruce Jan 11, 2024
7ab1c96
Generate the backup id inline
danieljbruce Jan 11, 2024
842b70b
readability - indent code so it is separate
danieljbruce Jan 11, 2024
3872ff0
Indent block to separate check
danieljbruce Jan 11, 2024
02078e5
indent cluster creation, expiry time check
danieljbruce Jan 11, 2024
5ac03be
Another test cleanup
danieljbruce Jan 11, 2024
4bd4777
Add comment
danieljbruce Jan 11, 2024
7ee2f18
Indent code for table data insertion
danieljbruce Jan 11, 2024
1a0e153
Rename variables to distinguish operations
danieljbruce Jan 11, 2024
e3fdbd2
Eliminate unused imports
danieljbruce Jan 11, 2024
b232315
should be lower case
danieljbruce Jan 11, 2024
9890fd8
Change comment to respect alignment
danieljbruce Jan 11, 2024
56c7b7f
Inline callback, config and cluster
danieljbruce Jan 11, 2024
83b7e7e
Remove unused operation
danieljbruce Jan 11, 2024
af77ce8
Merge branch 'main' into copy-backup-2-relevant-changes-only
danieljbruce Jan 11, 2024
b6fa45a
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jan 11, 2024
a3c83bd
Merge branch 'copy-backup-2-relevant-changes-only' of https://github.…
gcf-owl-bot[bot] Jan 11, 2024
d63f892
Delete an instance
danieljbruce Jan 11, 2024
d2de338
Merge branch 'copy-backup-2-relevant-changes-only' of https://github.…
danieljbruce Jan 11, 2024
9e8731d
Inline generate id
danieljbruce Jan 11, 2024
2695d96
Add second project to config
danieljbruce Jan 19, 2024
cc8ce59
Rename the function to setBackupExpiryTime
danieljbruce Jan 19, 2024
45ef624
Replace the project for backup path
danieljbruce Jan 19, 2024
835c827
Merge branch 'main' into copy-backup-2-relevant-changes-only
danieljbruce Jan 23, 2024
2518d3d
Run the linter
danieljbruce Jan 23, 2024
40f3cc2
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Jan 23, 2024
dc149e6
Merge branch 'main' into copy-backup-2-relevant-changes-only
danieljbruce Jan 29, 2024
165ce36
Merge branch 'main' of https://github.com/googleapis/nodejs-bigtable …
danieljbruce Jan 30, 2024
0720df1
Merge branch 'copy-backup-2-relevant-changes-only' of https://github.…
danieljbruce Jan 30, 2024
22e9bec
Update comments for the copy function
danieljbruce Feb 12, 2024
dfc03bf
Rename function that sets correct value of expiry
danieljbruce Feb 12, 2024
8cba026
Add an environment variable for second project
danieljbruce Feb 16, 2024
d0a0e1f
Add log for second project
danieljbruce Feb 20, 2024
3b16183
Remove the console log
danieljbruce Feb 20, 2024
6a46c8c
Merge branch 'main' into copy-backup-2-relevant-changes-only
danieljbruce Feb 22, 2024
9253c9c
Merge branch 'main' into copy-backup-2-relevant-changes-only
danieljbruce Apr 10, 2024
4e49ab7
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Apr 12, 2024
ff3e603
Rename bigtable to bigtableSecondaryProject
danieljbruce May 6, 2024
fcae645
Merge branch 'copy-backup-2-relevant-changes-only' of https://github.…
danieljbruce May 6, 2024
307d110
PR follow-ups - inline code and rename variables
danieljbruce May 6, 2024
69318a0
Merge branch 'main' into copy-backup-2-relevant-changes-only
danieljbruce May 8, 2024
1b334a1
Remove unused import
danieljbruce May 13, 2024
781516f
Add comments for expire time
danieljbruce May 13, 2024
9a48759
Add better garbage cleanups of the backups
danieljbruce May 13, 2024
71c5bef
These comments are not needed anymore.
danieljbruce May 13, 2024
2089c64
Remove only
danieljbruce May 13, 2024
4bc7a7d
Update system-test/bigtable.ts
danieljbruce May 21, 2024
661c927
Update system-test/bigtable.ts
danieljbruce May 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions src/backup.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,15 @@ import {
} from './cluster';
import {CallOptions, LROperation, Operation, ServiceError} from 'google-gax';
import {Instance} from './instance';
import {ClusterUtils} from './utils/cluster';

export type CopyBackupResponse = GenericBackupPromise<Operation>;
export type CopyBackupCallback = GenericBackupCallback<Operation>;
export interface CopyBackupConfig extends ModifiableBackupFields {
cluster: Cluster;
gaxOptions?: CallOptions;
kolea2 marked this conversation as resolved.
Show resolved Hide resolved
id: string;
}

type IEmpty = google.protobuf.IEmpty;
export type IBackup = google.bigtable.admin.v2.IBackup;
Expand All @@ -59,6 +68,7 @@ export interface GenericBackupCallback<T> {
apiResponse?: T | null
): void;
}
export type GenericBackupPromise<T> = [Backup, T];

export type DeleteBackupCallback = (
err: ServiceError | null,
Expand Down Expand Up @@ -243,6 +253,50 @@ Please use the format 'my-backup' or '${cluster.name}/backups/my-backup'.`);
});
}

/**
* When this backup object represents a backup that has already been created,
* copy will copy this created backup to the location and with the settings
* specified by the config parameter. After running this function the original
* backup will exist as well as a second backup matching the parameters given
* by the config argument.
*
* @param {CopyBackupConfig} [config] The config that specifies all of the
* information about the destination backup which is the new backup that gets
* created as a result of calling copy.
* @param {CopyBackupCallback} [callback] The callback function that passes an
* error or results back to the user.
*/
copy(config: CopyBackupConfig, callback: CopyBackupCallback): void;
copy(config: CopyBackupConfig): Promise<CopyBackupResponse>;
copy(
config: CopyBackupConfig,
callback?: CopyBackupCallback
): void | Promise<CopyBackupResponse> {
const reqOpts = {
parent: config.cluster.name,
backupId: config.id,
sourceBackup: `${this.cluster.name}/backups/${this.id}`,
expireTime: config?.expireTime,
};
ClusterUtils.formatBackupExpiryTime(reqOpts);
this.bigtable.request(
{
client: 'BigtableTableAdminClient',
method: 'copyBackup',
reqOpts,
gaxOpts: config.gaxOptions,
},
(err, ...args) => {
if (err) {
callback!(err, undefined, ...args);
return;
}
// Second argument is a backup for the new backup id
callback!(null, config.cluster.backup(config.id), ...args);
}
);
}

create(config: CreateBackupConfig, callback?: CreateBackupCallback): void;
create(config: CreateBackupConfig): Promise<CreateBackupResponse>;
/**
Expand Down
7 changes: 1 addition & 6 deletions src/cluster.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
// See the License for the specific language governing permissions and
// limitations under the License.

import {PreciseDate} from '@google-cloud/precise-date';
import {promisifyAll} from '@google-cloud/promisify';
import {CallOptions, LROperation, Operation, ServiceError} from 'google-gax';

Expand Down Expand Up @@ -314,11 +313,7 @@ Please use the format 'my-cluster' or '${instance.name}/clusters/my-cluster'.`);
},
};

if (reqOpts.backup.expireTime instanceof Date) {
reqOpts.backup.expireTime = new PreciseDate(
reqOpts.backup.expireTime
).toStruct();
}
ClusterUtils.formatBackupExpiryTime(reqOpts.backup);
leahecole marked this conversation as resolved.
Show resolved Hide resolved

delete reqOpts.backup.table;
delete reqOpts.backup.gaxOptions;
Expand Down
8 changes: 8 additions & 0 deletions src/utils/cluster.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ import {
SetClusterMetadataOptions,
} from '../cluster';
import {google} from '../../protos/protos';
import {ModifiableBackupFields} from '../backup';
import {PreciseDate} from '@google-cloud/precise-date';

export class ClusterUtils {
static noConfigError =
Expand Down Expand Up @@ -163,4 +165,10 @@ export class ClusterUtils {
updateMask: {paths: this.getUpdateMask(metadata)},
};
}

static formatBackupExpiryTime(backup: ModifiableBackupFields) {
if (backup.expireTime instanceof Date) {
kolea2 marked this conversation as resolved.
Show resolved Hide resolved
backup.expireTime = new PreciseDate(backup.expireTime).toStruct();
}
}
}
Loading
Loading