Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add the ability to use a secret persistence #6415

Merged
merged 37 commits into from
Sep 29, 2021
Merged

add the ability to use a secret persistence #6415

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
cdfcd39
test exposing secrets in configrepo
jrhizor Sep 23, 2021
5f3d949
fix local persistence sql
jrhizor Sep 23, 2021
a09b411
working propagation, just without check/discover replacements and wit…
jrhizor Sep 24, 2021
5c337f5
switch if statement
jrhizor Sep 24, 2021
df73b87
set up secret persistence for google secrets manager
jrhizor Sep 25, 2021
d9fa73d
add ttl-based secret persistence for check/discover usage in the future
jrhizor Sep 25, 2021
489d2d5
set up check/discover to pass around necessary parts
jrhizor Sep 25, 2021
075f472
Revert "set up check/discover to pass around necessary parts"
jrhizor Sep 25, 2021
fb0ef64
working updates + check/discover operations
jrhizor Sep 25, 2021
8532a02
fix additional configs created on deletion
jrhizor Sep 25, 2021
0e79eb1
clean up docker compose file
jrhizor Sep 25, 2021
896a8d9
finish up configrepo
jrhizor Sep 25, 2021
13be907
make api path optional
jrhizor Sep 25, 2021
02dfbee
clean up schedulerapp and local testing persistence
jrhizor Sep 25, 2021
58a41ff
make optional in the worker app
jrhizor Sep 25, 2021
6e76768
add rest of feature flagging
jrhizor Sep 25, 2021
a279edb
fmt
jrhizor Sep 25, 2021
f25bb36
remove completed todo
jrhizor Sep 25, 2021
be25474
fix refactoring typo
jrhizor Sep 25, 2021
a32fd49
fix another refactoring typo
jrhizor Sep 25, 2021
5dfb0ef
fix compilation error in test case
jrhizor Sep 25, 2021
6bc4b3b
fix tests
jrhizor Sep 25, 2021
405cd1c
final cleanups
jrhizor Sep 25, 2021
8d5ceb7
fix conditional
jrhizor Sep 25, 2021
948f307
Merge branch 'master' into jrhizor/secrets-in-configrepo
jrhizor Sep 28, 2021
5c865ca
address a couple of things
jrhizor Sep 28, 2021
c91d597
add hydrator interface
jrhizor Sep 28, 2021
91e0cd0
add replaceAllConfigs
jrhizor Sep 29, 2021
5e74c50
specfetcher handling
jrhizor Sep 29, 2021
9f8a932
fix constructor
jrhizor Sep 29, 2021
e5ed837
fix test
jrhizor Sep 29, 2021
64097aa
Merge branch 'master' into jrhizor/secrets-in-configrepo
jrhizor Sep 29, 2021
702c5a7
fix typo
jrhizor Sep 29, 2021
37d101a
fix merge build error
jrhizor Sep 29, 2021
051f8b0
remove extra config
jrhizor Sep 29, 2021
a2dca1d
fix integration test
jrhizor Sep 29, 2021
b90f9e8
fix final piece
jrhizor Sep 29, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,3 +75,6 @@ MAX_SYNC_JOB_ATTEMPTS=3

# Time in days to reach a timeout to cancel the synchronization
MAX_SYNC_TIMEOUT_DAYS=3

# Set secret persistence store to use. Do not change this for existing installations!
SECRET_PERSISTENCE=NONE
10 changes: 8 additions & 2 deletions airbyte-config/models/src/main/java/io/airbyte/config/Configs.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,6 @@ public interface Configs {

String getSecretStoreGcpCredentials();

String getSecretStoreForConfigs();

boolean runDatabaseMigrationOnStartup();

int getMaxSyncJobAttempts();
Expand Down Expand Up @@ -109,6 +107,8 @@ public interface Configs {

String getGoogleApplicationCredentials();

SecretPersistenceType getSecretPersistenceType();

enum TrackingStrategy {
SEGMENT,
LOGGING
Expand All @@ -124,4 +124,10 @@ enum DeploymentMode {
CLOUD
}

enum SecretPersistenceType {
NONE,
TESTING_CONFIG_DB_TABLE,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe just test? and then have a comment or readme?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bump

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm leaving as is. The usage is a bit confusing otherwise. I can add readmes after the fact too.

GOOGLE_SECRET_MANAGER
}

}
13 changes: 7 additions & 6 deletions airbyte-config/models/src/main/java/io/airbyte/config/EnvConfigs.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ public class EnvConfigs implements Configs {
private static final String RESOURCE_CPU_LIMIT = "RESOURCE_CPU_LIMIT";
private static final String RESOURCE_MEMORY_REQUEST = "RESOURCE_MEMORY_REQUEST";
private static final String RESOURCE_MEMORY_LIMIT = "RESOURCE_MEMORY_LIMIT";
private static final String SECRET_PERSISTENCE = "SECRET_PERSISTENCE";
private static final String JOBS_IMAGE_PULL_SECRET = "JOBS_IMAGE_PULL_SECRET";

// defaults
Expand All @@ -76,7 +77,6 @@ public class EnvConfigs implements Configs {
private static final String DEFAULT_RESOURCE_REQUIREMENT_MEMORY = null;
private static final String SECRET_STORE_GCP_PROJECT_ID = "SECRET_STORE_GCP_PROJECT_ID";
private static final String SECRET_STORE_GCP_CREDENTIALS = "SECRET_STORE_GCP_CREDENTIALS";
private static final String SECRET_STORE_FOR_CONFIGS = "SECRET_STORE_CONFIGS_ENABLE";
private static final long DEFAULT_MINIMUM_WORKSPACE_RETENTION_DAYS = 1;
private static final long DEFAULT_MAXIMUM_WORKSPACE_RETENTION_DAYS = 60;
private static final long DEFAULT_MAXIMUM_WORKSPACE_SIZE_MB = 5000;
Expand Down Expand Up @@ -191,11 +191,6 @@ public String getSecretStoreGcpProjectId() {
return getEnv(SECRET_STORE_GCP_PROJECT_ID);
}

@Override
public String getSecretStoreForConfigs() {
return getEnv(SECRET_STORE_FOR_CONFIGS);
}

@Override
public boolean runDatabaseMigrationOnStartup() {
return getEnvOrDefault(RUN_DATABASE_MIGRATION_ON_STARTUP, true);
Expand Down Expand Up @@ -428,6 +423,12 @@ public String getGoogleApplicationCredentials() {
return getEnvOrDefault(LogClientSingleton.GOOGLE_APPLICATION_CREDENTIALS, "");
}

@Override
public SecretPersistenceType getSecretPersistenceType() {
final var secretPersistenceStr = getEnvOrDefault(SECRET_PERSISTENCE, SecretPersistenceType.NONE.name());
return SecretPersistenceType.valueOf(secretPersistenceStr);
}

private String getEnvOrDefault(final String key, final String defaultValue) {
return getEnvOrDefault(key, defaultValue, Function.identity(), false);
}
Expand Down
1 change: 1 addition & 0 deletions airbyte-config/persistence/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ plugins {
dependencies {
implementation 'commons-io:commons-io:2.7'

implementation project(':airbyte-commons-docker')
implementation project(':airbyte-db:lib')
implementation project(':airbyte-db:jooq')
implementation project(':airbyte-protocol:models')
Expand Down
Loading