Feature flag to support the legacy column style #379
Conversation
istreeter
force-pushed
the
legacy-columns-in-v2
branch
from
82cd777 to
212b7f0
Compare
In v1 of BigQuery Loader, we created a separate column for each _minor_ version of an Iglu schema. Whereas in v2 of BigQuery Loader, schemas with the same major version share the same column. This might make it hard for users to upgrade from version 1 to version 2; and so users might miss out on all the other lovely stuff we added in v2. This PR adds back in support for the legacy style of columns. It is enabled by setting `legacyColumns = true` in the config file.
istreeter
force-pushed
the
legacy-columns-in-v2
branch
from
212b7f0 to
8465a17
Compare
...s/core/src/main/scala/com.snowplowanalytics.snowplow.bigquery/processing/LegacyColumns.scala
Show resolved
Hide resolved
| # -- For these columns, there is a column per _minor_ version of each schema. | ||
| "legacyColumns": [ | ||
| "iglu:com.acme/legacy/jsonschema/1-*-*" | ||
| "iglu:com.acme/legacy/jsonschema/2-*-*" |
There was a problem hiding this comment.
With first version we had a boolean toggle where we could just say 'I don't want v2 for now, just give me v1 for all columns'. Now If I want that behaviour, I have to list all my schemas here (well at least all names with *-*-* version pattern)?
I wonder if such toggle to disable v2 entirely could still be useful from user point of view. Maybe migration would be easier if we do the opposite - make v2Schemas configurable and all legacy by default...? Or some combination of boolean toggle + legacy schema list config.
There was a problem hiding this comment.
@stanch please could you comment on this idea?
I implemented it the current way based on a conversation we had. So users are forced to start using the new column style if they track a new entity. Based on that plan, I think there is no need to add a flag which turns on v1-by-default. And I liked that plan because it keeps the implementation simpler.
But now is the time to comment if we do want to allow v1-by-default for a subset of users.
There was a problem hiding this comment.
Actually @pondzix I don't think we need to wait for Nick's comment. I am confident this is implemented exactly as we discussed, and as written about in some internal document.
If closer to release we decide to implement v1-by-default as an option, then we can do so as an extra feature request.
modules/core/src/main/scala/com.snowplowanalytics.snowplow.bigquery/processing/Processing.scala
Outdated
Show resolved
Hide resolved
In v1 of BigQuery Loader, we created a separate column for each minor version of an Iglu schema. Whereas in v2 of BigQuery Loader, schemas with the same major version share the same column. This might make it hard for users to upgrade from version 1 to version 2; and so users might miss out on all the other lovely stuff we added in v2.
This PR adds back in support for the legacy style of columns. It is enabled by setting
legacyColumns = [...]in the config file, where the config value lists schema keys to treat as legacy.