Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StorageLoader: replace table configuration property with schema #2458

Closed
alexanderdean opened this issue Feb 20, 2016 · 0 comments
Closed

StorageLoader: replace table configuration property with schema #2458

alexanderdean opened this issue Feb 20, 2016 · 0 comments
Assignees

Comments

@alexanderdean
Copy link
Member

The problem with the current approach is that it suggests that:

:targets:
  - :name:     All time
    :type:     redshift
    :host:     redshift.acme.com
    :database: snowplow
    :table:    atomic.events
    ...
  - :name:     Rolling 24h
    :type:     redshift
    :host:     redshift.acme.com
    :database: snowplow
    :table:    atomic.events_last_1d

is safe. In fact, the above is highly unsafe - because both loads would populate the same child tables in atomic.

For this reason, we should remove the end user control over the events table (which is currently the only table the user can rename) and replace the outdated table option with a schema option (correctly implying that each distinct Snowplow load needs to be sandboxed into its own schema).

@alexanderdean alexanderdean added this to the Snowplow CLI #3 milestone Feb 20, 2016
@alexanderdean alexanderdean modified the milestones: R8x [HAD] Cross-batch dedupe, Snowplow CLI #3 Dec 6, 2016
chuwy added a commit to snowplow/snowplow-rdb-loader that referenced this issue Sep 5, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants