Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: implement helper init command for creating snowflake objects #21

Closed
pnadolny13 opened this issue Apr 27, 2023 · 2 comments · Fixed by #80
Closed

feat: implement helper init command for creating snowflake objects #21

pnadolny13 opened this issue Apr 27, 2023 · 2 comments · Fixed by #80

Comments

@pnadolny13
Copy link
Contributor

This was mentioned in meltano/meltano#2814 (comment) and Office Hours.

The premise is that taps/targets are all CLIs and they already have all the credentials and logic needed to connect to the source or destination so we could extend it to do other operations.

In this case it would be nice to help users initialize needed snowflake objects so onboarding is seamless. I'm not 100% this is relevant for this target but in pipelinewise's the user needs to create a file format + role + optional stage + apply permissions appropriately. I see in

target-snowflake/target_snowflake/sinks.py

Line 284 in f791770

# create file format in new schema
that we auto create some stuff though.

It would be cool to allow 2 options:

  • print SQL (default) - I'm sure most users would prefer to see the SQL before its run and likely run it themselves manually. Also some might require elevated permissions that the connector doesnt have.
  • execute SQL - if the connector has enough permissions it should just execute the SQL.
@kgpayne kgpayne mentioned this issue Apr 27, 2023
Merged
@pnadolny13
Copy link
Contributor Author

The use cases that I was thinking of for this issue are already handled within this target. In comparison to the pipelinewise variant the user no longer needs to create any objects or assign permissions, thats all done by this target during the sync.

I'll close this for now but if more use cases come up for this feature we can reopen it.

@pnadolny13
Copy link
Contributor Author

Reopening this because meltano/hub#1407 mentions more requirements for someone with a totally empty account. We could add an init command here that the hub recommends instead of the hub documenting a bunch of sql scripts.

@pnadolny13 pnadolny13 reopened this Jul 10, 2023
@pnadolny13 pnadolny13 mentioned this issue Jul 11, 2023
Merged
pnadolny13 added a commit that referenced this issue Jul 11, 2023
@pnadolny13 @edgarrmondragon
feat: --initialize command to onboard a new account (#80)
c45a206 
Closes #21

This is kind of a new idea that we talked about in office hours a while
back for leveraging the target's capabilities to configure the database.
Related to meltano/hub#1407.

Adds a CLI flag for initializing a new account. It interactively prompts
for all the information it needs and will execute it if you want it to,
otherwise it will just print the sql and you can run it yourself. Based
on https://fivetran.com/docs/destinations/snowflake/setup-guide.

I created a personal trial snowflake account and was able to initialize
using this script then immediately run a tap using the new
user/role/warehouse/database.

---------

Co-authored-by: Edgar R. M <[email protected]>
@github-project-automation github-project-automation bot moved this to Planned in Data Team Jul 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
MeltanoLabs Overview
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: --initialize command to onboard a new account MeltanoLabs/target-snowflake
1 participant
@pnadolny13