-
Notifications
You must be signed in to change notification settings - Fork 125
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ability to specify / disable temp directory #832
Comments
thank you so much @mihaitodor for creating this follow-up issue; now I can update this properly once there's any new progress on the improvement part |
#874 aims to implement this functionality. |
PR has been merged and will be part of the next release, which is expected towards the end of August |
@sfc-gh-dszmolka Awesome, thank you! Looking at the changes, I'm not sure if this also adds the ability to disable the usage of temp files. Is this something that you're planning to implement in the future? Also, I think it might be worth documenting this tempdir functionality in https://github.com/snowflakedb/gosnowflake/blob/master/doc.go so people can learn about it via the package docs: https://pkg.go.dev/github.com/snowflakedb/gosnowflake |
we'll definitely document it, it's in the plans, please bear with us @mihaitodor :) until it is 'officially' documented, I see it can be used with cfg: &Config{
User: "u",
Password: "p",
Account: "a.b.c",
TmpDirPath: "/tmp",
}, Regarding the other question about completely disabling usage of temp files, reading back the original request, it seems it was twofold:
The latter is not part of this PR and I would imagine it is maybe not even straightforward. I need to check with the rest of the team. |
got some update on this one:
Speaking of which, as with any feature request with any Snowflake connector: if you (or the future readers, who would need the ability of disabling the usage of temp directory completely) are already a Snowflake customer, please do speak to your Sales representative and let them know how important this feature would be to you. They have access to certain other aspects of your account, which often can help the product teams to prioritize the requests differently. Of course we'll still love to hear from the open-source community even if not a Snowflake customer, so please Future Readers: if you have a use-case for this feature (disabling temp dir usage completely, even at the cost of loss of driver functionality), let us know. |
Thank you for the update @sfc-gh-dszmolka! Regarding the disabling of the temp directory, I'd be curious to learn for which use case(s) it's beneficial to write temp files. I see from the updated docs that it's used for "encryption and compression", but I can't imagine a good reason to persist that data to disk. I haven't read all the code to see what it does, but I'm genuinely curious, because, for my use case at least, I'm using Additionally, there are use cases where people just want to do some small CRUD operations on tables without using a PUT statement. In that case, would temp files still get created? I guess it gets tricky when it falls back to an implicit stage like I mentioned in #540, but this isn't documented behaviour... |
Hello @mihaitodor ! Your questions made me to dig deeper. I believe that around PUT/GET support we can do some improvements, apart of giving up temp files, we can also move to more stream-based approach. It would require less memory. We have Q3 already planned, but we will consider to include it into our scope for the next quarters. |
i also raised #887 so the effort for the second request could be publicly tracked, once it's scheduled. |
Thank you @sfc-gh-pfus and @sfc-gh-dszmolka! I appreciate that! |
feature handled in this Issue
just got released with version 1.6.24, so closing this one. The other request is tracked separately. |
What is the current behavior?
Currently, users can specify the location where temp files are created via the Go-specific env var
TMPDIR
. The driver needs to write temp files as part of its workflow.What is the desired behavior?
We’d like the ability to specify the temp directory used by this driver via a dedicated mechanism, such as the
SNOWFLAKE_TMPDIR
env var instead of relying on Go’s general purposeTMPDIR
env var. Additionally, it would be great to be able to disable it completely, even if that means disabling some functionality.How would this improve
gosnowflake
?This would allow people to use the driver when the local file system is readonly.
References, Other Background
This is a follow-up to #700.
What is your Snowflake account identifier, if any?
The text was updated successfully, but these errors were encountered: