SNOW-723750: Panic with readonly file system

[mihaitodor]

Issue description

The gosnowflake driver panics for PUT operations when run in a container with a readonly file system. The panic happens here and the issue comes from here. I see a similar problem for download operations here.

Why does the driver require a disk buffer? I don't see any mention of this in the docs and there isn't any Snowflake-specific configuration variable for the disk path.

I can work around it by setting the TMPDIR environment variable as described in the Go docs here, but not being able to run this with a fully read-only file system can cause security challenges.

Example code

The sample code from the docs here should suffice.

Error log

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe0 pc=0x292c38c]

goroutine 74 [running]:
github.com/snowflakedb/gosnowflake.(*snowflakeFileTransferAgent).uploadFilesParallel(0xc001276b40, {0xc000c845d0, 0x1, 0x1})
	/Users/ntodor/go/pkg/mod/github.com/snowflakedb/[email protected]/file_transfer_agent.go:750 +0x40c
github.com/snowflakedb/gosnowflake.(*snowflakeFileTransferAgent).upload(0xc001276b40, {0xc001a2e2d0?, 0x0, 0x0}, {0xc000c845d0?, 0x1, 0x1})
	/Users/ntodor/go/pkg/mod/github.com/snowflakedb/[email protected]/file_transfer_agent.go:686 +0x191
github.com/snowflakedb/gosnowflake.(*snowflakeFileTransferAgent).execute(0xc001276b40)
	/Users/ntodor/go/pkg/mod/github.com/snowflakedb/[email protected]/file_transfer_agent.go:193 +0x585
github.com/snowflakedb/gosnowflake.(*snowflakeConn).processFileTransfer(0xc001345080, {0x41b3c38, 0xc00090be30}, 0xc001a44900, {0xc001b7e000, 0x8e}, 0x0)
	/Users/ntodor/go/pkg/mod/github.com/snowflakedb/[email protected]/connection_util.go:112 +0x1ef
github.com/snowflakedb/gosnowflake.(*snowflakeConn).exec(0xc001345080, {0x41b3c38, 0xc00090be30}, {0xc001b7e000, 0x8e}, 0x18?, 0x0, 0x0, {0x63ee368, 0x0, ...})
	/Users/ntodor/go/pkg/mod/github.com/snowflakedb/[email protected]/connection.go:136 +0x8e5
github.com/snowflakedb/gosnowflake.(*snowflakeConn).ExecContext(0xc001345080, {0x41b3c38, 0xc000906ba0}, {0xc001b7e000, 0x8e}, {0x63ee368, 0x0, 0x0})
	/Users/ntodor/go/pkg/mod/github.com/snowflakedb/[email protected]/connection.go:238 +0x225
database/sql.ctxDriverExec({0x41b3c38?, 0xc000906ba0?}, {0x7f18f6bf8188?, 0xc001345080?}, {0x0?, 0x0?}, {0xc001b7e000?, 0x2d8d1ae?}, {0x63ee368, 0x0, ...})
	/usr/local/Cellar/go/1.19.4/libexec/src/database/sql/ctxutil.go:31 +0xe5
database/sql.(*DB).execDC.func2()
	/usr/local/Cellar/go/1.19.4/libexec/src/database/sql/sql.go:1679 +0x175
database/sql.withLock({0x419e208, 0xc001094ea0}, 0xc001a2f720)
	/usr/local/Cellar/go/1.19.4/libexec/src/database/sql/sql.go:3439 +0x8c
database/sql.(*DB).execDC(0x101c001b30701?, {0x41b3c38, 0xc000906ba0}, 0xc001094ea0, 0x48cade?, {0xc001b7e000, 0x8e}, {0x0, 0x0, 0x0})
	/usr/local/Cellar/go/1.19.4/libexec/src/database/sql/sql.go:1674 +0x266
database/sql.(*DB).exec(0x4fbe4a?, {0x41b3c38, 0xc000906ba0}, {0xc001b7e000, 0x8e}, {0x0, 0x0, 0x0}, 0x8e?)
	/usr/local/Cellar/go/1.19.4/libexec/src/database/sql/sql.go:1659 +0xe8
database/sql.(*DB).ExecContext(0xc000f420a0?, {0x41b3c38, 0xc000906ba0}, {0xc001b7e000, 0x8e}, {0x0, 0x0, 0x0})
	/usr/local/Cellar/go/1.19.4/libexec/src/database/sql/sql.go:1633 +0xe5

Configuration

Driver version (or git SHA):
v1.6.16

Go version: run go version in your console
go version go1.19.4 darwin/amd64

Server version: E.g. 1.90.1
You may get the server version by running a query:

SELECT CURRENT_VERSION();

7.0.0

Client OS: E.g. Debian 8.1 (Jessie), Windows 10

golang:1.18-alpine Docker container.

[sfc-gh-dszmolka]

hi, thank you for submitting this issue with us and appreciate the detailed error description! we'll take a look.

[sfc-gh-dszmolka]

quick update: team found that the file encryption/decryption also create temporary directory and they are looking to stream the encrypted data instead of writing it into a file before upload.

[mihaitodor]

Thank you for the update @sfc-gh-dszmolka! Hope it will be possible to use this driver without any temp files being generated during processing

[sfc-gh-dszmolka]

PR #828 aims to address this issue

[mihaitodor]

PR #828 aims to address this issue

Thank you for taking this on! I had a quick look at the PR and I have some questions:

• Will it be possible to disable the temp file creation mechanism for certain workflows?
• Will there be a separate PR to document this functionality somehow?

[sfc-gh-dszmolka]

these are very valid questions. the second one, I'll work with the Documentation team to get it into the official product documentation. the first one, I think you can directly comment on the PR at this point, it's still very early.

[sfc-gh-dszmolka]

so the 'bug' part of the issue is fixed with the linked PR which will be part of the next release (coming towards mid- or end of July)

the 'enhancement' part is tracked separately internally (ability to specify temp directory and generally making the driver more resilient in read-only environments), and i think it would be worth creating a separate issue here too.

[mihaitodor]

@sfc-gh-dszmolka OK, thank you! I opened #832 to track the enhancements.

[sfc-gh-dszmolka]

version 1.6.23 of the driver has been released with the fix