Add io/fs#FS Driver #471 #472

johejo · 2020-11-12T16:50:43Z

close #471

coveralls · 2020-11-12T17:22:22Z

Pull Request Test Coverage Report for Build 929

90 of 136 (66.18%) changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 55.427%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
source/file/file_go115.go	11	13	84.62%
source/file/file_go116.go	11	13	84.62%
source/file/file.go	2	5	40.0%
source/iofs/iofs.go	66	105	62.86%

Totals
Change from base Build 919:	0.2%
Covered Lines:	3125
Relevant Lines:	5638

💛 - Coveralls

dhui

Thanks for the PR!

You'll need to add an internal/cli/build_iofs.go file and update Makefile with the build constraint for the official CLI to support iofs.

dhui · 2020-11-13T05:07:59Z

source/iofs/README.md

+
+Driver with file system interface (`io/fs#FS`) supported from Go 1.16.
+
+This Driver cannot be used with versions below Go 1.15.


Update to: This Driver cannot be used with Go versions 1.15 and below

dhui · 2020-11-13T05:22:20Z

source/iofs/iofs.go

+
+// Iofs is a source driver for io/fs#FS.
+type Iofs struct {
+	httpfs.PartialDriver


A refactor may be in-order.
It makes sense to go the other direction. e.g. add an iofs.PartialDriver and have httpfs embed that, since io/fs.File interface is a subset of the net/http.File interface

dhui · 2020-11-13T05:31:07Z

source/iofs/iofs.go

+// WithInstance wraps io/fs#FS as source.Driver.
+func WithInstance(fsys fs.FS, path string) (source.Driver, error) {
+	var i Iofs
+	if err := i.Init(http.FS(fsys), path); err != nil {


Converting from fs.FS to http.FileSystem may fail later if the underlying struct doesn't implement io.Seeker. Seek() isn't used in the httpfs source driver right now, but if that changes, this would break.

See: https://tip.golang.org/src/net/http/fs.go

func (f ioFile) Seek(offset int64, whence int) (int64, error) { s, ok := f.file.(io.Seeker) if !ok { return 0, errMissingSeek } return s.Seek(offset, whence) }

dhui

Thanks for updating this to not depend on httpfs!

dhui · 2020-11-13T17:30:05Z

source/iofs/iofs.go

+}
+
+// Iofs is a source driver for io/fs#FS.
+type Iofs struct {


Rename to IoFS since abbreviations should be in all caps.

dhui · 2020-11-14T04:52:31Z

source/iofs/iofs.go

+
+// Init prepares not initialized Iofs instance to read migrations from a
+// fs.ReadDirFS instance and a relative path.
+func (p *Iofs) Init(fsys fs.ReadDirFS, path string) error {


Why use fs.ReadDirFS instead of fs.FS? Does using ReadDir directly through the FS save iops? e.g. don't need to open a file, stat it, and finally readdir?

Does this optimization reduce compatibility with different filesystems?

We're probably better off using fs.ReadDir

dhui · 2020-11-14T05:08:16Z

source/errors.go

 // ErrDuplicateMigration is an error type for reporting duplicate migration
 // files.
 type ErrDuplicateMigration struct {
 	Migration
-	os.FileInfo
+	FileInfo


Why not use os.FileInfo or fs.FileInfo?

You can use fs.DirEntry.Info() to get the fs.FileInfo

johejo · 2020-11-14T13:00:42Z

source/iofs/iofs.go

+		}
+		file, err := e.Info()
+		if err != nil {
+			continue


Is continue OK like any other error case?

According to https://tip.golang.org/pkg/io/fs/#DirEntry an error in this case may mean:

If the file has been removed or renamed since the directory read, Info may return an error satisfying errors.Is(err, ErrNotExist).

But I think it can also just represent any other IO error, in e.g. the case the FS interface is backed by a real FS and not embed. I think ignoring the error and continueing is wrong here, instead any IO error should be reported back to the caller of the function I think (also in case of DefaultParse error above)

Agreed that errors should be returned and not ignored. Ignored errors will make bugs harder to debug.

johejo · 2020-11-14T13:02:24Z

Thanks for the great review.
I got a better understanding of io/fs.

… the changes in ErrDuplicateMigration

slimsag · 2020-11-14T19:55:11Z

source/iofs/README.md

+
+Driver with file system interface (`io/fs#FS`) supported from Go 1.16.
+
+This Driver cannot be used with Go versions 1.15 and below.


suggestion: Reword this to something like just:

Package iofs provides a Go 1.16+ io/fs driver which is compatible with Go 1.16+ file //embed directives.

Side note: I think all the documentation in this file would be better placed into a package-level godoc documentation comment, so that on https://pkg.go.dev this package shows up with package docs and usage examples similar to this and this

Feel free to ignore if this is not helpful :)

slimsag · 2020-11-14T20:29:00Z

source/iofs/README.md

+	"github.com/golang-migrate/migrate/v4/source/iofs"
+)
+
+//go:embed migrations


Suggested change

//go:embed migrations

//go:embed migrations/*.sql

May be useful to describe this to avoid people from accidentally embedding their own README.md, .gitignore, .DS_Store, etc.

dhui

Thanks for addressing my feedback!

dhui · 2020-11-15T04:24:10Z

source/iofs/iofs.go

+
+// Init prepares not initialized IoFS instance to read migrations from a
+// fs.FS instance and a relative path.
+func (p *IoFS) Init(fsys fs.FS, path string) error {


What are your thoughts about moving most of this implementation to a PartialDriver, similar to httpfs? We'd also need to rename WithInstance() to New() and changeIoFS to driver. That way other drivers can embed the partial driver and implement their own Open() receiver function.

…rtialDriver

dhui · 2020-11-16T04:38:04Z

.golangci.yml

@@ -1,6 +1,6 @@
 run:
  # timeout for analysis, e.g. 30s, 5m, default is 1m
-  timeout: 2m
+  timeout: 5m


Leave this at 2m I'll re-run any CI jobs/tests that fail due to timeout issues.

dhui · 2020-11-16T04:40:11Z

source/iofs/example_test.go

+//go:embed testdata/migrations/*.sql
+var fs embed.FS
+
+func Example() {


Thanks for using executable examples!

dhui · 2020-11-16T04:45:52Z

source/file/file.go

-	url  string
-	path string
-}
+type File = file


Is a type alias necessary? I think you can define type struct File in both the go 1.16 and non-1.16 files.

What kind of struct do you envision?
Sorry, I can't understand well.

I think you can avoid the type alias by doing:

// In file_go115.go type File struct { httpfs.PartialDriver url string path string } // In file_go116.go type File struct { iofs.PartialDriver url string path string }

dhui · 2020-11-16T04:46:22Z

source/file/file_go116.go

+)
+
+type file struct {
+	iofs.Driver


Thanks for doing this!

dhui · 2020-11-16T04:46:50Z

source/iofs/iofs.go

+)
+
+// Driver is a source driver that wraps io/fs#FS.
+type Driver struct {


Don't export this. e.g. rename to driver

If did not export it, you will not be able to embed it in the file driver.
httpfs.PartialDriver is public now.

I understand it, I will separate into PartialDriver and driver.

dhui · 2020-11-16T04:47:40Z

source/iofs/iofs.go

+}
+
+// NewDriver returns a new Driver from io/fs#FS and a relative path.
+func NewDriver(fsys fs.FS, path string) (source.Driver, error) {


Rename to New to follow the same pattern as httpfs.New()

dhui · 2020-11-16T04:49:03Z

source/iofs/iofs.go

+
+// Init prepares not initialized IoFS instance to read migrations from a
+// io/fs#FS instance and a relative path.
+func (d *Driver) Init(fsys fs.FS, path string) error {


Move all of the driver receiver methods (except for Open()) to PartialDriver to follow the same patter than httpfs uses.

This reverts commit 2cd12a6.

…driver

dhui

Thanks for making the PartialDriver change! One last thing and we're good to go!

dhui · 2020-11-18T05:16:12Z

source/iofs/iofs.go

-func (d *Driver) Open(url string) (source.Driver, error) {
-	panic("iofs: Driver does not support open with url")
+func (d *driver) Open(url string) (source.Driver, error) {
+	panic("iofs: driver does not support open with url")


Don't panic and instead return an error: return nil, errors.New("Open() cannot be called on the iofs passthrough driver")

This reverts commit fedbe42.

jeffwidman · 2021-03-16T05:24:15Z

source/iofs/iofs_test.go

+
+func Test(t *testing.T) {
+	//go:embed testdata/migrations/*.sql
+	var fs embed.FS


I'm just curious--how did this even work?

Per the golang docs:

The //go:embed directive can be used with both exported and unexported variables, depending on whether the package wants to make the data available to other packages. It can only be used with global variables at package scope, not with local variables.

To me this looks like a local variable, not a global variable at package scope... What am I missing?

Local variables were still supported when implementing this PR.
golang/go#43216
It is a remnant of that.

Ah, thank you!

Add io/fs#FS Driver golang-migrate#471

489e6d3

dhui reviewed Nov 13, 2020

View reviewed changes

Refactor iofs.Iofs to be independent of httpfs

65cbb0f

dhui requested changes Nov 14, 2020

View reviewed changes

johejo commented Nov 14, 2020

View reviewed changes

Rename Iofs to IoFS, accepts fs.FS and use fs.DirEntry.Info() to undo…

e18e378

… the changes in ErrDuplicateMigration

johejo force-pushed the iofs_source_driver branch from cf97126 to e18e378 Compare November 14, 2020 13:05

slimsag reviewed Nov 14, 2020

View reviewed changes

dhui requested changes Nov 15, 2020

View reviewed changes

johejo added 9 commits November 15, 2020 13:38

fix documents and migration directory for testing

e9ae08d

Fix iofs.Driver so that it can be used by embedding it like httpfs.Pa…

416f878

…rtialDriver

iofs.Driver closes the file system if possible

45ec3dc

Rename receiver

3af8acf

Refactor file driver to use iofs.Driver

0e9937b

iofs build constraint is not needed so remove it

7fb518d

Properly return an error in a corner case

3c0290b

Increase golangci-lint timeout

2cd12a6

Fix example code

ca6628d

dhui requested changes Nov 16, 2020

View reviewed changes

johejo added 2 commits November 17, 2020 19:57

Revert "Increase golangci-lint timeout"

58e9837

This reverts commit 2cd12a6.

unexport driver, add iofs.PartialDriver, remove type alias from file …

321662e

…driver

dhui requested changes Nov 18, 2020

View reviewed changes

do not panic in Open, return error

02da3b6

dhui approved these changes Nov 18, 2020

View reviewed changes

dhui merged commit fedbe42 into golang-migrate:master Nov 18, 2020

johejo added a commit to johejo/migrate that referenced this pull request Nov 21, 2020

Revert "Add io/fs#FS Driver golang-migrate#471 (golang-migrate#472)"

67546bf

This reverts commit fedbe42.

johejo mentioned this pull request Nov 21, 2020

Revert "Add io/fs#FS Driver #471" #481

Merged

This was referenced Nov 21, 2020

"package io/fs is not in GOROOT" on Go 1.15 #480

Closed

feat: embed, io/fs support #471

Closed

jeffwidman reviewed Mar 16, 2021

View reviewed changes

renovate bot mentioned this pull request May 22, 2021

fix(deps): update module github.com/golang-migrate/migrate to v4 - autoclosed advenjourney/advenjourney#20

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add io/fs#FS Driver #471 #472

Add io/fs#FS Driver #471 #472

johejo commented Nov 12, 2020 •

edited

Loading

coveralls commented Nov 12, 2020 •

edited

Loading

dhui left a comment

dhui Nov 13, 2020

dhui Nov 13, 2020

dhui Nov 13, 2020

dhui left a comment

dhui Nov 13, 2020

dhui Nov 14, 2020

dhui Nov 14, 2020

johejo Nov 14, 2020

slimsag Nov 14, 2020

dhui Nov 15, 2020

johejo commented Nov 14, 2020

slimsag Nov 14, 2020

slimsag Nov 14, 2020

dhui left a comment

dhui Nov 15, 2020

dhui Nov 16, 2020

dhui Nov 16, 2020

dhui Nov 16, 2020

johejo Nov 16, 2020

dhui Nov 16, 2020

dhui Nov 16, 2020

dhui Nov 16, 2020

johejo Nov 16, 2020

johejo Nov 16, 2020

dhui Nov 16, 2020

dhui Nov 16, 2020

dhui left a comment

dhui Nov 18, 2020

jeffwidman Mar 16, 2021

johejo Mar 18, 2021

jeffwidman Mar 19, 2021


		Driver with file system interface (`io/fs#FS`) supported from Go 1.16.

		This Driver cannot be used with versions below Go 1.15.


		Driver with file system interface (`io/fs#FS`) supported from Go 1.16.

		This Driver cannot be used with Go versions 1.15 and below.

Add io/fs#FS Driver #471 #472

Add io/fs#FS Driver #471 #472

Conversation

johejo commented Nov 12, 2020 • edited Loading

coveralls commented Nov 12, 2020 • edited Loading

Pull Request Test Coverage Report for Build 929

💛 - Coveralls

dhui left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhui left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johejo commented Nov 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhui left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhui left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johejo commented Nov 12, 2020 •

edited

Loading

coveralls commented Nov 12, 2020 •

edited

Loading