storage: Fix read all content at once when readdir

[bokket]

Signed-off-by: bokket [email protected]
fix: #16

[Xuanwo]

s.hdfs.ReadDir(input.rp) always read all content at once, see code here: https://github.com/colinmarc/hdfs/blob/master/readdir.go

func (c *Client) ReadDir(dirname string) ([]os.FileInfo, error) {
	f, err := c.Open(dirname)
	if err != nil {
		return nil, err
	}

	return f.Readdir(0)
}

Instead, we need to open the dir and read them via count.

And we don't need to store fileList/counter in listDirInput.

[bokket]

So count is up to us? For example const defaultListObjectCount=100
Otherwise, we still need to readDir to get the slice and calculate the count

[Xuanwo]

What we need is limit instead of count, there is no need for list to calculate the count.

We only need to f.Readdir(limit) and return IteratorDone if we meet io.EOF.

Ref: https://github.com/colinmarc/hdfs/blob/master/file_reader.go#L255-L257

[bokket]

Yes, this is slightly confusing, and I thought of using a constant to represent the read upper limit? Maybe there is something wrong with my description

[Xuanwo]

Maybe we need to store dir in page.Status?

• If we open dir every time in listDirNext, we can only list first 100 files.
• We can close dir before returning IterateDone

[bokket]

• If we open dir every time in listDirNext, we can only list first 100 files.

Maybe we could increase the limit each time until we reach EOF ? but that would repeat the read of the previous file.
So maybe we could probably use a slicing process to weed out the files before 100 each time. Do you think it would ok?

[JinnyYi]

Maybe we should use errors.Is to assert error's type?

[Prnyself]

remove the blank line

[Prnyself]

We should add nil check for err before assign f.Close() to it.

In this way, the real err may be covered by f.Close()

[Prnyself]

take https://github.com/beyondstorage/go-service-ftp/blob/master/storage.go#L111 as a reference