WalkDir of gcs only iterate at most 1000 keys

[july2993]

Bug Report

ref

tidb/br/pkg/storage/gcs.go

Lines 183 to 197 in 891517f

	maxKeys := int64(1000)
	if opt.ListCount > 0 {
	maxKeys = opt.ListCount
	}
	prefix := path.Join(s.gcs.Prefix, opt.SubDir)
	if len(prefix) > 0 && !strings.HasSuffix(prefix, "/") {
	prefix += "/"
	}
	query := &storage.Query{Prefix: prefix}
	// only need each object's name and size
	query.SetAttrSelection([]string{"Name", "Size"})
	iter := s.bucket.Objects(ctx, query)
	for i := int64(0); i != maxKeys; i++ {

note maxKeys is 1000, and we will end the loop after iterate maxKeys keys

1. Minimal reproduce step (Required)

import data more than 1000 keys/files

2. What did you expect to see? (Required)

import all data under directory.

3. What did you see instead (Required)

only scan at most 1000 files

4. What is your TiDB version? (Required)

[kennytm]

maxKeys is 1000 only if opt.ListCount is left has default (0).

[glorv]

I think this do is a bug. For s3 the maxKeys is used to limit the fetch keys for each request but still ensure to visit all the keys. But gcs only visit maxKeys keys and then stop.

[glorv]

/cc @3pointer

[github-actions]

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.