fix minio storage iterator path

[fuxiaohei]

minio storage iterator shows different behavior with local fs iterator.

in local fs storage:

s.IterateObjects("prefix", func(path,obj)
     println(path) // show "prefix/xxx.file"
})

in minio storage:

s.IterateObjects("prefix", func(path,obj)
     println(path) // show "xxx.file"
})

I think local fs is correct, minio use wrong basePath to trim storage path prefix.

[lunny]

Are there any features affected by this?

[wxiaoguang]

Interfaces should have the same behaviors.

[wxiaoguang]

ps: I guess we need some common test code for different storages, to make sure they behave consistently

[fuxiaohei]

Are there any features affected by this?

I think there are no affects. All are s.IterateObjects("")

[lunny]

There is a test TestLocalStorageIterator but only for local disk, maybe we can have another test for minio with the same test data.

[lunny]

tests needed.

[wxiaoguang]

How about using a common function to test all storages? Then it could guarantee that they have the consistent behavior.

ps: when rewriting the queue, I introduced a general test function, then all queues are tested by it, so all queues should have the same behavior now

[wxiaoguang]

I think util.PathJoinRelX already removes the prefix slash. no need to trim it ahead?

[fuxiaohei]

If p = "/" and m.base = "", it generates "./" which is invalid to minio api, shows Received unexpected error: Object name cannot be empty

[wxiaoguang]

Nope, it never generates "./" here.

See the comment of "PathJoinRel" family functions, it only outputs "", "." or "foo/bar" path

Even if you trim "/" here, if the input p is "./" , you still get "." and later your code append another suffix slash.

So ideally, I think it should be:

p = util.PathJoinRelX(m.basePath, p)
if p == "." {  p = "" }   // minio doesn't use dot as relative path
return p

I guess there could be a test case for it.

[fuxiaohei]

if p == "." { p = "" } is ok

[GiteaBot]

I was unable to create a backport for 1.19. @fuxiaohei, please send one manually. 🍵

go run ./contrib/backport 24691
...  // fix git conflicts if any
go run ./contrib/backport --continue

[lunny]

Since v1.19 hasn't prefix, it's unnecessary to backport.

[DmitryFrolovTri]

@fuxiaohei @lunny I think this broke local tests at least for me - when they are run locally here is what I get consistently when I run make test tried with go1.19, go1.19.9, go 1.20.2
Does anything in gitea actually start the http://127.0.0.1:9000/gitea/ for those tests?

Can we modify the standard testing that it avoids running minio test if no minio? May be to run this test
we define an env variable minio sort of like with sqlite? What is your view? Or may be nominio
or add some code that verifies that minio is actually running and if not simply skips the test?

2023/05/17 08:27:10 ...les/storage/minio.go:101:NewMinioStorage() [I] Creating Minio storage at 127.0.0.1:9000:gitea with base path 
--- FAIL: TestMinioStorageIterator (4.98s)
    storage_test.go:15: 
        	Error Trace:	/home/puni/sources/mine/gitea-sapk-fork-test/modules/storage/storage_test.go:15
        	            				/home/puni/sources/mine/gitea-sapk-fork-test/modules/storage/minio_test.go:11
        	Error:      	Received unexpected error:
        	            	Put "http://127.0.0.1:9000/gitea/": dial tcp 127.0.0.1:9000: connect: connection refused
        	Test:       	TestMinioStorageIterator
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0xa35537]

goroutine 47 [running]:
testing.tRunner.func1.2({0xb7f5e0, 0x10d6330})
	/home/puni/.gvm/gos/go1.20.2/src/testing/testing.go:1526 +0x24e
testing.tRunner.func1()
	/home/puni/.gvm/gos/go1.20.2/src/testing/testing.go:1529 +0x39f
panic({0xb7f5e0, 0x10d6330})
	/home/puni/.gvm/gos/go1.20.2/src/runtime/panic.go:884 +0x213
code.gitea.io/gitea/modules/storage.testStorageIterator(0xc00036c4e0, {0xc16a8a?, 0x5?}, {0xbec960?, 0xc00036e080?})
	/home/puni/sources/mine/gitea-sapk-fork-test/modules/storage/storage_test.go:27 +0x3d7
code.gitea.io/gitea/modules/storage.TestMinioStorageIterator(0x0?)
	/home/puni/sources/mine/gitea-sapk-fork-test/modules/storage/minio_test.go:11 +0x87
testing.tRunner(0xc00036c4e0, 0xc739d8)
	/home/puni/.gvm/gos/go1.20.2/src/testing/testing.go:1576 +0x10b
created by testing.(*T).Run
	/home/puni/.gvm/gos/go1.20.2/src/testing/testing.go:1629 +0x3ea
FAIL	code.gitea.io/gitea/modules/storage	5.635s

to run tests I clone a fresh copy of repo and then:

TAGS="bindata sqlite sqlite_unlock_notify" make build
TAGS="bindata sqlite sqlite_unlock_notify" make test