TiCDC cannot use azblob:// and gcs:// cloud storage sink since v6.5.3 due to use of ObjPrefix

[kennytm]

What did you do?

cdc cli changefeed create --sink-uri="azure://bucket/prefix?protocol=canal-json&enable-tidb-extension=true" -c cf

What did you expect to see?

Changefeed created and run normally

What did you see instead?

The changefeed failed with error code CDC:ErrProcessorUnknown with message azure storage not support ObjPrefix for now.

The log contained an error:

[2024/02/05 17:19:53.999 +00:00] [ERROR] [dml_worker.go:164] ["failed to write schema file to external storage"] [workerID=13] [namespace=default] [changefeed=cf] [error="azure storage not support ObjPrefix for now"] [errorVerbose="azure storage not support ObjPrefix for now\n
github.com/pingcap/tidb/br/pkg/storage.(*AzureBlobStorage).WalkDir\n
\tgithub.com/pingcap/[email protected]/br/pkg/storage/azblob.go:367\n
github.com/pingcap/tiflow/pkg/sink/cloudstorage.(*FilePathGenerator).CheckOrWriteSchema\n
\tgithub.com/pingcap/tiflow/pkg/sink/cloudstorage/path.go:203\n
github.com/pingcap/tiflow/cdc/sinkv2/eventsink/cloudstorage.(*dmlWorker).flushMessages\n
\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/cloudstorage/dml_worker.go:162\n
github.com/pingcap/tiflow/cdc/sinkv2/eventsink/cloudstorage.(*dmlWorker).run.func1\n
\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/cloudstorage/dml_worker.go:135\n
golang.org/x/sync/errgroup.(*Group).Go.func1\n
\tgolang.org/x/[email protected]/errgroup/errgroup.go:75\n
runtime.goexit\n\truntime/asm_amd64.s:1594"]

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(irrelevant)

Upstream TiKV version (execute tikv-server --version):

(irrelevant)

TiCDC version (execute cdc version):

v6.5.6

[kennytm]

The problem was introduced by #8921 (since v6.5.3), which started to use the ObjPrefix option in WalkDir (it still existed as of v6.5.8).

tiflow/pkg/sink/cloudstorage/path.go

Lines 203 to 206 in c86e701

	err = f.storage.WalkDir(ctx, &storage.WalkOption{
	SubDir: subDir, /* use subDir to prevent walk the whole storage */
	ObjPrefix: subDir + "schema_",
	}, func(path string, _ int64) error {

The ObjPrefix option was first introduced by pingcap/tidb#33409 since TiDB v6.1.0. This was supported on s3:// and local:// option, while on azblob:// and gcs:// using ObjPrefix will cause an "unsupported" error. Then, since TiDB v7.0.0 by pingcap/tidb#42050 they start to recognize ObjPrefix.

This means TiCDC in the version range [v6.5.3, v6.5.8] are unable to write to azblob:// and gcs://.

v6.6.0 seems not affected because #8881 was not cherry-picked to release-6.6. v7.x are not affected because TiDB "fixed" the error.

I think the ObjPrefix fix introduced by pingcap/tidb#42050 should be cherry-picked to TiDB's release-6.5 and -6.6. Alternatively, avoid using the ObjPrefix option, but that will degrade performance of S3 sink.

[fubinzh]

/severity major

[CharlesCheung96]

closed by #10732

[flowbehappy]

This issue only affects v6.5.