cdc/cloudstorage: potential memory leak in cloud storage

[wenyihu6]

Describe the problem

We clear s.files right after we call s.flushFile on every files (no errors returned yet since we were able to send all files into the asyncflusherCh)

cockroach/pkg/ccl/changefeedccl/sink_cloudstorage.go

Lines 643 to 650 in 388bb8b

	s.files.Ascend(func(i btree.Item) (wantMore bool) {
	err = s.flushFile(ctx, i.(*cloudStorageSinkFile))
	return err == nil
	})
	if err != nil {
	return err
	}
	s.files.Clear(true /* addNodesToFreeList */)

. We then clear all files s.Clear() since no error returned

cockroach/pkg/ccl/changefeedccl/sink_cloudstorage.go

Lines 796 to 797 in e9a6d3d

	case s.asyncFlushCh <- flushRequest{file: file, dest: dest}:
	return nil

. But this is async_flush and it is possible that they later return an error

cockroach/pkg/ccl/changefeedccl/sink_cloudstorage.go

Lines 824 to 828 in e9a6d3d

	if err != nil {
	log.Errorf(ctx, "error flushing file to storage: %s", err)
	s.asyncFlushErr = err
	return err
	}

. We then left with some of the files uncleaned. And s.Close() were not able to clean the files properly since files were removed in s.Clear() already.

See more in https://github.com/cockroachlabs/support/issues/3068.
Repro: #129935

Jira issue: CRDB-41791

[blathers-crl]

Hi @wenyihu6, please add branch-* labels to identify which branch(es) this C-bug affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

cc @cockroachdb/cdc

[blathers-crl]

Based on the specified backports for linked PR #130204, I applied the following new label(s) to this issue: branch-release-24.1.5-rc, branch-release-24.2.2-rc. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

Based on the specified backports for linked PR #130204, I applied the following new label(s) to this issue: branch-release-23.2.12-rc. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}