Failed to mark S3 object with deleted tag

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

deploy tiflash with S3 disagg on AWS

2. What did you expect to see? (Required)

3. What did you see instead (Required)

[2023/03/23 13:33:47.985 +00:00] [ERROR] [S3Common.cpp:126] ["tag=AWSXmlClient message=HTTP response code: 404\nResolved remote host IP address: 52.219.141.74\nRequest ID: \nException name: \nError message: No response body.\n5 response headers:\ncontent-type : application/xml\ndate : Thu, 23 Mar 2023 13:33:47 GMT\nserver : AmazonS3\nx-amz-id-2 : ZsI44c25a9l+8aV6iPsYs1Zmk5c3rASuzrqsS1CbiEL3Nbju1r5hO9YoKfgvicuyQnA7YXo+ZX8=\nx-amz-request-id : 67ZJJRMX9PREZGTX"] [source=AWSClient] [thread_id=19]
[2023/03/23 13:33:48.034 +00:00] [ERROR] [S3Common.cpp:126] ["tag=AWSXmlClient message=HTTP response code: 400\nResolved remote host IP address: 52.219.141.74\nRequest ID: NWPVXWV0071FA4N6\nException name: InvalidRequest\nError message: Unable to parse ExceptionName: InvalidRequest Message: This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes.\n7 response headers:\nconnection : close\ncontent-type : application/xml\ndate : Thu, 23 Mar 2023 13:33:47 GMT\nserver : AmazonS3\ntransfer-encoding : chunked\nx-amz-id-2 : mT5rgJ2SFb1y4ctSyzTpdb7yB/8JWMsD/UY+6Q1KENxEfJjJK4S6bdzlYHYPm3Ad5Fy+5pxGrNE=\nx-amz-request-id : NWPVXWV0071FA4N6"] [source=AWSClient] [thread_id=19]
[2023/03/23 13:33:48.035 +00:00] [ERROR] [Exception.cpp:90] ["Code: 11000, e.displayText() = DB::Exception: S3 CopyObject failed, bucket=jinhe-cloud-native-830511b9-tiflash root=2/ key=s172/data/dat_11_0 s3error=UNKNOWN s3msg=Unable to parse ExceptionName: InvalidRequest Message: This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes., e.what() = DB::Exception, Stack trace:\n\n\n       0x1b4d7c3\tStackTrace::StackTrace() [tiflash+28628931]\n                \tdbms/src/Common/StackTrace.cpp:23\n       0x7d8dfa9\tDB::Exception::Exception<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+131653545]\n                \tdbms/src/Common/Exception.h:53\n       0x7d87f3a\tDB::Exception DB::S3::fromS3Error<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(Aws::S3::S3Error const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+131628858]\n                \tdbms/src/Storages/S3/S3Common.h:45\n       0x7d83948\tDB::S3::rewriteObjectWithTagging(DB::S3::TiFlashS3Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+131610952]\n                \tdbms/src/Storages/S3/S3Common.cpp:521\n       0x7e9b27d\tDB::S3::S3GCManager::lifecycleMarkDataFileDeleted(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) [tiflash+132756093]\n                \tdbms/src/Storages/S3/S3GCManager.cpp:432\n       0x7e9adbb\tDB::S3::S3GCManager::cleanOneLock(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::S3::S3FilenameView const&, Aws::Utils::DateTime const&) [tiflash+132754875]\n                \tdbms/src/Storages/S3/S3GCManager.cpp:321\n       0x7e9c9c3\tstd::__1::__function::__func<DB::S3::S3GCManager::cleanUnusedLocks(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, std::__1::unordered_set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, Aws::Utils::DateTime const&)::$_7, std::__1::allocator<DB::S3::S3GCManager::cleanUnusedLocks(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, std::__1::unordered_set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, Aws::Utils::DateTime const&)::$_7>, DB::S3::PageResult (Aws::S3::Model::Object const&)>::operator()(Aws::S3::Model::Object const&) [tiflash+132762051]\n                \t/data3/jaysonhuang/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345\n       0x7d84869\tDB::S3::listPrefix(DB::S3::TiFlashS3Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::function<DB::S3::PageResult (Aws::S3::Model::Object const&)>) [tiflash+131614825]\n                \tdbms/src/Storages/S3/S3Common.cpp:668\n       0x7e9a478\tDB::S3::S3GCManager::cleanUnusedLocks(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, std::__1::unordered_set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, Aws::Utils::DateTime const&) [tiflash+132752504]\n                \tdbms/src/Storages/S3/S3GCManager.cpp:233\n       0x7e998c1\tDB::S3::S3GCManager::runForStore(unsigned long) [tiflash+132749505]\n                \tdbms/src/Storages/S3/S3GCManager.cpp:160\n       0x7e98673\tDB::S3::S3GCManager::runOnAllStores() [tiflash+132744819]\n                \tdbms/src/Storages/S3/S3GCManager.cpp:126\n       0x7e9d9a1\tstd::__1::__function::__func<DB::S3::S3GCManagerService::S3GCManagerService(DB::Context&, std::__1::shared_ptr<pingcap::pd::IClient>, std::__1::shared_ptr<DB::OwnerManager>, std::__1::shared_ptr<DB::S3::IS3LockClient>, DB::S3::S3GCConfig const&)::$_23, std::__1::allocator<DB::S3::S3GCManagerService::S3GCManagerService(DB::Context&, std::__1::shared_ptr<pingcap::pd::IClient>, std::__1::shared_ptr<DB::OwnerManager>, std::__1::shared_ptr<DB::S3::IS3LockClient>, DB::S3::S3GCConfig const&)::$_23>, bool ()>::operator()() [tiflash+132766113]\n                \t/data3/jaysonhuang/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345\n       0x7cdbe6d\tDB::BackgroundProcessingPool::threadFunction(unsigned long) [tiflash+130924141]\n                \tdbms/src/Storages/BackgroundProcessingPool.cpp:234\n       0x7cdc855\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, DB::BackgroundProcessingPool::BackgroundProcessingPool(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::$_1> >(void*) [tiflash+130926677]\n                \t/data3/jaysonhuang/tiflash-env-13/sysroot/bin/../include/c++/v1/thread:291\n  0x7f4810f93609\tstart_thread [libpthread.so.0+34313]\n                \t/build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477\n  0x7f4810dda133\tclone [libc.so.6+1175859]\n                \t/build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95"] [source="void DB::BackgroundProcessingPool::threadFunction(size_t)"] [thread_id=19]

4. What is your TiFlash version? (Required)

master

[JaySon-Huang]

ref #6882