[CI] S3SnapshotRepoTestKitIT testRepositoryAnalysis failing #102294
Labels
:Distributed Coordination/Snapshot/Restore
Anything directly related to the `_snapshot/*` APIs
medium-risk
An open issue or test failure that is a medium risk to future releases
stalled
Team:Distributed (Obsolete)
Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
>test-failure
Triaged test failures from CI
Comments
kkrik-es
added
:Distributed Coordination/Snapshot/Restore
elasticsearchmachine
added
blocker
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
|
Hmm there's not much to go on here. I opened #102299 to make things more verbose. |
DaveCTurner
added a commit
that referenced
this issue
Nov 16, 2023
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
Nov 16, 2023
henningandersen
added
medium-risk
|
I realised that the logs in the build scan don't include the node logs, which contain a little more detail: |
|
Hmm |
DaveCTurner
added a commit
that referenced
this issue
Nov 17, 2023
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
Nov 17, 2023
Moves the logging added in elastic#102299 onto the test cluster where it belongs, and adds wire logging to this test suite too. Relates elastic#102294
|
I've tried various ideas with our own simulated S3 test fixture to see if I can reproduce the |
przemekwitek
pushed a commit
to przemekwitek/elasticsearch
that referenced
this issue
Nov 20, 2023
Moves the logging added in elastic#102299 onto the test cluster where it belongs, and adds wire logging to this test suite too. Relates elastic#102294
|
@DaveCTurner I'd like check with you whether we can close this issue. It has not seen any failure for last 6 months. I think at this point it's OK to view it as not actionable and hence closing. I also think it is low-risk since it feels more infrastructureish (either our s3 fixuture or java's http server). |
|
I believe this failure came from a test running against the real S3, not our own infra, so I remain concerned that there is some surprising real-life behaviour that we're not handling correctly. |
|
I see this in the console log
Note the host is |
|
Nevermind. The above is host is for the elasticsearch API and the host is the ES node |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Hi again, @DaveCTurner - I reproduced the original I hope this is helpful and on-topic. Let me know if it should actually be a new issue. Setting up the keystore: export ES_PATH_CONF=/Users/ppatterson/src/elasticsearch/build/testclusters/runTask-0/config
echo ${AWS_ACCESS_KEY_ID} | bin/elasticsearch-keystore add --stdin s3.client.s3-client.access_key -f
echo ${AWS_SECRET_ACCESS_KEY} | bin/elasticsearch-keystore add --stdin s3.client.s3-client.secret_key -f
bin/elasticsearch-keystore list
curl -X POST "localhost:9200/_nodes/reload_secure_settings?pretty" -H 'Content-Type: application/json' -d '{}'Creating the S3 repository: curl -X PUT "localhost:9200/_snapshot/my_s3_repository?pretty" -H 'Content-Type: application/json' -d'
{
"type": "s3",
"settings": {
"bucket": "metadaddy-elasticsearch2",
"client": "s3-client",
"region": "us-west-1",
"endpoint": "s3.us-west-1.amazonaws.com"
}
}
'I wrote a script to repeatedly hit the Repository Analysis API until it encountered an error: #! /bin/zsh
# Pass repo name as script argument
repo=$1
outfile=${repo}_output.txt
while true
do
curl -s --fail-with-body -X POST "localhost:9200/_snapshot/${repo}/_analyze?blob_count=10&max_blob_size=1mb&timeout=120s&pretty&error_trace=true" &>> ${outfile}
if [ $? -ne 0 ]
then
# Request failed
break
fi
sleep 1
doneEventually, the script terminated. Error response: {
"error" : {
"root_cause" : [
{
"type" : "amazon_s3_exception",
"reason" : "The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: FNCRS543R65YW12D; S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==; Proxy: null)",
"stack_trace" : "com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: FNCRS543R65YW12D; S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==; Proxy: null), S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CompleteMultipartUploadHandler.doEndElement(XmlResponsesSaxParser.java:1853)\n\tat com.amazonaws.services.s3.model.transform.AbstractHandler.endElement(AbstractHandler.java:52)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:619)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1737)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2908)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:635)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:113)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:551)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:890)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:826)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:134)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1225)\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:167)\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseCompleteMultipartUploadResponse(XmlResponsesSaxParser.java:502)\n\tat com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:316)\n\tat com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:313)\n\tat com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)\n\tat com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44)\n\tat com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30)\n\tat com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1794)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1477)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\tat com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:3630)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$completeMultipartUpload$13(S3BlobContainer.java:885)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:319)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.completeMultipartUpload(S3BlobContainer.java:885)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$2(S3BlobContainer.java:675)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:91)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)\n\tat [email protected]/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:362)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.getRegister(S3BlobContainer.java:917)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$1(S3BlobContainer.java:668)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseDroppingActionListener.onResponse(ActionListenerImplementations.java:274)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:400)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener.finish(RefCountingListener.java:122)\n\tat [email protected]/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:125)\n\tat [email protected]/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:77)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:113)\n\tat [email protected]/org.elasticsearch.core.Releasables$4.close(Releasables.java:161)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:152)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:147)\n\tat [email protected]/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:38)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\n"
}
],
"type" : "repository_verification_exception",
"reason" : "[my_s3_repository] Elasticsearch observed the storage system underneath this repository behaved incorrectly which indicates it is not suitable for use with Elasticsearch snapshots. Typically this happens when using storage other than AWS S3 which incorrectly claims to be S3-compatible. If so, please report this incompatibility to your storage supplier. Do not report Elasticsearch issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository. See [https://www.elastic.co/guide/en/elasticsearch/reference/master/repo-analysis-api.html] for further information about repository analysis, and [https://www.elastic.co/guide/en/elasticsearch/reference/master/repository-s3.html#repository-s3-compatible-services] for further information about support for S3-compatible repository implementations. Elasticsearch attempted to remove the data it wrote at [temp-analysis-yBmseJExSNiitdirP1iyaw] but may have left some behind. If so, please now remove this data manually.",
"caused_by" : {
"type" : "amazon_s3_exception",
"reason" : "The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: FNCRS543R65YW12D; S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==; Proxy: null)",
"stack_trace" : "com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: FNCRS543R65YW12D; S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==; Proxy: null), S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CompleteMultipartUploadHandler.doEndElement(XmlResponsesSaxParser.java:1853)\n\tat com.amazonaws.services.s3.model.transform.AbstractHandler.endElement(AbstractHandler.java:52)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:619)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1737)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2908)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:635)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:113)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:551)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:890)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:826)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:134)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1225)\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:167)\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseCompleteMultipartUploadResponse(XmlResponsesSaxParser.java:502)\n\tat com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:316)\n\tat com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:313)\n\tat com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)\n\tat com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44)\n\tat com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30)\n\tat com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1794)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1477)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\tat com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:3630)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$completeMultipartUpload$13(S3BlobContainer.java:885)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:319)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.completeMultipartUpload(S3BlobContainer.java:885)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$2(S3BlobContainer.java:675)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:91)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)\n\tat [email protected]/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:362)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.getRegister(S3BlobContainer.java:917)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$1(S3BlobContainer.java:668)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseDroppingActionListener.onResponse(ActionListenerImplementations.java:274)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:400)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener.finish(RefCountingListener.java:122)\n\tat [email protected]/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:125)\n\tat [email protected]/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:77)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:113)\n\tat [email protected]/org.elasticsearch.core.Releasables$4.close(Releasables.java:161)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:152)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:147)\n\tat [email protected]/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:38)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\n"
},
"stack_trace" : "org.elasticsearch.repositories.RepositoryVerificationException: [my_s3_repository] Elasticsearch observed the storage system underneath this repository behaved incorrectly which indicates it is not suitable for use with Elasticsearch snapshots. Typically this happens when using storage other than AWS S3 which incorrectly claims to be S3-compatible. If so, please report this incompatibility to your storage supplier. Do not report Elasticsearch issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository. See [https://www.elastic.co/guide/en/elasticsearch/reference/master/repo-analysis-api.html] for further information about repository analysis, and [https://www.elastic.co/guide/en/elasticsearch/reference/master/repository-s3.html#repository-s3-compatible-services] for further information about support for S3-compatible repository implementations. Elasticsearch attempted to remove the data it wrote at [temp-analysis-yBmseJExSNiitdirP1iyaw] but may have left some behind. If so, please now remove this data manually.\nCaused by: org.elasticsearch.transport.RemoteTransportException: [runTask-0][127.0.0.1:9300][cluster:admin/repository/analyze/register]\nCaused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: null; Status Code: 0; Error Code: NoSuchUpload; Request ID: FNCRS543R65YW12D; S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==; Proxy: null), S3 Extended Request ID: 4TeAgJEQ6UUWvcVRLK+4GOMwKNr63nAK67zvlZjpZ92LuRhMM63rkgNFYrDvdGEv0qa/QMbsKze0Es+J8Ji1hw==\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CompleteMultipartUploadHandler.doEndElement(XmlResponsesSaxParser.java:1853)\n\tat com.amazonaws.services.s3.model.transform.AbstractHandler.endElement(AbstractHandler.java:52)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:619)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1737)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2908)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:635)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:113)\n\tat java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:551)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:890)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:826)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:134)\n\tat java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1225)\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:167)\n\tat com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseCompleteMultipartUploadResponse(XmlResponsesSaxParser.java:502)\n\tat com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:316)\n\tat com.amazonaws.services.s3.model.transform.Unmarshallers$CompleteMultipartUploadResultUnmarshaller.unmarshall(Unmarshallers.java:313)\n\tat com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)\n\tat com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44)\n\tat com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30)\n\tat com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:69)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1794)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleSuccessResponse(AmazonHttpClient.java:1477)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1384)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\tat com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:3630)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$completeMultipartUpload$13(S3BlobContainer.java:885)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:319)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.completeMultipartUpload(S3BlobContainer.java:885)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$2(S3BlobContainer.java:675)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:91)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)\n\tat [email protected]/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:362)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.getRegister(S3BlobContainer.java:917)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer$CompareAndExchangeOperation.lambda$run$1(S3BlobContainer.java:668)\n\tat [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseDroppingActionListener.onResponse(ActionListenerImplementations.java:274)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:387)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:307)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat [email protected]/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:250)\n\tat [email protected]/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:400)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener.finish(RefCountingListener.java:122)\n\tat [email protected]/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:125)\n\tat [email protected]/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:77)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:113)\n\tat [email protected]/org.elasticsearch.core.Releasables$4.close(Releasables.java:161)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:152)\n\tat [email protected]/org.elasticsearch.action.support.RefCountingListener$1.onResponse(RefCountingListener.java:147)\n\tat [email protected]/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:38)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\n"
},
"status" : 500
}Here's the stack trace in a more digestible form (the three traces in the response are identical): This is actually a different error from the one I was seeing with Backblaze B2 in that it occurs when Why is the status code Looking at the stack trace, we can see that the operation being processed is
At the top of the stack trace we see |
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
Nov 4, 2024
A `CompleteMultipartUpload` action may fail after sending the `200 OK` response line. In this case the response body describes the error, and the SDK translates this situation to an exception with status code 0 but with the `ErrorCode` string set appropriately. This commit enhances the exception handling in `S3BlobContainer` to handle this possibility. Closes elastic#102294 Co-authored-by: [email protected]
|
Aha that does explain it indeed. Thanks for the thorough investigation and writeup. I opened #116212 to address this. |
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
Nov 5, 2024
A `CompleteMultipartUpload` action may fail after sending the `200 OK` response line. In this case the response body describes the error, and the SDK translates this situation to an exception with status code 0 but with the `ErrorCode` string set appropriately. This commit enhances the exception handling in `S3BlobContainer` to handle this possibility. Closes elastic#102294 Co-authored-by: Pat Patterson <[email protected]>
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
Nov 5, 2024
A `CompleteMultipartUpload` action may fail after sending the `200 OK` response line. In this case the response body describes the error, and the SDK translates this situation to an exception with status code 0 but with the `ErrorCode` string set appropriately. This commit enhances the exception handling in `S3BlobContainer` to handle this possibility. Closes elastic#102294 Co-authored-by: Pat Patterson <[email protected]>
elasticsearchmachine
pushed a commit
that referenced
this issue
Nov 5, 2024
A `CompleteMultipartUpload` action may fail after sending the `200 OK` response line. In this case the response body describes the error, and the SDK translates this situation to an exception with status code 0 but with the `ErrorCode` string set appropriately. This commit enhances the exception handling in `S3BlobContainer` to handle this possibility. Closes #102294 Co-authored-by: Pat Patterson <[email protected]>
elasticsearchmachine
pushed a commit
that referenced
this issue
Nov 5, 2024
A `CompleteMultipartUpload` action may fail after sending the `200 OK` response line. In this case the response body describes the error, and the SDK translates this situation to an exception with status code 0 but with the `ErrorCode` string set appropriately. This commit enhances the exception handling in `S3BlobContainer` to handle this possibility. Closes #102294 Co-authored-by: Pat Patterson <[email protected]>
|
For the record, the root cause of the repository analyzer's failure with Backblaze B2 is that B2's Thanks for your patience as I got to the bottom of this, @DaveCTurner, and thanks for creating an awesome tool to check compatibility! |
|
@metadaddy Thanks for the great Investigation! TIL. I wonder whether it is worthwhile to also report this as a bug for aws-sdk-java. While the v1 client is on its way out, it seems they are still taking issue reports. |
|
Hi @ywangd - I'm not sure it's a bug in |
jozala
pushed a commit
that referenced
this issue
Nov 13, 2024
A `CompleteMultipartUpload` action may fail after sending the `200 OK` response line. In this case the response body describes the error, and the SDK translates this situation to an exception with status code 0 but with the `ErrorCode` string set appropriately. This commit enhances the exception handling in `S3BlobContainer` to handle this possibility. Closes #102294 Co-authored-by: Pat Patterson <[email protected]>
alexey-ivanov-es
pushed a commit
to alexey-ivanov-es/elasticsearch
that referenced
this issue
Nov 28, 2024
A `CompleteMultipartUpload` action may fail after sending the `200 OK` response line. In this case the response body describes the error, and the SDK translates this situation to an exception with status code 0 but with the `ErrorCode` string set appropriately. This commit enhances the exception handling in `S3BlobContainer` to handle this possibility. Closes elastic#102294 Co-authored-by: Pat Patterson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Build scan:
https://gradle-enterprise.elastic.co/s/d3hlyfatxvzxc/tests/:x-pack:plugin:snapshot-repo-test-kit:qa:s3:javaRestTest/org.elasticsearch.repositories.blobstore.testkit.S3SnapshotRepoTestKitIT/testRepositoryAnalysis
Reproduction line:
Applicable branches:
main
Reproduces locally?:
No
Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.repositories.blobstore.testkit.S3SnapshotRepoTestKitIT&tests.test=testRepositoryAnalysis
Failure excerpt:
The text was updated successfully, but these errors were encountered: