S3 gw fails to perform get-object or list-objects buckets

[MaxGelbakhiani]

Sometimes I get 404 in responce to list-objects and get-object s3 commands. Few requests in row could be susedull and then fail.d, the second one did not.

max@max0-0413:~/xk6-neofs$ aws s3api list-objects --bucket 424244  --endpoint http://max0-0412:8084
{
    "Contents": [
        {
            "Key": "748389e6-8cfa-4778-bfd1-a7141947630b",
            "LastModified": "2023-07-13T04:21:09+00:00",
            "ETag": "9adb29b748caaeec3f9086e01443e5520e7d9c00ea6a5bacd73249c4c504628a",
            "Size": 109,
            "Owner": {
                "DisplayName": "NRLBBncQxwvAyQpa1fnc7WSwyKcZUe2LuU",
                "ID": "NRLBBncQxwvAyQpa1fnc7WSwyKcZUe2LuU"
            }
        }
    ],
    "RequestCharged": null
}
max@max0-0413:~/xk6-neofs$
max@max0-0413:~/xk6-neofs$ aws s3api list-objects --bucket 424244  --endpoint http://max0-0412:8084

An error occurred (NoSuchBucket) when calling the ListObjects operation: The specified bucket does not exist
max@max0-0413:~/xk6-neofs$

S3 gw logs:

Jul 13 05:02:56 max0-0412 neofs-s3-gw[533464]: 2023-07-13T05:02:56.729Z        info        api/router.go:162        call method        {"status": 200, "host": "max0-0412:8084", "request_id": "69bd45bc-365c-48c1-b3c2-22eebe8e53e6", "method": "ListObjectsV1", "bucket": "424244", "object": "", "description": "OK"}
Jul 13 05:04:18 max0-0412 neofs-s3-gw[533464]: 2023-07-13T05:04:18.266Z        debug        layer/layer.go:355        bucket not found        {"error": "nns: couldn't resolve container '424244': contract invocation: connection lost before registering response channel"}
Jul 13 05:04:18 max0-0412 neofs-s3-gw[533464]: 2023-07-13T05:04:18.266Z        error        handler/util.go:29        call method        {"status": 404, "request_id": "3ae139fd-9361-4313-ae51-78b1aa15c8ad", "method": "ListObjectsV1", "bucket": "424244", "object": "", "description": "could not get bucket info", "error": "NoSuchBucket: 404 => The specified bucket does not exist"}

Bucket is there, even 2 of them:

max@max0-0413:~/xk6-neofs$
max@max0-0413:~/xk6-neofs$ aws s3api list-buckets  --endpoint http://max0-0412:8084
{
    "Buckets": [
        {
            "Name": "BtxYTGik2gosNDEw4uzK4WruD2K4D6PBb95q7NEa28R7",
            "CreationDate": "2023-07-13T04:17:26+00:00"
        },
        {
            "Name": "424244",
            "CreationDate": "2023-07-13T04:20:29+00:00"
        }
    ],
    "Owner": {
        "DisplayName": "Nge3U4wJpDGK2BWGfH5VcZ5PAbC6Ro7GHY",
        "ID": "Nge3U4wJpDGK2BWGfH5VcZ5PAbC6Ro7GHY"
    }
}

get-object sucessfull after list-buckets at least once:

max@max0-0413:~/xk6-neofs$
max@max0-0413:~/xk6-neofs$ aws s3api get-object --bucket 424244 --key 748389e6-8cfa-4778-bfd1-a7141947630b --endpoint http://max0-0412:8084 ~/hosts_from_s3
{
    "LastModified": "2023-07-13T04:21:09+00:00",
    "ContentLength": 109,
    "ETag": "9adb29b748caaeec3f9086e01443e5520e7d9c00ea6a5bacd73249c4c504628a",
    "ContentType": "text/plain; charset=utf-8",
    "Metadata": {},
    "TagCount": 0
}
max@max0-0413:~/xk6-neofs$ aws s3api get-object --bucket 424244 --key 748389e6-8cfa-4778-bfd1-a7141947630b --endpoint http://max0-0412:8084 ~/hosts_from_s3

An error occurred (NoSuchBucket) when calling the GetObject operation: The specified bucket does not exist
max@max0-0413:~/xk6-neofs$

S3 gw logs:

Jul 13 05:06:30 max0-0412 neofs-s3-gw[533464]: 2023-07-13T05:06:30.299Z        info        api/router.go:162        call method        {"status": 200, "host": "max0-0412:8084", "request_id": "037ab7aa-24e1-487e-96a2-730b928f0cdd", "method": "ListBuckets", "bucket": "", "object": "", "description": "OK"}
Jul 13 05:07:21 max0-0412 neofs-s3-gw[533464]: 2023-07-13T05:07:21.412Z        debug        layer/layer.go:499        get object        {"reqId": "821e3302-0e5e-40dc-bbb0-3857b80a8073", "bucket": "424244", "cid": "Fs4PTqqnRAt4pSV1cdGfNu4bqnZSFiFzS7xKmXzszUDY", "object": "748389e6-8cfa-4778-bfd1-a7141947630b", "oid": "9tQ3P4nFKPdxuDk3V77BGfuV97C35oLFRTyxg2HWE9Pu"}
Jul 13 05:07:21 max0-0412 neofs-s3-gw[533464]: 2023-07-13T05:07:21.429Z        info        api/router.go:162        call method        {"status": 200, "host": "max0-0412:8084", "request_id": "821e3302-0e5e-40dc-bbb0-3857b80a8073", "method": "GetObject", "bucket": "424244", "object": "748389e6-8cfa-4778-bfd1-a7141947630b", "description": "OK"}
Jul 13 05:08:17 max0-0412 neofs-s3-gw[533464]: 2023-07-13T05:08:17.901Z        debug        layer/layer.go:355        bucket not found        {"error": "nns: couldn't resolve container '424244': contract invocation: connection lost before registering response channel"}
Jul 13 05:08:17 max0-0412 neofs-s3-gw[533464]: 2023-07-13T05:08:17.901Z        error        handler/util.go:29        call method        {"status": 404, "request_id": "43ec6a1a-db61-4ce2-93ac-d6a3f3d59c35", "method": "GetObject", "bucket": "424244", "object": "748389e6-8cfa-4778-bfd1-a7141947630b", "description": "could not get bucket info", "error": "NoSuchBucket: 404 => The specified bucket does not exist"}

Versions:

root@max0-0412:~# neofs-s3-gw --version
NeoFS S3 Gateway
Version: v0.27.1
GoVersion: go1.20.4
root@max0-0412:~# neofs-node --version
NeoFS Storage node
Version: v0.37.0
GoVersion: go1.20.4
root@max0-0412:~#

Network consists of 4 IR nodes, 22 storage nodes and 1 S3-gw.

Configs from IR, Storage and S3-gw nodes are enclosed:
NeoFS-configs.zip

[cthulhu-rider]

i suggest next improvements:

1. try to reconnect when connections falls
2. write exact reason why connection was lost (NeoGo WS client started to support this recently)

[roman-khimov]

BTW, reconnection problem is exactly the same as nspcc-dev/neofs-http-gw#256, it just wasn't created in this repo (HTTP was supposed to be fixed first).

But we can rework the scheme to stop using ns package at the same time.

[roman-khimov]

After the recent changes we no longer support WSClient, while regular RPC client doesn't have this problem. For the current use of the RPC client working via regular interface is good enough, we don't need any WS-specific features. Therefore, this issue is no longer relevant (at least until we need a real WS client).