Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failure to delete an object #2971

Closed
roman-khimov opened this issue Oct 18, 2024 · 5 comments
Closed

Failure to delete an object #2971

roman-khimov opened this issue Oct 18, 2024 · 5 comments
Assignees
Labels
bug Something isn't working I4 No visible changes neofs-storage Storage node application issues S4 Routine U2 Seriously planned
Milestone

Comments

@roman-khimov
Copy link
Member

Expected Behavior

No errors.

Current Behavior

Oct 18 09:22:07 metis2 neofs-node[2993338]: 2024-10-18T09:22:07.680Z        warn        metabase/delete.go:112        failed to delete object        {"shard_id": "YMGgmv7yj3y5s7K3Xr94kN", "addr": "7eyrRq3wALGwAEwuwSKfH3DcMFVU7jLPhqWmAnmyM7wm/3rvsak6c3eCfM6J33obcHU5G1uzZUaBqnvXjJKBD4FN2", "error": "proto: cannot parse invalid wire-format data"}

Possible Solution

Let objects die in any event, even if they're malformed (which is strange, but that's what it is).

Steps to Reproduce (for bugs)

Unknown

Context

Mainnet.

Regression

Unknown.

Your Environment

  • Version used: 0.43.0
  • Operating System and version (uname -a): Debian stable
@roman-khimov roman-khimov added bug Something isn't working neofs-storage Storage node application issues U2 Seriously planned S4 Routine I4 No visible changes labels Oct 18, 2024
@roman-khimov roman-khimov added this to the v0.44.0 milestone Oct 18, 2024
@carpawell
Copy link
Member

carpawell commented Oct 24, 2024

2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2/DsX2krdGGPE1pgR1PPrUXF1tEzPdJQHMZqQA8CfqjpG6
2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2/3yGt6PwGjqFq94DsgJfuKC88Bo5ZRQeVLQswXz2MASJN

two objects cannot be dropped from metis2 but from a single container. WC has been cleaned

@carpawell
Copy link
Member

Some more problematic objects.

"2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2/2hUsdvs7tdxnXK9hLMd1BYgcVrSezS56XtmoNW1KN75k",
"2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2/RDvnKLMNUVHZUZHka4qH5a7AkZGdPH2cFzN6tyy8zXU",
"2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2/Wg5R8vaNcYuGk9GUtkxgFGymoG8G8iJnwXTamqtZZos",
"2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2/yiiBB9Dm6gDwav3aNu7fXZXL3rkr42hZs5XZ2G19Ujh",

@carpawell
Copy link
Member

carpawell commented Oct 30, 2024

So I have cleaned manually bucket with object headers for this (2W2c7qHqicZJJdRG134k7eusZqidkk3KBDNgeLL45nn2) container and also cleared WC that could not update metabase with storage ID for objects of containers that do not exists. Now node should not do strange useless things with disks and logs are cleared (all the nodes only clear their metabases since they can now continue their GC cycle).

@roman-khimov
Copy link
Member Author

In fact, it gets contents from the metabase, so #2978 is also relevant here as the one leading to #2988.

@carpawell
Copy link
Member

@roman-khimov, also pretty sure it is #2978 (comment).

carpawell added a commit that referenced this issue Nov 25, 2024
If ants pool is busy and cannot take task, early `return` without `wg.Wait()`
leads to `iterateNodesForObject`'s `return` and all the buffers for binary
replication from now may be reused while are still in use by the other routines
inside the pool. Wait for WG before any `return` is called. Closes #2978, #2988,
#2975, #2971.

Signed-off-by: Pavel Karpy <[email protected]>
carpawell added a commit that referenced this issue Nov 25, 2024
If ants pool is busy and cannot take task, early `return` without `wg.Wait()`
leads to `iterateNodesForObject`'s `return` and all the buffers for binary
replication from now may be reused while are still in use by the other routines
inside the pool. Wait for WG before any `return` is called. Closes #2978,
closes #2988, closes #2975, closes #2971.

Signed-off-by: Pavel Karpy <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working I4 No visible changes neofs-storage Storage node application issues S4 Routine U2 Seriously planned
Projects
None yet
Development

No branches or pull requests

2 participants