Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestTailFollowRotateLogs is still failing #3225

Closed
apostasie opened this issue Jul 16, 2024 · 4 comments · Fixed by #3259
Closed

TestTailFollowRotateLogs is still failing #3225

apostasie opened this issue Jul 16, 2024 · 4 comments · Fixed by #3259
Labels
kind/unconfirmed-bug-claim Unconfirmed bug claim

Comments

@apostasie
Copy link
Contributor

Description

Despite many tries to tweak the timeout, log length or log parameters, this test still fails regularly.

Even locally, this:

while true; do go test ./cmd/nerdctl -count 1 -run TestTailFollowRotateLogs; done

Will fail at least once in ten times.

I am now thinking there might actually be an underlying bug.

Could it be the same issue as #3199 ?

Log specialists, any idea? (@fahedouch maybe?)

Steps to reproduce the issue

na

Describe the results you received and expected

na

What version of nerdctl are you using?

main

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response
@apostasie apostasie added the kind/unconfirmed-bug-claim Unconfirmed bug claim label Jul 16, 2024
@apostasie
Copy link
Contributor Author

Note: after testing against PR #3200 the issue is still there.

@apostasie apostasie mentioned this issue Jul 16, 2024
Closed
@apostasie apostasie mentioned this issue Jul 25, 2024
Merged
15 tasks
@xyz-li
Copy link
Contributor

xyz-li commented Jul 29, 2024 •
edited
Loading

nerdctl/cmd/nerdctl/container_logs_test.go

Lines 224 to 229 in bec95c4

const sampleJSONLog = `{"log":"A\n","stream":"stdout","time":"2024-04-11T12:01:09.800288974Z"}`
const linesPerFile = 2
defer base.Cmd("rm", "-f", containerName).Run()
base.Cmd("run", "-d", "--log-driver", "json-file",
"--log-opt", fmt.Sprintf("max-size=%d", len(sampleJSONLog)*linesPerFile),

I think max-size is too small and log output is too quicklly. There is one situation, startTail return with true, and nerdctl rotate the logfile, then openFileShareDelete will return an error. At last this test will fail.

How about change linesPerFile to 1000.

root@lh:/root/go/src/github.com/containerd/nerdctl# while true; do go test ./cmd/nerdctl -count 1 -run TestTailFollowRotateLogs; done
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.168s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.688s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.447s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	3.688s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.488s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.658s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.904s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	2.362s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.581s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.527s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.124s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.902s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.020s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.410s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.846s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.540s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	2.973s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.168s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.679s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.667s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.335s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.582s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.449s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.467s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	2.151s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	2.055s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.971s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.490s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.785s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.126s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.863s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	2.637s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.546s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.115s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.578s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.573s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.567s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	3.680s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	4.220s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.636s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.087s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	5.068s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	3.755s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.726s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.672s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.745s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.535s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.689s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.542s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	2.160s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.372s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.938s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.543s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.634s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.220s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	2.223s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.270s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.510s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.615s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	3.755s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.567s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.673s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.727s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.399s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.160s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.675s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.667s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	2.495s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	4.484s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.106s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.194s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	0.612s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	1.059s

@apostasie
Copy link
Contributor Author

How about change linesPerFile to 1000.

@xyz-li anything to make this test stop failing of course :-)

Do you think there is an underlying issue we can look into though?

@xyz-li
Copy link
Contributor

xyz-li commented Jul 30, 2024

I think max-size is too small and log output is too quicklly. There is one situation, startTail(or os.OpenFile) return with true, and nerdctl rotate the logfile, then openFileShareDelete will return an error. At last this test will fail.

After change linesPerFile to 2000.

# while true; do go test ./cmd/nerdctl -count 1 -run TestTailFollowRotateLogs; done
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.327s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.355s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.337s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.349s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.487s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.404s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.360s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.275s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	13.349s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.660s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.700s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.613s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.335s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.574s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.307s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.373s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.491s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.445s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.464s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.497s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.739s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.517s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.321s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.399s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.396s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.423s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.558s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.411s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.406s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.393s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.385s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.415s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.439s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.446s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.690s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.456s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.411s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.483s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.416s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.422s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.421s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.291s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.334s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.357s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.461s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.412s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.440s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.505s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.519s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.276s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.427s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.369s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.484s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.440s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.437s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.397s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.384s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.362s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.649s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.787s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.520s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.410s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.432s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.381s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.158s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.953s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.980s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.473s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.412s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.251s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.419s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.795s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.522s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.618s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.277s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.356s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.178s
ok  	github.com/containerd/nerdctl/v2/cmd/nerdctl	12.406s

@xyz-li xyz-li mentioned this issue Jul 30, 2024
Merged
@fahedouch fahedouch mentioned this issue Jul 31, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/unconfirmed-bug-claim Unconfirmed bug claim
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

test: increase logs max-size xyz-li/nerdctl
2 participants
@xyz-li @apostasie