Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downgrade pty to v1.1.19 #2791

Merged
merged 1 commit into from
May 23, 2024
Merged

Downgrade pty to v1.1.19 #2791

merged 1 commit into from
May 23, 2024

Conversation

moskyb
Copy link
Contributor

@moskyb moskyb commented May 23, 2024

Description

1.1.20 and 1.1.21 appear to have a bug where occasionally, running processes under the pty will return a fork/exec: operation not permitted error

fixes #2790

Context

#2790
Coda escalations 1 and 2

Testing

  • Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
  • Code is formatted (with go fmt ./...)

1.1.20 and 1.1.21 appear to have a bug where occasionally, running processes under the pty will return a fork/exec: operation not permitted error
@moskyb moskyb requested review from mhaylock and tessereth May 23, 2024 03:59
@moskyb moskyb enabled auto-merge May 23, 2024 03:59
@moskyb moskyb merged commit 0aa38aa into main May 23, 2024
1 check passed
@moskyb moskyb deleted the downgrade-pty branch May 23, 2024 04:18
@flokli
Copy link

flokli commented Jul 31, 2024

I get "open /dev/pts/0: permission denied" with buildkite-agent-3.59.0, which is using creack/[email protected] creack/[email protected], so I'm not sure this downgrade actually fixes the issue so I'll upgrade buildkite-agent to check if the pty downgrade fixes the issue for me too. See #2912 for details.

@danktec
Copy link

danktec commented Aug 27, 2024

I just started getting Error running job: error starting pty: fork/exec /usr/bin/buildkite-agent: permission denied on the latest agent version.
I tried downgrading back to v3.36.0 and i'm now getting fork/exec /usr/bin/buildkite-agent: permission denied
so i'm at a loss... i have no idea how this was introduced

@DrJosh9000
Copy link
Contributor

Just checking @danktec, /usr/bin/buildkite-agent is where the agent is installed, and it is executable by the user you're running as? e.g.

$ docker run --rm -it --entrypoint sh buildkite/agent:latest
/ # which buildkite-agent
/usr/local/bin/buildkite-agent
/ # stat $(which buildkite-agent)
  File: /usr/local/bin/buildkite-agent
  Size: 44301452  	Blocks: 86528      IO Block: 4096   regular file
Device: f5h/245d	Inode: 571703      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-08-08 02:50:50.000000000 +0000
Modify: 2024-08-08 02:50:50.000000000 +0000
Change: 2024-08-08 05:43:06.593159640 +0000

@danktec
Copy link

danktec commented Aug 27, 2024

@DrJosh9000 I did ensure all permissions were correct in all the places.
I have since rebuild my image and made some changes which have made this issue go away.

The only thing i can point to potentially is having a bad git-clone hook in the cfg file.

PS: I was running the agent via the apt installer on Ubuntu. (Agent runs on the host, then it calls docker on the same host, (trying to avoid other DinD issues))

@DrJosh9000
Copy link
Contributor

Thanks @danktec , just have to make sure we aren't missing something.

If you (or anyone) suspects pty problems, and are particularly motivated to help debug: part of the process of blaming pty for the bug was running the agent under strace -f, poring over the output, and noticing EPERM from an ioctl call involved in setting up a pty.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Buildkite Agent PTY randomly failed with fork/exec... operation not permitted
5 participants