Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachprod: move /tmp/cores files somewhere under /mnt/data1 on gce #35621

Closed
danhhz opened this issue Mar 11, 2019 · 2 comments
Closed

roachprod: move /tmp/cores files somewhere under /mnt/data1 on gce #35621

danhhz opened this issue Mar 11, 2019 · 2 comments
Assignees

Comments

@danhhz
Copy link
Contributor

danhhz commented Mar 11, 2019

Found while looking into a roachtest failure, this is gce specific, I haven't investigated aws. These files are huge and taking most of the root partition, which then causes cascading badness. I think /mnt/data1 is basically the only place they could go:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.1G     0  7.1G   0% /dev
tmpfs           1.5G  9.6M  1.4G   1% /run
/dev/sda1       9.7G  9.7G     0 100% /
tmpfs           7.1G     0  7.1G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.1G     0  7.1G   0% /sys/fs/cgroup
/dev/sdb        369G  425M  350G   1% /mnt/data1
tmpfs           1.5G     0  1.5G   0% /run/user/1005
@danhhz
Copy link
Contributor Author

danhhz commented Mar 11, 2019

One of these files was 8.5G on a 9.7G root partition.

ls -l -h /tmp/cores/core.cockroach.3494.dan-1552335208-cdc-ledger-rangefeed-true-0001.1552335817
-rw------- 1 dan dan 8.5G Mar 11 20:28 /tmp/cores/core.cockroach.3494.dan-1552335208-cdc-ledger-rangefeed-true-0001.1552335817

@yuzefovich
Copy link
Member

I ran into this problem on aws as well.

petermattis added a commit to petermattis/cockroach that referenced this issue Mar 29, 2019
/tmp is only 10GB in size, while /mnt/data1 is 370GB.

Fixes cockroachdb#35621

Release note: None
craig bot pushed a commit that referenced this issue Mar 29, 2019
36338: roachprod: change the location cores are written to r=danhhz a=petermattis

/tmp is only 10GB in size, while /mnt/data1 is 370GB.

Fixes #35621

Release note: None

Co-authored-by: Peter Mattis <[email protected]>
@craig craig bot closed this as completed in #36338 Mar 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants