Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

travis-ci: build fails with write error #1145

Closed
grondo opened this issue Aug 8, 2017 · 9 comments
Closed

travis-ci: build fails with write error #1145

grondo opened this issue Aug 8, 2017 · 9 comments

Comments

@grondo
Copy link
Contributor

grondo commented Aug 8, 2017

Occasionally travis builders are getting write errors during the final phase of make distcheck. Maybe a disk space issue? Hard to say because it doesn't happen every time. However in #1143, I had to restart 2 of 7 builds.

This might be more likely for some reason since make -j2 was added in pr #1142
@grondo grondo mentioned this issue Aug 11, 2017
Merged
@garlick garlick mentioned this issue Aug 17, 2017
Merged
@garlick
Copy link
Member

garlick commented Aug 25, 2017

I always seem to be stuck looking through the raw travis log lately as the pretty one is usually truncated without clues. Could someone post some sample output here demonstrating the error so that I know what to search for in the raw log?

@chu11
Copy link
Member

chu11 commented Aug 25, 2017

What I've been doing is making sure all unit tests pass on the failed build, to make sure the build was "successful" in the way we care about. Then if that's the case, search for write error in the raw logs. If I find it, I assume that's the reason for the build fail.

@grondo grondo mentioned this issue Aug 29, 2017
Merged
@garlick
Copy link
Member

garlick commented Aug 31, 2017

My be a clue here

travis-ci/travis-ci#4704

@grondo
Copy link
Contributor Author

grondo commented Sep 1, 2017

Hm, piping output through their "careful-cat" program might not be a bad idea. In a latest build I saw /bin/bash: echo: write error, so it may be that programs writing to travis output are getting EAGAIN. I don't think that the other suggested workaround (set filter-secrets: false) is secure for us, but I'll see if there is a potential for any of our secret env vars to get issued to stdout/err.

@garlick garlick mentioned this issue Sep 29, 2017
Merged
grondo added a commit to grondo/flux-core that referenced this issue Oct 20, 2017
@grondo
travis-ci: remove -j 2from distcheck builds
864d26a 
The "write errors" (flux-framework#1145) in travis-ci seemed to have increased
with the introduction of '-j2' to make distcheck. Perhaps the amount
of data on stdout/err or written to disk is overwhelming the
travis builders.

Remove -j2 for now in hopes that it will make builds more reliable.
@grondo grondo mentioned this issue Oct 20, 2017
Merged
grondo added a commit to grondo/flux-core that referenced this issue Oct 20, 2017
@grondo
travis-ci: remove -j2 from distcheck builds
c51255e 
The "write errors" (flux-framework#1145) in travis-ci seemed to have increased
with the introduction of '-j2' to make distcheck. Perhaps the amount
of data on stdout/err or written to disk is overwhelming the
travis builders.

Remove -j2 for now in hopes that it will make builds more reliable.
grondo added a commit to grondo/flux-core that referenced this issue Oct 20, 2017
@grondo
travis-ci: remove -j2 from distcheck builds
28ec3f4 
The "write errors" (flux-framework#1145) in travis-ci seemed to have increased
with the introduction of '-j2' to make distcheck. Perhaps the amount
of data on stdout/err or written to disk is overwhelming the
travis builders.

Remove -j2 for now in hopes that it will make builds more reliable.
@garlick
Copy link
Member

garlick commented Oct 27, 2017

I haven't seen a single occurrence of this since #1245 was merged. If anybody else does, please log it here.

@garlick
Copy link
Member

garlick commented Nov 13, 2017

Just finally saw one of these

configure:17828: checking for sys/param.h
configure:17828: result: yes
configure:17828: checking stdarg.h usabilitycat: write error: Resource temporarily unavailable

build.log

@garlick
Copy link
Member

garlick commented Nov 13, 2017

Just saw four more in #1286

@garlick garlick mentioned this issue Nov 13, 2017
Merged
@felker felker mentioned this issue Nov 14, 2017
Closed
@SteVwonder
Copy link
Member

Ran into this building locally with docker:

mawk: write failure (Resource temporarily unavailable)mawk: write failure (Resource temporarily unavailable)
tap-driver.sh: fatal: I/O or internal error
Makefile:1232: recipe for target 'test_wrap.log' failed
make[6]: *** [test_wrap.log] Error 1
make[6]: *** Waiting for unfinished jobs....
tap-driver.sh: fatal: I/O or internal error
Makefile:1232: recipe for target 'test_clique.log' failed

@grondo
Copy link
Contributor Author

grondo commented Feb 5, 2019

This seems to have been fixed with some combination of upstream travis fixes, local fixes for I/O handling in make, etc. Closing.

@grondo grondo closed this as completed Feb 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@garlick @chu11 @grondo @SteVwonder