Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure ffmpeg temp files are always fully written #78

Merged
merged 2 commits into from
Jan 21, 2020
Merged

Ensure ffmpeg temp files are always fully written #78

merged 2 commits into from
Jan 21, 2020

Conversation

qtiki
Copy link
Contributor

@qtiki qtiki commented Jan 5, 2020

This PR fixes audiosprite generation failures on congested hard drives where the individual ffmpeg write streams may not have finished before the sprite is generated.

Some background:

I've already done a fix to this package regarding a similar issue almost two years ago: #73

Since then we've upgraded our build system and are running much more concurrent builds than before on a single machine. Recently we've seen a lot of these builds fail randomly due to audiosprite generation, where we validate that the individual samples in the generated sprites actually have a length greater than zero.

It took me quite a while to pinpoint the problem but it turns out to be caused by hard drive congestion which slows the disk I/O enough to bring up this particular race condition. I managed to isolate the problem by simulating a really slow hard disk according to this great blog post: https://planet.jboss.org/post/debugging_tip_how_to_simulate_a_slow_hardisk

So when I created a network drive and slowed it down to 1 Mb/s read and write speed I managed to get the audiosprite package's own tests to fail around 20% of the time - tests which were working 100% of the time with a regular speed hard drive. The failed tests sometimes had sprites of length 0 and sometimes even more scarier lengths, where the samples had some data clipped from the end. These are scarier since we do not catch in our build system and thus they could result in corrupted audio in our production builds.

Needless to say that with this fix the tests work 100% even with the slow hard drive.

The implementation:

The write stream didn't have any listener so even though my earlier fix ensured that the ffmpeg's stdout stream had closed there was no guarantee that the write stream had finished flushing everything to disk. So I added a 'close' event listener for that stream and then used Lodash after to postpone the callback until both the process has exited and the write stream has finished. The code would have been much cleaner if we only listened to the write stream but we do need the process listener as well to catch error codes and I didn't really want to break existing functionality there.

I've tested this on Windows with Node v10.17.0 and in Linux with Node v13.5.0. I don't see any reason why it wouldn't work with older versions of Node or other platforms as well.

…ite is generated

Fixes audiosprite generation failures on congested hard drives where the individual ffmpeg write streams may not have finished before the sprite is generated.
@qtiki
Copy link
Contributor Author

qtiki commented Jan 5, 2020

Bonus commit:

The Travis CI config contained an obsolete ffmpeg binary. I updated it to fix tests. Not a fan of broken builds. 😉

@vkarponen
Copy link

ping @tonistiigi . Would you have the time to take a brief glimpse into this?

@tonistiigi tonistiigi merged commit a198a90 into tonistiigi:master Jan 21, 2020
@vkarponen
Copy link

👍 @tonistiigi 👍
Maybe also a new release on npm?

@tonistiigi
Copy link
Owner

@SargoDarya ^

@SargoDarya
Copy link
Collaborator

Sorry folks, was a bit busy. I'll prepare a release for end of the week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants