Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Playback does not start when audio is too much early than video #291

Closed
goffioul opened this issue Apr 2, 2023 · 1 comment
Closed

Playback does not start when audio is too much early than video #291

goffioul opened this issue Apr 2, 2023 · 1 comment
Assignees
@tonihei
Labels
bug

Comments

@goffioul
Copy link
Contributor

goffioul commented Apr 2, 2023

[I'm posting this as a question instead of a bug, as you might not consider it a bug per-se]

I'm using ExoPlayer 2.18.1 and I've encountered a problem with a test stream (I will post link via email), whereby ExoPlayer never starts playback on multiple platforms. ExoPlayer actually never enters ready state, but if the playback position is moved manually (e.g. using progress bar in demo app), then video plays normally.

This particular stream, a capture from a real-life scenario, has a rather bumpy start, with initial AC3 audio data with a timestamp that is significantly early compared to first video frames. This difference is larger than the threshold in https://github.com/androidx/media/blob/release/libraries/exoplayer/src/main/java/androidx/media3/exoplayer/audio/MediaCodecAudioRenderer.java#L664, so the audio clock is updated (backwards). But then the first video frames produced by the codec are not rendered, as they have a later timestamp, and MediaCodecVideoRenderer never enters ready state.

I've seen failure to start playback on multiple platforms:

  • rockchip with Android 11
  • amlogic with Android 7
  • android-x86 with Android 11 (ffmpeg-omx) and Android 13 (ffmpeg-codec2)
  • using the platform AC3 codec (if present) or using the ffmpeg audio extension of ExoPlayer (with ac3 enabled)

If I increase the threshold mentioned above to 750000, then playback starts normally. Whether this is a proper fix, I don't know. There's a comment there that states it's a hack, and refers to an internal bug report.

The question is really whether ExoPlayer could handle this more gracefully than not starting playback?
@tonihei
Copy link
Collaborator

tonihei commented Apr 5, 2023

Thanks for reporting! This is in fact a known bug in the logic of MediaCodecVideoRenderer of when to render the first frame. The line you linked in MediaCodecAudioRenderer accidentally avoids the problem if the offset is small, but this isn't the solution. The logic for the real fix is quite complicated to get right as it needs to work for many different scenarios and that's why we haven't yet fixed it. [internal bug ref: b/160461756#comment11]

@tonihei tonihei self-assigned this Apr 5, 2023
@christosts christosts mentioned this issue Apr 25, 2023
Open
1 task
tianyif pushed a commit that referenced this issue Aug 7, 2023
@tonihei @tianyif
Only use first frame stream start position check at stream transitions
6965677 
We currently only force the first frame if the frame timestamp is
greater than the stream *offset*.

This is wrong for two reasons:
 1. The timestamp and the offset are not comparable and it should be
    the stream start position.
 2. The check should only be applied at stream transitions where we
    need to make sure that a new first frame isn't rendered until we
    passed the transition point.

We have to fix both issues together, because fixing just issue (1)
causes seeks to before the start position to no longer render the
frame (and playback will be stuck). A new test covers this case.

We also amend the stream transition test case to actually test what it
promises to test and add a test for prerolling samples at the
beginning, to ensure the first frame is still renderered.

Issue: #291
PiperOrigin-RevId: 552858967
tianyif pushed a commit that referenced this issue Aug 7, 2023
@tonihei @tianyif
Remove old workaround for timestamp jumps
57f3278 
This workaround was added for TS streams that do not adjust their
timestamps to start from zero. Over time, the default audio sink
logic has become more robust towards unexpected timestamps and
we no longer need this workaround to jump forward in time.

The workaround also actively caused issues by adjusting the audio
timestamps backwards if the stream starts with large negative
values. See Issue: #291. This caused playback to get stuck due
to another bug in the first-frame rendering logic in the video
renderer that is now fixed.

PiperOrigin-RevId: 553493618
tianyif pushed a commit to google/ExoPlayer that referenced this issue Aug 7, 2023
@tonihei @tianyif
Only use first frame stream start position check at stream transitions
bf299d3 
We currently only force the first frame if the frame timestamp is
greater than the stream *offset*.

This is wrong for two reasons:
 1. The timestamp and the offset are not comparable and it should be
    the stream start position.
 2. The check should only be applied at stream transitions where we
    need to make sure that a new first frame isn't rendered until we
    passed the transition point.

We have to fix both issues together, because fixing just issue (1)
causes seeks to before the start position to no longer render the
frame (and playback will be stuck). A new test covers this case.

We also amend the stream transition test case to actually test what it
promises to test and add a test for prerolling samples at the
beginning, to ensure the first frame is still renderered.

Issue: androidx/media#291
PiperOrigin-RevId: 552858967
tianyif pushed a commit to google/ExoPlayer that referenced this issue Aug 7, 2023
@tonihei @tianyif
Remove old workaround for timestamp jumps
2f5ded8 
This workaround was added for TS streams that do not adjust their
timestamps to start from zero. Over time, the default audio sink
logic has become more robust towards unexpected timestamps and
we no longer need this workaround to jump forward in time.

The workaround also actively caused issues by adjusting the audio
timestamps backwards if the stream starts with large negative
values. See Issue: androidx/media#291. This caused playback to get stuck due
to another bug in the first-frame rendering logic in the video
renderer that is now fixed.

PiperOrigin-RevId: 553493618
@tonihei tonihei closed this as completed Aug 9, 2023
@androidx androidx locked and limited conversation to collaborators Oct 9, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tonihei tonihei

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@goffioul @tonihei