iPadOS 13 interrupted sound

[KatieWoe]

Found during phetsims/qa#461. Found in a number of published sims. Some in tambo and some not (Plinko Probability). Doesn't happen in John Travoltage for some reason. To get it to occur, leave the page using a double tap of the home button for awhile by going to settings or some other app. Then, double tap the home button again to go back to the sim. Sound won't work.

[KatieWoe]

Saw in phetsims/qa#480 on the app. Occurs if you put the iPad to sleep and wake it up again.

[zepumph]

@emily-phet should this block GFL publication?

[emily-phet]

@kathy-phet For clarification - once you return to the sim, there's no way to get it back other than restarting the sim.

@zepumph No, I don't think it should block publication.

@jbphet Any thoughts on this in general?

[KatieWoe]

@emily-phet if you meant me, then that is correct. I don't think it gets rid of VO in any way, but tambo sounds and vibe sounds.

[kathy-phet]

Agreed that it shouldn’t block publication but it is worth some investigation to see if it’s fixable. K

…

Sent from my iPhone On Feb 20, 2020, at 2:09 PM, emily-phet <[email protected]> wrote:  @kathy-phet<https://github.com/kathy-phet> For clarification - once you return to the sim, there's no way to get it back other than restarting the sim. @zepumph<https://github.com/zepumph> No, I don't think it should block publication. @jbphet<https://github.com/jbphet> Any thoughts on this in general? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#90?email_source=notifications&email_token=ABG4KZBBRSRTSI3Y3VVL6CDRD3WRXA5CNFSM4J3S5SD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMQE4EY#issuecomment-589319699>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABG4KZCOQMLPXIZKTEHQLCDRD3WRXANCNFSM4J3S5SDQ>.

[emily-phet]

@kathy-phet - sorry Kathy, I was intending to ask Katie that question. Thanks for weighing in anyway!

[zepumph]

Alright great. I think of this as a sound related, sim-unspecific issue, so I will assign to @jbphet. Let me know if you want assistance.

[jbphet]

@emily-phet said:

@jbphet Any thoughts on this in general?

Yes, I've seen some out-of-spec behavior of the audio context in iOS before. The main thing I've seen is that sometimes the state of the audio context gets set to interrupted, which doesn't even exist in the spec. My guess is that iOS tries to save battery by turning off the power to the amplifier that drives the speakers if the page that was creating sound is no longer active, and that's when the 'interrupted' state is set. This seems closely related to phetsims/friction#173, and I added code in tambo to try to recover the audio context if it ever became interrupted. Maybe Apple has changed things such that you can't re-enabled the audio context unless the page is being shown again. I'll investigate.

[jbphet]

The issues below are all related to this one, and have to do with the audio context being set to states other than "running":

• Sound stops when laptop is closed #59
• Sound sometimes gone or delayed in full-screen on Safari fractions-common#82
• Sound disappears on iOS 12 friction#173
• Sound disappears sometimes in Full-Screen Mode resistance-in-a-wire#190

[jbphet]

As suspected, this is due to state changes for the audio context. The behavior of the code when I started working on this was to detect changes to the 'interrupted' state and try to transition right away back to the 'running' state. That worked in many cases, such as for the full screen issue, but it wasn't working in this case - the attempt to resume either didn't work at all, or it appeared to work based on the state value but no sound was produced (this was all based on debug output that I added temporarily). I then tried resuming the audio context when the window became active again, but that didn't work either. I then tried resuming on the first user interaction after the sim window becomes active again, and that appears to work, at least so far.

I did most of my testing and experimentation for this on GFL, and spent way longer on it than I had hoped. @KatieWoe - can you test sims on master and see if this seems to be fixed? Also, please test the issues listed above and verify that none of them have reappeared as a result of these changes.

[jbphet]

By the way, when working on this, I found that I could duplicate the problem somewhat sporadically in a regular window but more consistently in a "Private" window.

Also, my testing was done on the PhET iPad Air 2 named "Pauling" running iOS 13.4.

[Nancy-Salpepi]

On the iPad Air 2 iOS 15.0:

If sleep mode is enabled while voicing is on, when I return to the sim the voicing still works, but no other sounds are heard. However, if I didn't turn on voicing and returned to the sim after sleep mode, the sounds still worked. This was seen on John Travoltage (published and in Master) and GFL:B in Master.

nosoundsleep.mov

Update 11/15: moved this comment to its own issue #157 so @jbphet you are now free to close.

[jbphet]

@Nancy-Salpepi said:

[I've} moved this comment to its own issue #157 so @jbphet you are now free to close.

Sounds good, I'll follow up on that one and close this one.