Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

General crash and hang fixes #9698

Merged
merged 8 commits into from
May 18, 2017
Merged

General crash and hang fixes #9698

merged 8 commits into from
May 18, 2017

Conversation

hrydgard
Copy link
Owner

The Play console gained improved crash and ANR (application not responding) reporting, I'm going through them and finding some issues.

The top crashes is ManagedTexture related, and that means it's likely related to the GameInfoCache, so I've made some reliability improvements to that.

@hrydgard hrydgard changed the title Managedtexture reliability General crash and hang fixes May 18, 2017
@hrydgard
Copy link
Owner Author

hrydgard commented May 18, 2017

Ooh, found a fun deadlock between eventMutex_ in view.cpp and inputLock_ in screen.cpp, this is reported ~17 times/week, which isn't much though:

DispatchEvent locks eventMutex and ScreenManager::push locks inputLock_:

  native: pc 00000000004496af  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (ScreenManager::push(Screen*, int)+26)
  native: pc 000000000044c777  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::PopupMultiChoice::HandleClick(UI::EventParams&)+814)
  native: pc 0000000000452c05  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::DispatchEvents()+260)
  native: pc 000000000045e805  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::UpdateViewHierarchy(UI::ViewGroup*)+1084)
  native: pc 00000000003e445f  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (GameSettingsScreen::update()+10)
  native: pc 00000000003adbbf  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (NativeUpdate()+366)
  native: pc 00000000003a7e3d  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (Java_org_ppsspp_ppsspp_NativeRenderer_displayRender+40)
  native: pc 0000000000004ea7  /data/dalvik-cache/arm/data@[email protected]@[email protected] (Java_org_ppsspp_ppsspp_NativeRenderer_displayRender__+82)
  at org.ppsspp.ppsspp.NativeRenderer.displayRender (Native method)
  at org.ppsspp.ppsspp.NativeRenderer.onDrawFrame (NativeRenderer.java:57)
  at android.opengl.GLSurfaceView$GLThread.guardedRun (GLSurfaceView.java:1522)
  at android.opengl.GLSurfaceView$GLThread.run (GLSurfaceView.java:1239)

vs

ScreenManager:touch lock inputLock_ and EventTrigger locks eventMutex_:

  native: pc 0000000000453217  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::Event::Trigger(UI::EventParams&)+54)
  native: pc 0000000000453ac7  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::Clickable::Click()+46)
  native: pc 0000000000453d19  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::Clickable::Touch(TouchInput const&)+376)
  native: pc 0000000000459cc9  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::ViewGroup::Touch(TouchInput const&)+56)
  native: pc 0000000000459cc9  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::ViewGroup::Touch(TouchInput const&)+56)
  native: pc 000000000045b933  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::ScrollView::Touch(TouchInput const&)+338)
  native: pc 0000000000459cc9  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::ViewGroup::Touch(TouchInput const&)+56)
  native: pc 0000000000459cc9  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::ViewGroup::Touch(TouchInput const&)+56)
  native: pc 000000000045e395  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UI::TouchEvent(TouchInput const&, UI::ViewGroup*)+24)
  native: pc 000000000044b147  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (UIScreen::touch(TouchInput const&)+18)
  native: pc 00000000004491eb  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (ScreenManager::touch(TouchInput const&)+74)
  native: pc 00000000003add19  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (NativeTouch(TouchInput const&)+20)
  native: pc 00000000003a8643  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (Java_org_ppsspp_ppsspp_NativeApp_touch+62)
  native: pc 0000000000008c8b  /data/dalvik-cache/arm/data@[email protected]@[email protected] (Java_org_ppsspp_ppsspp_NativeApp_touch__FFII+110)
  at org.ppsspp.ppsspp.NativeApp.touch (Native method)
  at org.ppsspp.ppsspp.NativeGLView.onTouchEvent (NativeGLView.java:111)

@hrydgard
Copy link
Owner Author

hrydgard commented May 18, 2017

I'm starting to think we should serialize touch events and similar and process them once per frame in the UI layer (we'll still immediately forward them to the emulation during gameplay)...

Actually this one is just a missing lock in ScreenManager::update() - that fixes the lock ordering issue.

…Mutex_ by locking here so the reverse lock order can't happen.
@hrydgard
Copy link
Owner Author

Here's one I don't know what to make of:

"GLThread 27158" prio=5 tid=11 Native
  | group="main" sCount=1 dsCount=0 obj=0x12c03080 self=0xb8696438
  | sysTid=15996 nice=0 cgrp=default sched=0/0 handle=0xa33b5930
  | state=R schedstat=( 0 0 0 ) utm=6827 stm=2137 core=1 HZ=100
  | stack=0xa32b3000-0xa32b5000 stackSize=1038KB
  | held mutexes=
  native: pc 00000000000435c4  /system/lib/libc.so (gettimeofday+12)
  native: pc 0000000000037da5  /system/lib/libc.so (time+8)
  native: pc 00000000003a4913  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer16GetTimeFormattedEPc+22)
  native: pc 00000000003a4911  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer16GetTimeFormattedEPc+20)
  native: pc 00000000003a490f  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer16GetTimeFormattedEPc+18)
  native: pc 00000000003a490d  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer16GetTimeFormattedEPc+16)
  native: pc 00000000003a490b  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer16GetTimeFormattedEPc+14)
  native: pc 00000000003a4909  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer16GetTimeFormattedEPc+12)
  native: pc 00000000003a4907  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer16GetTimeFormattedEPc+10)
  native: pc 00000000003a4905  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer16GetTimeFormattedEPc+8)
  native: pc 00000000003a4903  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer16GetTimeFormattedEPc+6)
  native: pc 00000000003a4901  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer16GetTimeFormattedEPc+4)
  native: pc 00000000003a48ff  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer16GetTimeFormattedEPc+2)
  native: pc 00000000003a48fd  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer16GetTimeFormattedEPc)
  native: pc 00000000003a48fb  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+90)
  native: pc 00000000003a48f9  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+88)
  native: pc 00000000003a48f7  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+86)
  native: pc 00000000003a48f5  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+84)
  native: pc 00000000003a48f1  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+80)
  native: pc 00000000003a48ef  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+78)
  native: pc 00000000003a48ed  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+76)
  native: pc 00000000003a48eb  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+74)
  native: pc 00000000003a48e9  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+72)
  native: pc 00000000003a48e7  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+70)
  native: pc 00000000003a48e5  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+68)
  native: pc 00000000003a48e3  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+66)
  native: pc 00000000003a48e1  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+64)
  native: pc 00000000003a48df  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+62)
  native: pc 00000000003a48dd  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+60)
  native: pc 00000000003a48db  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+58)
  native: pc 00000000003a48d9  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+56)
  native: pc 00000000003a48d7  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+54)
  native: pc 00000000003a48d5  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+52)
  native: pc 00000000003a48d3  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+50)
  native: pc 00000000003a48d1  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+48)
  native: pc 00000000003a48cf  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+46)
  native: pc 00000000003a48cd  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+44)
  native: pc 00000000003a48c9  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+40)
  native: pc 00000000003a48c7  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+38)
  native: pc 00000000003a48c3  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+34)
  native: pc 00000000003a48c1  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+32)
  native: pc 00000000003a48bf  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+30)
  native: pc 00000000003a48bb  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+26)
  native: pc 00000000003a48b9  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+24)
  native: pc 00000000003a48b5  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+20)
  native: pc 00000000003a48b3  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+18)
  native: pc 00000000003a48b1  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+16)
  native: pc 00000000003a48af  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+14)
  native: pc 00000000003a48ad  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+12)
  native: pc 00000000003a48ab  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+10)
  native: pc 00000000003a48a9  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+8)
  native: pc 00000000003a48a7  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+6)
  native: pc 00000000003a48a5  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+4)
  native: pc 00000000003a48a3  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev+2)
  native: pc 00000000003a48a1  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer24GetLocalTimeSinceJan1970Ev)
  native: pc 00000000003a489f  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer19GetTimeSinceJan1970Ev+54)
  native: pc 00000000003a489d  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer19GetTimeSinceJan1970Ev+52)
  native: pc 00000000003a489b  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer19GetTimeSinceJan1970Ev+50)
  native: pc 00000000003a4899  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer19GetTimeSinceJan1970Ev+48)
  native: pc 00000000003a4895  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer19GetTimeSinceJan1970Ev+44)
  native: pc 00000000003a4893  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer19GetTimeSinceJan1970Ev+42)
  native: pc 00000000003a4891  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer19GetTimeSinceJan1970Ev+40)
  native: pc 00000000003a488f  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer19GetTimeSinceJan1970Ev+38)
  native: pc 00000000003a488d  /data/app/org.ppsspp.ppsspp-1/lib/arm/libppsspp_jni.so (_ZN6Common5Timer19GetTimeSinceJan1970Ev+36)
  at org.ppsspp.ppsspp.NativeRenderer.displayRender (Native method)
  at org.ppsspp.ppsspp.NativeRenderer.onDrawFrame (NativeRenderer.java:57)
  at android.opengl.GLSurfaceView$GLThread.guardedRun (GLSurfaceView.java:1535)
  at android.opengl.GLSurfaceView$GLThread.run (GLSurfaceView.java:1240)

That's the only thread running (the GL thread). The main thread is paused but has Activity.pause() (java) in the callstack.

@hrydgard
Copy link
Owner Author

hrydgard commented May 18, 2017

This one is surprisingly common. I think people are somehow enabling the software renderer by accident... Still, don't see how it can crash.

  native: pc 0000000000347952  /data/app-lib/org.ppsspp.ppsspp-1/libppsspp_jni.so (VertexDecoder::Step_PosS16Through() const+25)
  native: pc 0000000000348673  /data/app-lib/org.ppsspp.ppsspp-1/libppsspp_jni.so (VertexDecoder::DecodeVerts(unsigned char*, void const*, int, int) const+90)
  native: pc 000000000036bd51  /data/app-lib/org.ppsspp.ppsspp-1/libppsspp_jni.so (TransformUnit::SubmitPrimitive(void*, void*, unsigned int, int, unsigned int, int*)+160)
  native: pc 000000000036a367  /data/app-lib/org.ppsspp.ppsspp-1/libppsspp_jni.so (SoftGPU::ExecuteOp(unsigned int, unsigned int)+2234)
  native: pc 0000000000369a8f  /data/app-lib/org.ppsspp.ppsspp-1/libppsspp_jni.so (SoftGPU::FastRunLoop(DisplayList&)+74)
  native: pc 000000000035e741  /data/app-lib/org.ppsspp.ppsspp-1/libppsspp_jni.so (GPUCommon::InterpretList(DisplayList&)+620)

…sier to turn it off if enabled by accident. Also warn if used and it runs slow.
@Sarkie
Copy link

Sarkie commented May 18, 2017

Could you share a screenshot of the anpr stuff in the new console?

I'm really interested in this as I naively thought it'd already be there

@hrydgard
Copy link
Owner Author

hrydgard commented May 18, 2017

@Sarkie crash reporting has been there forever, but ANR reporting is new (or maybe just greatly improved). Here:

anr

When you drill down you get graphs, stack traces and stuff.

@hrydgard
Copy link
Owner Author

I'm finding a lot more variations on the deadlock above, should be fixed with that change though.

Guess it's time for 1.4.1....

@Sarkie
Copy link

Sarkie commented May 18, 2017

@hrydgard yeah I just assumed ANPRs were there as I as get them the most, especially on old devices and wondered why developers never fixed them, as I assume they run their apps with the best hardware, the ANPRs don't appear.

So this is great news.

@hrydgard
Copy link
Owner Author

hrydgard commented May 18, 2017

The travis fails seems to be a bad mac buildbot....

Oh yeah, it did have some ANR reporting before but it's definitely been improved, and made more visible which might be even more important because I had never noticed it :)

@hrydgard hrydgard merged commit 85654aa into master May 18, 2017
@hrydgard hrydgard deleted the managedtexture-reliability branch May 18, 2017 15:15
hrydgard added a commit that referenced this pull request May 18, 2017
…n interacting with the queue. Might fix further deadlocks (see #9698)
@hrydgard
Copy link
Owner Author

@Sarkie I'll fix.

Here's more details about the new stuff in the Play Console, about 10 minutes in:

https://events.google.com/io/schedule/?section=may-17&sid=a3bec051-b37b-4bca-a3bf-1a45a7e9bba2

@Sarkie
Copy link

Sarkie commented May 19, 2017

@hrydgard Thanks for the link, can't believe they only just added this!

@unknownbrackets
Copy link
Collaborator

Hm, that BATTERY_LOW one is odd.

The ManagedTexture ones aren't shutdown, though, are they? I mean, GameInfo should only destruct on shutdown, so we can see in the stack trace if that will help (not saying it's a bad change either way.)

I wonder if the software rendering is some game that has or had corruption. Possible that it may even be improved now versus v1.4.0, but who knows.

-[Unknown]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants