Books.app window not showing in AltTab

[zzamboni]

Describe the bug

The Books.app window is ignored by AltTab.

Screenshots / video

A short screen recording which shows the problem is attached. Note how Books does not appear in the AltTab switcher, despite it being running.

Steps to reproduce the bug

1. Open Books.app
2. Trigger AltTab
3. Note that the Books window does not appear in the switcher.

[lwouis]

Hi @zzamboni! Thanks for sharing this issue!

I was able to reproduce it. It is very strange. The app is visible to AltTab, but when using the Accessibility API to query the app's windows, it fails with error kAXErrorAttributeUnsupported. This kind of error should not be returned here, as querying an application for windows is correct.

DRM issue?

Looking online, some people are mentioning potential issues with DRM. Indeed, perhaps having accessibility apps parse the content of these books could violate some legalities.

Bug from Books.app?

Using the Accessibility Inspector (part of XCode), it properly list Books.app window:

Also weird, while asking for windows (kAXWindowsAttribute) fails, asking for the main (kAXMainWindowAttribute) or focused window (kAXFocusedWindowAttribute) returns the window. That would seem to indicate a bug, since it would be ridiculous that they protect accessing all windows, but not the main/focused window.

• Potential explanation
• Also

[lwouis]

@koekeishiya I checked if you had tickets about Books.app, and found this one. Reading this makes me think that you don't have the issue of not getting the windows with the AX API in the first place. Do you remember doing something special here, or maybe it's the Dock script injection that elevate privileges and let yabai access these?

Also during my research of these com.apple.private entitlements, I found this fascinating post about how to hijack an existing Apple .app in order to use its elevated privileges to do stuff. I'm thinking that this may interest you! It may be a simpler hoop to jump through than the other hoops we currently need, to get our apps to work.

[koekeishiya]

I can confirm that an open book is not reported through the kAXWindows attribute. However, if I open a book while yabai is running it will correctly trigger a kAXWindowCreatedNotification that passes along an AXUIElementRef that work just fine.

Regarding entitlements.. these are unfortunately not enough to be able to achieve the functionality that yabai uses the scripting-addition for. Every application on macOS has a connection to the WindowServer, and only the connection that owns a window is allowed to manipulate certain properties. The Dock is registered as a universal owner and is therefore able to alter certain properties of all windows. We also need to inject code into the Dock specifically to support space operations, as there are internal datastructures in the Dock application that has to be updated.

If you really wanted to, the Dock application could be "uninstalled" and you could re-create all mission-control/spaces functionality, cmd-tab, wallpaper drawing etc. (Oh, and gestures), and make it all custom. The underlying APIs are CGS/SLSCreateSpace,DestroySpace, and whatever else is inside CoreGraphics/SkyLight etc.

In the latest Catalina there is actually an entitlement necessary to be able to call SLSSetUniversalOwner, but prior to that you could easily do the above.

[lwouis]

I can confirm that an open book is not reported through the kAXWindows attribute. However, if I open a book while yabai is running it will correctly trigger a kAXWindowCreatedNotification that passes along an AXUIElementRef that work just fine.

What about just opening Books.app like on my screenshot, or like on the OP gif? The library view. On my system that window, which looks very normal/standard is not returned by the AX call for kAXWindowsAttribute. I find this very strange. It looks like a strict bug on the app's part. Especially since asking the kAXFocusedWindowAttribute works

[koekeishiya]

Simply opening Books.app works fine, the «library» window is detected properly.

…

Sent from my iPhone On 3 Aug 2020, at 17:11, lwouis <[email protected]> wrote:  I can confirm that an open book is not reported through the kAXWindows attribute. However, if I open a book while yabai is running it will correctly trigger a kAXWindowCreatedNotification that passes along an AXUIElementRef that work just fine. What about just opening Books.app like on my screenshot, or like on the OP gif? The library view. On my system that window, which looks very normal/standard is not returned by the AX call for kAXWindowsAttribute. I find this very strange. It looks like a strict bug on the app's part. Especially since asking the kAXFocusedWindowAttribute works — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#481 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABPDZV73NAGFRLXU6SCNXSTR63HSFANCNFSM4PTEYZVA>.

[lwouis]

HyperSwitch is detecting is properly too. It's failing to focus the window though. I wonder if they use the CG APIs as a fallback. That would explain why they can't focus the window.

I think for yabai it works because of the Dock script additions that elevate the AX API calls. I tested manual basic AX calls, and they just return nothing, so it's not about some complication in AltTab. It's the base AX calls that fail.

[koekeishiya]

The scripting-addition does not elevate the AX API, it just lets me call stuff in the SkyLight.framework using the Docks connection. All I am doing to list windows is the following AX call:

    CFTypeRef window_list_ref = NULL;
    AXUIElementCopyAttributeValue(application->ref, kAXWindowsAttribute, &window_list_ref);
    return window_list_ref;

[lwouis]

The scripting-addition does not elevate the AX API,

Oh! I thought it would. That's really strange then, because I manually did this exact AXUIElementCopyAttributeValue call, and it returns kAXErrorAttributeUnsupported and the variable is nil. What could be the difference?

[MrBaseMax]

I also do not face that trouble with the main app window, but after opening a book its window is absent.
I found smth interesting: window appear in the switcher after
a) switching to the book window with Contexts app (from the sidebar, it is visible there)
b) opening the second book. both books appear

[lwouis]

There are 2 issues at play here:

• A bug on Apple side, where for some strange reasons, the Accessibility API doesn't return the list of windows for Books.app, when queried directly. However, during Accessibility events such as "window created" or "window focused", the Books.app windows are correctly received by AltTab. This means AltTab will not be able to display Books windows if AltTab is launched after Books.app. Focusing the window should always fix the issue though, and the issue won't happen if the user launches Books.app after AltTab also.
• Books.app has animations on windows creation, for the books themselves. This means that the window is created with subrole AXUnknown then becomes AXStandardWindow after a moment. This is the only app I know of that has shift in subrole like this, thus I decided to simply whitelist Books.app rather than consider observing every subrole of every open window in case it changes to AXStandardWindow after a while.

[lwouis]

Regarding the first issue, I investigated a bit more:

The commercial app UI Browser shows Books.app windows in its Hierarchy view, but not in its Attribute or Report views, which I believe feed of AX APIs:

I was suspicious that they use the CGWindowListCopyWindowInfo API to get the list of windows, but you can double-click the row to focus it, and they have the windows's AX Attributes.

I don't understand how they are doing it. I sent them an email to ask. Maybe they'll be kind enough to help us here.

I also bought the Max Inspect app to compare Entitlements from UI Browser and AltTab. There is nothing special to see here. Both apps are non-sandboxed, notarized correctly, etc.

Finally, interesting to note that AppleScript doesn't return the AX windows either:

tell application "System Events"
    get name of every window of every process
end tell

[lwouis]

I think this issue is no longer relevant. I can see Books.app and even open books in it, and everything shows properly in AltTab v6.42.0