Better tabs detection

[lwouis]

Today, AltTab uses a full bag of tricks to detect windows which are grouped as tabs. These tricks have limitations that result in features we can't provide, and bugs where AltTab fails to show or not show certain windows.

Current implementation (very simplified and high-level)

• AltTab subscribe to the Accessibility event called kAXFocusedUIElementChangedNotification on every app
• The event triggers when the user interacts with the app UI in almost any way
• On every trigger, we make an API call to ask for the app windows
• If the windows returned by the API are different from some of the windows we already were tracking, it means some windows just became tabs

Limitations / Bugs

• If tabs are created before AltTab is launched, then we simply can't know about them. Not even as standalone windows. We won't list them until the user visits them. Before that, the OS will not return them when asking for the app's windows.
• If there are multiple tab-groups for a given app, AltTab doesn't know which tab-group has which tabs. All it knows is which windows are the active tabs of the tab-groups.
• Tab detection seems to be flacky, especially when dealing with animations like Space transitions, windows going full-screen, etc. Overall, the kAXFocusedUIElementChangedNotification doesn't cover all use-cases for tabs, and doesn't happen always at the right time (i.e. it may trigger too early or too late, and we get the wrong understanding of the OS state)

Literature

• Same ticket in yabai
• Private APIs (maybe a new one to exploit?)
  • TotalFinder and XtraFinder show 2 windows for Finder #84 (comment)

[lwouis]

FYI, we can't use the Accessibility API to simply ask the tab-group for its children. The API simply doesn't return children windows. We can only ask children windows of the whole app, and we will get a single list, with no sorting depending on which tab-group the windows are in.

Furthermore, the OS handles tabs in a lazy way. A window becoming a tab will not be resized, repositioned, and redrawned, until it is clicked/seen by the user. So after a user makes a window a tab, if they don't click on it, and we make an Accessibility call to get its size, position, or screenshot, we will get the values of when it used to be a window - so completely wrong.

[lwouis]

I looked into the APIs discussed here:

let wins = SLSCopyAssociatedWindows(cgsMainConnectionId, wid)

---

It returns the same ID as the one given as input

let query = SLSWindowQueryWindows(cgsMainConnectionId, [wid] as CFArray, 1)
let iterator = SLSWindowQueryResultCopyWindows(query)
while SLSWindowIteratorAdvance(iterator) == .success {
  let parent_wid = SLSWindowIteratorGetParentID(iterator)
  let wid = SLSWindowIteratorGetWindowID(iterator)
  let tags = SLSWindowIteratorGetTags(iterator)
}

SLSWindowIteratorGetParentID returns 0 (window is top level / has no parent, I guess)
SLSWindowIteratorGetWindowID returns the same ID as input (window has no child, I guess)
SLSWindowIteratorGetTags returns a bitmask which I don't know how to break down and what it represents

---

var windowList = [] as CFArray
var windowCount = 10 as UInt
CGSCopyWindowGroup(cgsMainConnectionId, wid, "movementGroup" as CFString, &windowList, &windowCount)

windowList is empty, and windowCount is 0

---

I think these APIs are meant to handle child windows, and not tabs.

[lwouis]

I looked into it more today:

• Using CGSCopyWindowsWithOptionsAndTags doesn't show tabbed windows
• Using CGWindowListCopyWindowInfo shows tabbed windows
  • Their kCGWindowBounds are not up-to-date (see my comment above about tabs being updated lazily by macOS)
  • Their kCGWindowAlpha, kCGWindowLayer are not helpful to discriminate tabs
  • Their kCGWindowIsOnscreen returns nil for tabbed windows, and true for non-tabbed windows. However, it may also return nil for other reasons than tabs (e.g. window is minimized), so probably unreliable
• On the AXref of a window which has tabs, there is nothing we can get except the names of the tabs (not enough to ID the tab) and their count. I tried kAXIdentifierAttribute, kAXWindowAttribute, kAXWindowsAttribute, kAXMainWindowAttribute, kAXFocusedWindowAttribute, kAXEnabledAttribute, kAXContentsAttribute, kAXTabsAttribute, but nothing is helpful here.
• There are no private API in the SDK with the word tab in it.

So at a given point of time (e.g. on initial launch), without prior knowledge, there is no way to know which windows are tabbed. Can't use their name, position, other attributes; they are not precise enough.

That's why today we look for the only event that triggers when the user tabs/un-tabs a window: kAXFocusedUIElementChangedNotification. It's really unfortunate it's the only event we can observe to know when to check if a window disappeared (which we assume is a tab then) because that event creates issue in many apps (e.g. IntelliJ, Matlab, LibreOffice, Android Studio, etc) and even in macOS itself (#718).

But I've found a replacement I think. I observed that the following APIs will list windows, or not, in the following cases:

API	Normal	Fullscreen	Other Space	Minimized	Hidden	Tabbed
CGSCopyWindowsWithOptionsAndTags	Yes	Yes	Yes	No	No	No
CGWindowListCopyWindowInfo	Yes	Yes	Yes	Yes	Yes	Yes

Interestingly CGSCopyWindowsWithOptionsAndTags doesn't list tabbed windows. It's useful because we already know which windows are minimized/hidden. So we can know for sure which ones are tabbed using this API.

The big advantage of this CGS API over the current AX API is that AX APIs can block for a long time, as the OS is asking the app for accessibility data. If the WindowServer is busy, it can take even seconds. This is why we try to do the work in the background with kAXFocusedUIElementChangedNotification normally.

But since the CGS API is pretty fast, we can run it on every AltTab trigger. This way we don't need to observe kAXFocusedUIElementChangedNotification and maintain the state of tabs. We simply decide tab/not-tab on trigger before showing the UI. I've implemented that, and it works great in my tests.

It's exciting because it should be more reliable than the flaky current approach, and importantly, it should stop breaking apps and the OS.

My only concern is: is the CGS call always fast enough, or could it delay the display of AltTab's UI?

Also: this still doesn't let AltTab list tabbed windows that existed before it was opened. This can be pursued in the scope of #447.

[koekeishiya]

@lwouis

You can pass 0x7 as the value to the 4th parameter of CGSCopyWindowsWithOptionsAndTags and it will include minimized windows. Maybe there are other combinations of values that will change which windows are included.

[yossizahn]

Since one of the recent releases, Alt-tab has started recognizing tabs in Fork and Tableplus as seperate windows.
Safari and Iterm2 tabs don't show as seperate windows.

[lwouis]

@yossizahn Safari and iTerm2 tabs aren't standard tabs. They are custom UI painted inside an NSWindow. So there is no chance AltTab can get information about them.

Imagine that it's only when you click on a non-active tab that these apps will draw their content on screen. We can't get the screenshot of these since they are not NSWindow instance and are rendered lazily when shown, on top of the same NSWindow instance.

That's why the preference is called "Show standard tabs as separate windows" and not "Show tabs as separate windows". Standard tabs are specific tech from macOS SDK that lets you merge windows together seamlessly. What these 2 apps are doing is custom UI / custom rendering of pixels on screen.

[yossizahn]

@lwouis Thanks for the reply, but I still don't understand.
I have always had "Show standard tabs as separate windows" switched off, and I was never shown tabs as seperate windows. In a recent release something changed and I am now shown tabs as seperate windows regardless of the setting.
Is this a bug or am I missing somthing?
Thanks

Edit: NVM, I just noticed #1656 it seems fixed now

[lwouis]

Yes, sorry @yossizahn, I forgot to update you here when I realized through #1656 that it was an actual regression. Hopefully you can use the latest version and your problem will be gone~

[yossizahn]

No need to apologize, thanks for a great app!