Windows relative mode issues caused by clipping the cursor to the center of the window

[Susko3]

Current issues include:

• mouse grab doesn't work if window is partially off-screen (when the center of window is off screen)
• mouse warps to center when window is unfocused
• mouse sometimes warps to center of window when relative mouse is disabled

These issues are caused by clipping the cursor to the center of the window when relative mouse is enabled. Cursor clipping logic was changed so the cursor is clipped to the whole window client area in #4266, but this was silently reverted in a1fabca.
Was this revert intentional or accidental?

Clipping the cursor to the whole client area fixes all of the mentioned issues.
The proposed fix is to remove the absurdly small clip rect logic here and have relative mode use the clipping logic in the else block. Which will clip to the window client area by default. Applications which require the old clipping logic (for some reason) can use https://wiki.libsdl.org/SDL_SetWindowMouseRect to set the old rect.

More info as to why:

Mouse grab doesn't work if window is partially off-screen

If the ClipCursor rect is set off screen, Windows will ignore it. By setting the ClipCursor to the client area, windows will clip the cursor in the window area that is visible on screen (as expected).

(green = relative mode enabled, red = disabled)

video.mp4

Mouse warps to center when window is unfocused

Self-explanatory. Disabling relative mode is supposed to warp the cursor to the current SDL cursor location, but this doesn't appear to work when the window is unfocused. (Relative mode is disabled here when SDL_WINDOWEVENT_FOCUS_LOST arrives.)

Clipping to the client area means that the SDL cursor/mouse position will be the same as the one reported by Windows, so no warp/jump will occur. Though when using SDL_HINT_MOUSE_RELATIVE_SPEED_SCALE, this is still somewhat broken, for our specific use case, the scale is set to 1.0.

video.mp4

Mouse sometimes warps to center of window when relative mouse is disabled

Caused by the WinAPI SetCursorPos() sometimes not working (especially when using high pool rate - 1000Hz mouse).
This is somewhat fixed by having the clip rect be the client area as the SDL and Windows cursors will mostly stay in sync (with desync occurring if SDL_HINT_MOUSE_RELATIVE_SPEED_SCALE is not 1.0).

Explained in more detail here: #4339 (comment)

(Don't have video of this because it rarely happens.)

---

Code used for testing: https://gist.github.com/Susko3/1a82687e0e85f6cbb0fbf78e0ec110be

[DomGries]

Was this revert intentional or accidental?

I think intentional as

1. It caused the mouse leaving the window by one pixel if the app does not use dpi scaling and...
2. It increased the chance that other windows/notifications overlap the mouse which on mouse click causes app focus loss.

See #4823 (comment)

The first can be fixed by app users and we might be fine with the second if we feel that there are more advantages with your proposed solution.

[Susko3]

For 2., notifications are disabled by default in Windows while playing games, with focus assist. So these shouldn't be an issue, and user should be instructed to enable focus assist if they want uninterrupted gameplay.

Though some apps use non-native notifications, so those may be a problem (especially on older Windows versions).

Another always-on-top window that was mentioned was the taskbar. If a user has manually moved their window over the taskbar, they're kinda asking for relative mode not to work properly. But I do remember taskbar sometimes appearing over borderless fullscreen games and windows. It's unfortunate that we have to have local workarounds for Windows bugs like these.

In some cases, it may be beneficial that the mouse exits the window when another is covering it. Prime example is IME candidate window, so that the user can click on it/interact with it while entering text.

Anyways, I think that we have two (kinda) conflicting use cases for relative mode:

3D games

With relative mode enabled only during gameplay, and (presumably) disabling it when menus are open. Disabling makes sense, as the user can see their cursor to interact with the menus. The cursor can also escape the window, which is beneficial in multi monitor setups, when you can interact with other displays during non-gameplay. Examples for this UX are Minecraft and Genshin Impact (neither of which use SDL), and probably many more.

When relative mode is enabled, the cursor is thought to be logically in the center of the screen, so when you disable relative mode, you expect the cursor to appear in the center. With the current state of SDL, that is not the case. The SDL cursor freely moves within the window bounds (even if the Windows one is clipped), so when you disable relative mode, SDL will warp it to where it thinks it should be. Do we want this behaviour? (I've not programmed a 3D SDL game, do you just warp the cursor to the center before disabling relative mode to get the desired behavior?)

2D games

It's probably expected that a 2D (point and click style controls) game won't use relative mode, but we use it in osu! to get raw input while the cursor is within window bounds. For us, it makes sense that the cursor is not in the center of the screen, as it can freely move around in the game. We also allow users to change whether the cursor is confined or not. If it's not confined, we allow the cursor to leave the window by disabling relative mode when it moves past the window bounds.

We've solved "Mouse warps to center when window is unfocused" locally by warping on unfocus, but we're still fighting Windows for mouse position -- less that ideal. "Mouse sometimes warps to center of window when relative mouse is disabled" is the most noticeable problem. In addition to SDL warping the mouse when relative is disabled, we also warp locally, but unfortunately, Windows sometimes misses both those SetCursorPos() calls.

---

Maybe it would be beneficial to have a hint that changes how SDL clips the cursor in relative mode. That controls both native (Windows, etc.) and SDL cursor clipping. One mode always keeping the cursor in the middle of the screen, while the other clips it to the whole client area.

[slouken]

Constraining the mouse to the center is also important when running over RDP.

[slouken]

I've added a hint to control this, defaulting to the current behavior, in 6c96217.

[expikr]

We've solved "Mouse warps to center when window is unfocused" locally by warping on unfocus, but we're still fighting Windows for mouse position -- less that ideal. "Mouse sometimes warps to center of window when relative mouse is disabled" is the most noticeable problem. In addition to SDL warping the mouse when relative is disabled, we also warp locally, but unfortunately, Windows sometimes misses both those SetCursorPos() calls.

@Susko3 Out of curiosity, have you explored using ClipCursor(RECT(x,y,x+1,y+1));ClipCursor(NULL) instead of SetCursorPos(POINT(x,y)) before?

Also, I've always been baffled by the motivation behind Osu devs using rawinput for the usecase that it's specifically unsuited for, since software cursors are framerate-dependent, and even when you're running at high framerate it will still visibly lag behind the actual hardware cursor. Do you have any insight on the rationale behind this design decision? Or is it just a case of the devs hearing "rawinput is the future!" and misguidedly thought that adding rawinput immediately makes everything better?

[Susko3]

have you explored using ClipCursor(RECT(x,y,x+1,y+1));ClipCursor(NULL) instead of SetCursorPos(POINT(x,y)) before?

We have not, but I can imagine that causing issues. Could "eat" one frame of user input if the mouse moves right as the first call is made, causing lost input when leaving the window.

Do you have any insight on the rationale behind this design decision?

Not sure exactly how hardware cursor works. But I can imagine it only supporting a static cursor image. Which might not align all that well with osu! which may use dynamic animations and special effects with the cursor.

And, more valid, is that having custom cursor sensitivity (i.e. by reading the relative delta and warping the cursor) would invalidate all the hardware cursor gains. Since we're manually setting the cursor position, the hardware cursor would lag the same as a software cursor. Or, even worse, would be in the wrong spot!

newHardwareCursorPosition = lastFramePosition + delta * scale + hardwareCursorInterFrameDelta, which doesn't scale the hardware cursor delta.
(If my educated guess of how hardware cursor works is correct.)