Synchronized Output

[christianparpart]

Hey guys,

I was searching the issues list (and my email inbox) as I remember I was once in talk with @jerch at lesat about this subject (I think), but I couldn't find it, so I want to ask here officially.

Is it of interest for you to implement the VT mode ?2026 (synchronized output)?
I am currently compiling a list of supporting software (TEs, toolkits, apps) and would like to add you there, too, if that is of interest for you.

What do you think?

[jerch]

@christianparpart Yes I am interested in that feature, I think that is a very valuable addition to the terminal interface, esp. for fullpage apps. But I cannot tell if and when we can get to it. The problem I foresee here - we have 3 renderers with very different internal semantics, that all would have to support it (maybe we could de-bounce it at higher level for all, no clue yet).

So plz dont count us in yet, until we had some discussion/thinking about it.

[Tyriar]

I can't find docs on this feature but it sounds like it would be really nice if it just prevents rendering until a block is parsed. @jerch if that's how it works we could just delay rendering until after the synchronized block is hit (by skipping the render and re-queueing to next frame), this should be renderer agnostic in RenderService.

[christianparpart]

I once summaries that for me, and yesterday (as said) tried to compile a list of supporting software. I've now moved this little document into a dedicated public page: https://gist.github.com/christianparpart/d8a62cc1ab659194337d73e399004036

This does contain the spec as well as the adoption state.

if it just prevents rendering until a block is parsed. @jerch if that's how it works we could just delay rendering until after the synchronized block is hit

I initially implemented it in my TE by queuing all VT sequences that are printable, and upon flushing the frame (CSI ? 2026 l) I was executing them in one go (atomically).
While this works I've changed this so that it always updates the internal buffer but indeed (I think this is how you'd want it), only updates the render output when CSI mode 2026 is disabled. If I must render during 2026 being enabled I just rerender the old state without fetching a fresh render buffer reflecting the current visual cells. I am not so knowledgable about HTML here, but I think it should be even easier for you.
Also a note on different render backends: Using the above suggested way (my second implementation), the 2026 mode interpretation is completely independant from the rendering implementation. Maybe it would be possible for you, too?

[jerch]

@christianparpart Thx for writing down the mechanics of that mode. So DECSET 2026 enters that mode and basically should tell the TE to wait with the screen output until DECRST 2026 or the timeout happens? Or is there some 3rd sequence I have overlooked to mark the inner "video frame" end-start border?

About possible implementation details:
Re-queueing a screen update up to some upper timeout should be easy doable for all renderers in one place (@Tyriar guess we would have to abstract requestAnimationFrame to some own impl with a max threshold referring to the real requestAnimationFrame). That would be a best-effort solution in terms of multiples of browser FPS, but not a strict sync mode.

But the sync part raises a basic question in my head - If the sync frame sequence happens within the timeout, should a TE try to render things fully in sync to the band position of the marker? Thus exactly at the position, when that marker is seen by the parser? (No print data missing, no follow-up bytes are shown).

Such a "true sync" mode is quite problematic for xterm.js, due to the inner mechanics: In theory the parser and the renderer run independently in xterm.js (beside the fact they are blocking each other on the mainthread currently, but even that might change in the future with webworkers), the renderer is only snapshotting the buffer from time to time. With a sync marker in the band data we would have to align the buffer state and the renderer tightly, thus either stall the parser until rendering happened (possible with my recent async parser patch) or do some buffer freeze copy to be shown next. Not sure yet if anything that tightly coupled is feasible.

Edit:
For "true sync" rendering imho a buffer freeze copy is the best way to deal with this, as it would allow the real terminal buffer to progress independently not blocking the parser IO. For that we could implement a "scraping" buffer on renderers to be filled/updated whenever a screen output should happen. The scraping could be triggered by our normal FPS loop in WriteBuffer or explicitly by the sync marker. The renderer would not touch the real terminal buffer anymore, but just render things from that scraping buffer with its own timing FPS logic. Schematically:

                    ┌────────┐      ┌──────────┐
   Input     parse  │        │      │          │
     │     ┌────────► Parser ├──────► Terminal │
     │     │        │        │      │  Buffer  │
     │     │        └───┬────┘      │          │
     │     │            │           └──┬───────┘
┌────▼─────┴────────┐   │              │
│                   │   │ sync request │
│ Parse/Render-Loop │   └─────────────►│ scrape
│  in WriteBuffer   │          ▲       │
│                   │          │    ┌──▼───────┐       ┌──────────┐
└──────────┬────────┘          │    │          │       │          │
           │                   │    │ Scraping ├───────► Renderer │
           │                   │    │  Buffer  │   ▲   │          │
           │                   │    │          │   │   └────┬─────┘
           │                   │    └──────────┘   │        │
           │                   │                   │        │
           └───────────────────┴───────────────────┘        ▼
                         render request                   Output

The scraping buffer pretty much works like a bit blit buffer in graphics handling, but with text cells. Ideally the renderer only depends on that buffer afterwards, thus we have a strict separation of renderer concerns from the parsing and full terminal buffer. The scraping itself will introduce additional workload, but that can be further optimized by some caching tricks (like @mofux indicated in #3368 (comment) with the diff calculation). And yes, such a separation would further pave the way for a webworker isolation of the parser, as only the scraping buffer needs to be visible from both workers (and the parse/render loop in WriteBuffer is not needed anymore).

[christianparpart]

Ideally the renderer only depends on that buffer afterwards

That is exactly how i have implemented it. Except that i call the scraping buffer a render buffer and that it is actually a double buffer (front/back) to reduce locking times. When scraping into the render buffer you render into the back buffer and the rendering engine only accesses the front buffer. Thr front/back buffers are swapped based on the screens refresh rate, whether there was any update at all, and now also if updating the back buffer and swapping is permitted (2026 disabled). (See here and there).

So yeah, that should decouple the renderer somewhat completely from there terminal as it only needs access to the (scraped) render front buffer.

Another optimization step that proved to be useful: more often than not the visible screen area contains empty grid cells (no text no SGR), these are skipped in my render buffer. So the render buffer only contains cells that do contain something.

I hope that's useful :)

[jerch]

@christianparpart Thx for the technical insights. I still stumble over the "mode" aspects of this - if I get it right, there is only an "on" and an "off" sequence, where "off" also means "paint now". Is that enough on TE side, or would it be easier to have a 3rd sequence with the meaning of "paint now", that only would be respected if 2026 is high? Thus making it a real operation mode on the TE, not just an on/off flag.
A real operation mode might come handy, if a TE has to do more preparation steps, that should stay active between the paint calls. So what is your experience here? Is the on/off flag sufficient?

[Tyriar]

FWIW the webgl renderer already uses a double buffer:

xterm.js/addons/xterm-addon-webgl/src/GlyphRenderer.ts

Lines 20 to 26 in d046e2a

	/**
	* These buffers are the ones used to bind to WebGL, the reason there are
	* multiple is to allow double buffering to work as you cannot modify the
	* buffer while it's being used by the GPU. Having multiple lets us start
	* working on the next frame.
	*/
	attributesBuffers: Float32Array[];

I think we could always trigger a render immediately when the sync start is hit without that many changes.

[jerch]

FWIW the webgl renderer already uses a double buffer:

Yeah I already suspected that. We def. should try to avoid triple+ buffering where possible 😸

[christianparpart]

I am not sure that's the same thing. Sorry for not knowing your architecture well enough. in OpenGL at least the double buffer is rendered pixels and you swap between them (always rendering to the back), whereas how I implemented Synchronized updates is that the renderer - completely decoupled from the internals of the terminal - only has access to a render buffer that contains some pre-fetch cells from the current view to the terminal's screen (scroll offset applied, selection applied - not sure how applicable that is to you though). so in normal cases, that render buffer is just refreshed on each paintGL-like call. Now, with 2026 mode enabled, fetching the render-buffer is skipped and therefore the paintGL-like call renders the exact same scene as last time, effectively implementing synchronized updates / avoiding tearing.

p.s.: I checked how others are implementing it, and at least Alacritty (though it's using a different VT sequence) seems to implement it via the same strategy. (at least they're also having the same layers).