ANSI escape sequences: a possible path forward

[sunfishcode]

ANSI escape sequences are really useful, but we need a plan, both for portability and security, as escape sequences can cause a variety of havoc. Here's a skeleton of a plan; feedback welcome!

When output is connected to a terminal, by default:

• With only a few exceptions, control characters should be sanitized, perhaps replaced by control pictures.
• Control characters which are permitted are:
  • LF, but with the "newline" interpretation
  • BS, CR (for pretty progress indicators etc.). Output devices that don't support these may interpret CR the same as LF above, and emit ␈ (or ⌫?) in place of BS. Applications should avoid displaying progress indicators when the libc function isatty returns false (TODO: this isn't a simple topic), and should prefer CR to BS for progress indicators in general as CR translates better to log output.
  • HT (aka TAB).

(One could argue BEL could be included here too; I omitted it for aesthetic reasons. Feel free to disagree!). VT and FF are omitted because they aren't very useful and would add complexity.

Notably, this would neutralize ESC and thus ANSI escape sequences.

Then, we add a rights flag on a file descriptor indicating the right to use these features. When WASI gains the ability for programs to request capabilities (some early discussion here), programs could then request the ability to use ANSI escape sequences. And implementations should reset the terminal to normal settings when the program starts and terminates.

Then, we identify the escape sequences that are the most useful -- color, cursor positioning, and possibly other things. Then we have a choice -- either:

• Add an explicit API for those things, with functions that change the current color, set the cursor position, etc., or
• Allow applications to write ANSI escape sequences and interpret those that are recognized, and continue suppress those that are not. Platforms whose terminal applications don't natively support ANSI escape sequences would have to emulate them.

An API is simpler, and it makes it straightforward for applications to feature-detect which functions are available. And it makes a clean break from the ANSI escape sequence encoding, freeing us from some legacy concerns. On the other hand, just allowing select ANSI codes to go through would be better for compatibility.

As for TERM, terminfo, termcap, there's a lot of legacy there that it'd be nice to avoid carrying forward. If we define the semantics of output to a terminal, it seems like it ought to be possible to avoid supporting these interfaces at the WASI level (they could still be emulated in libc for compatibility).

For NO_COLOR, this could be implemented by wasm engines rather than by applications, either by making color a distinct rights flag, or by just stripping (or translating to gray-scale, if the output supports that) color.

There are surely a lot more details to be figured out, but I wanted to put some ideas out there to start a discussion.

[Edits: add HT (TAB), discuss VT and FF, mention that implementations should reset the terminal on startup, and add some advice for applications implementing progress bars.]

[programmerjake]

Please allow tab characters!! (assuming oversight)

[sunfishcode]

Yeah, HT (aka TAB) was an oversight. It's probably in sufficiently wide use that we don't have much choice but to allow it. It seems pretty harmless from a security POV. HT in Unix-like platforms seems to consistently do the 8-character tab-stop thing. Does anyone know what HT does in a Windows terminal?

For the record, the other two obvious whitespace-like characters are VT and FF. On various terminals I have handy, these act like line feeds (with the non-newline intepretation), which isn't very useful, so replacing them with control pictures seems fine and reduces overall complexity.

[programmerjake]

Yeah, HT (aka TAB) was an oversight. It's probably in sufficiently wide use that we don't have much choice but to allow it. It seems pretty harmless from a security POV. HT in Unix-like platforms seems to consistently do the 8-character tab-stop thing. Does anyone know what HT does in a Windows terminal?

I'm pretty sure Windows also does 8-character tab-stops.

I seem to remember being able to set custom horizontal/vertical tab stops & stuff by using escape sequences on Unix, though.

I think it may be worthwhile to note somewhere that CR is preferred over BS for pretty progress indicators since CR is much easier to handle for translating terminal output to a log file, such as on CI or similar.

[sunfishcode]

I'm pretty sure Windows also does 8-character tab-stops.

That makes it easy then :-).

I seem to remember being able to set custom horizontal/vertical tab stops & stuff by using escape sequences on Unix, though.

To be safe, a WASI implementation should probably reset terminal features like this on startup.

I think it may be worthwhile to note somewhere that CR is preferred over BS for pretty progress indicators since CR is much easier to handle for translating terminal output to a log file, such as on CI or similar.

I've now updated the post above to incorporate your feedback!

[mash-graz]

i don't think, the dangers of escape sequences resp. stdio filtering can/should be manged by WASM. if this kind of risc can be managed at all, it has to be done on the side of the actual used virtual terminal applications. otherwise it will just lead to unnecessary restrictions and a multitude of hard foreseeable side effects -- i.e. incompatibilities with mature terminal handling libraries etc. but that's just my amateurish privat opinion about this question.

[sunfishcode]

We're not really in a position to change the way virtual terminals work. We want to run untrusted code, virtual terminals aren't really designed to do that, so we need to intermediate between that code and actual terminals.

There is a basic set of escape sequences which appears pretty portable and supports colors and basic cursor positioning and a few other things. I'm now thinking it would be good for WASI to support that, and if we support features beyond that, rely on applications using terminfo to use the additional functionality.

[mash-graz]

We're not really in a position to change the way virtual terminals work. We want to run untrusted code, virtual terminals aren't really designed to do that, so we need to intermediate between that code and actual terminals.

although i'm aware of the existence of malicious terminal code sequences, i really never felt much worrying about this dangers, because i usually do not execute completely untrusted or dubious software on my my machine, but rather software which is coming from trusted (debian) repositories, trustworthy manufactures or somehow serious sources of self compiled code. sure, even in this case i could become the victim of such on attack, because someone could e.g. abuse printed meta data information in any media file to reconfigure my terminal etc.

but can we really prevent this remaining uncommon dangers without paying an unreasonable high price resp. unacceptable restrictions for the defense?

i definitely want to see the capability of WASI CLI applications to output arbitrary binary data, which can be piped and further processed by other tools on the command line of the host system. and if you preserve this very simple but tremendous useful common feature, you will hardly be able to fight this class of dangers, because the attack could always affect the [perhaps modified] output of any successive command in the pipeline...

no -- sorry -- fighting this kind of dangers on the given API layer looks to me just as hopeless as trying to establish an reliable e_mail spam protection just by means of inspecting and blocking all suspect TCP package content.

nevertheless i could perhaps appreciate the idea of some form of provided optional text output sanity check/filter, which could be utilized by some applications for additional safty. but IMHO that's better realized as an independent high level library/crate, not as mode of WASI IO operation resp. mandatory restriction.

[tschneidereit]

because i usually do not execute completely untrusted or dubious software on my my machine

The ability to do this is explicitly part of the goals of WASI. If we were to weaken this to say "we'll hold to it for everything but this set of features" (which would then presumably include ANSI escape sequences, that'd be equivalent to giving up on it entirely.

i definitely want to see the capability of WASI CLI applications to output arbitrary binary data, which can be piped and further processed by other tools on the command line of the host system

I guess it's an interesting question if writing arbitrary data to stdout might be palatable if stdout has been redirected. I'm neither sure that's a good idea, nor how to sensibly express it in languages' standard libraries, though.

no -- sorry -- fighting this kind of dangers on the given API layer looks to me just as hopeless as trying to establish an reliable e_mail spam protection just by means of inspecting and blocking all suspect TCP package content.

These things have very little in common, so I don't see how an analogy can be drawn here.

nevertheless i could perhaps appreciate the idea of some form of provided optional text output sanity check/filter, which could be utilized by some applications for additional safty. but IMHO that's better realized as an independent high level library/crate, not as mode of WASI IO operation resp. mandatory restriction.

Optional security is not what WASI is about.

What's more, the second big goal of WASI is maximal portability, and as @sunfishcode showed, ANSI escape sequences aren't portable, so integrating full support for them into WASI isn't an option for that reason, too.

As with quite a few things WASI needs to tackle, higher-level APIs for specific use cases for escape sequences might be the better approach.

[mash-graz]

no -- sorry -- fighting this kind of dangers on the given API layer looks to me just as hopeless as trying to establish an reliable e_mail spam protection just by means of inspecting and blocking all suspect TCP package content.

These things have very little in common, so I don't see how an analogy can be drawn here.

the main problem, which i wanted to express by this metaphor, should be seen in the fact, that you are in general unable to selectively filter or block only the unwanted or really malicious data without unpleasant collateral damage, as long as you are just inspecting/handling isolated small particles of raw data at a rather low layer of the communication process resp. interface to the outside world.

[tschneidereit]

Ok, that makes sense—thank you for the clarification. Can you give a concrete example of a security property that's not addressed by what @sunfishcode proposes for the specific scenario of emitting output to a terminal?

[mash-graz]

Can you give a concrete example of a security property that's not addressed by what @sunfishcode proposes for the specific scenario of emitting output to a terminal?

in fact i'm simply not good enough to give anybody a serious advice concerning such a complex security related question. i just can try bring in a few ideas and subjective estimations into debate.

searching the CVE database for "terminal" and "terminal escape" related topics i got the impression, that serious vulnerabilities (e.g. changing the window title and output it as entry on the command line etc.) in virtual terminal implementations have become a rather rare exception in the meanwhile. but this could always change again, when it suddenly becomes a fashion to compete with each other in creating/reinventing everything from scratch again, as it's unfortunately often observable e.g. in the rust community.

nevertheless a large share of the recent CVE entries concerns insufficient protection against escape sequences generated by logging facilities of network connected server processes. and although this kind of flaw will most likely do cause much harm in practice, because of the already hardened terminal implementations, it's still generally valuated as an unacceptable security risk (see: CWE-150: Improper Neutralization of Escape, Meta, or Control Sequences). so it's at the end an issue, which doesn't only affect direct terminal output, but also write operations, which contain non-printable characters without sanitizing, to open files just the same!

i still have no clear idea, how i should think about @sunfishcode's proposal to handle this issue by a gradual scale of rights flags. in this respect i'm still not able to overcame all my doubts about the actual usefulness, practical manageability and consequences concerning the portability of already existing code. but maybe i'll agree with him after few days more meditating about this puzzling issue and the clear described goals of WASI.

right now i would at least see three different kinds of desirable access modes:

• a very restrictive text mode -- maybe the default behavior -- which really doesn't allow any untranslated control characters

• a unfiltered binary mode -- to preserve portability of real unix-like CLI tools, etc.

• and perhaps a third option somewhere between both of them, which should be mainly focused on portability requirements. but in contrast to @sunfishcode's draft i wouldn't search for clear defined set of provided control features in this case, but rather suggest some kind of filtering, similar to those used in tmux, which does suppress all escape sequences, that are not supported by the actual used terminal, but otherwise preserves the full set of capabilities defined in the terminfo entry for the given device.

[joshtriplett]

@sunfishcode

• Add an explicit API for those things, with functions that change the current color, set the cursor position, etc., or

• Allow applications to write ANSI escape sequences and interpret those that are recognized, and continue suppress those that are not. Platforms whose terminal applications don't natively support ANSI escape sequences would have to emulate them.

This seems like, roughly speaking, the Windows (pre-10) model versus the UNIX (and Windows 10) model. It seems notable that Windows itself has moved away from the API-based model, for compatibility.

I do think we want to support minimally modified applications, and in particular applications that just assume escape sequences are something they can print in-band rather than API calls they have to make out-of-band. (And, of course, we have to allow arbitrary binary over file descriptors if they're not hooked up to the terminal, or even if they are but the user of the WASM runtime passes an option, since they might be using a PTY/TTY pair for other reasons or want to pass through a terminal with 100% compatibility.)

But at the same time, to filter "compatibly", we would need to filter out all but a safelist of sequences, with options to expand that safelist. And that filtering is hard to parse, and especially hard to parse rapidly. Terminals support a wide variety of escape sequences and aliases for those sequences. TTY "proxying" is hard, especially if we eventually want to support some compatibility for TTY ioctls too (like TIOCGWINSZ to get the window size).

Given that, I would propose that we do support escape sequences rather than API calls (so that applications can assume in-band sequences they can just write), but that we don't actually want to aim for 100% compatibility when in "proxying" or "filtering" mode. If people want 100% compatibility with a terminal, they should pass everything through unfiltered; with modern terminals that shouldn't be a problem. We can eventually provide API calls for the tiny number of terminal ioctls we want to support (for a start, just getting the size). Then, in "filtering" mode, we can carefully support a very small safelist, that we can always expand over time; we don't need to support every variation of how to write a given escape sequence, for instance, just one.

(I do think we should just never support any kind of escape sequence that causes the terminal to "return" data by providing it as input. Those should require API calls. That includes things like "what color is the background color".)

[hholst80]

Anything that sit between the terminal and the application, will eventually become a blocker. Add an optimal sanity checker as opt-in leave the default for normal I/O access.

[Serentty]

I think it's important to note that Windows didn't just move away from the API model for compatibility, but because it was based around the idea of terminal programs running locally as opposed to remotely, which is simply not the way that they're run a good percentage of the time, and required complicated conversion between these API calls and serialized RPCs.

I think trying to reinvent the terminal is too ambitious of a project for WASI, and is doomed to fail. That's not saying that the terminal can't be reinvented one day—just that it won't be done as a minor afterthought as part of something like WASI. ANSI escape sequences are the only realistic solution as far as I can see.

[sunfishcode]

@Serentty I agree. I think at this point in time, the thing that makes sense here is to work within how existing systems work, both for compatibility with existing terminals and for compatibility with applications. I'm working on a design along these lines.

It seems to me the most valuable thing to focus on here is adding a layer between terminals (as we know them today) and applications though. There are a lot of ways that one can cause both surprising, non-portable, and dangerous behavior using escape sequences. Escape sequences by themselves can cause problems and they can also be used to make other exploits more effective.

That said, I don't think such a layer will be as problematic as it may initially seem. The way we use terminals has evolved over the decades -- many kinds of programs which once would have used terminals have moved onto other UIs, so a lot of the things that it was once considered important to be able to do with a terminal are no longer relevant. Many features that one still uses with terminals have moved to being implemented in applications rather than relying on terminals (local editing, multiple sessions, etc.). Some features which were once important to optimize for bandwidth no longer even work in most popular terminals (eg. vertical tab).

[Serentty]

I'm not opposed to some sort of escape sequence filter as long as there is a permission that applications can be given to remove it. Safety by default, power by choice is a good philosophy.

[sunfishcode]

WebAssembly itself doesn't follow such a "Safety by default, power by choice" philosophy. There is no escape hatch for mixing in AVX-512, or disabling linear memory bounds checks when you know you're good, or doing a fast-path indirect call that skips the signature check. While that does make WebAssembly worse in some moments, it also has the effect of making it stronger over time.

WASI will always have a way to write arbitrary binary bytes to a byte stream. And presumably people will figure out ways to hook such a stream up to a terminal. But people working on standards, tools, and VMs do have choices to make. When the sandbox isn't sufficient, we'll end up in a better place if we work to extend the sandbox rather than if we work to make it easy to bypass the sandbox.

[Serentty]

Maybe we have different interpretations of what “power by choice” means. That's exactly what I view WASI's rights system as being already. As for the memory safety issues you described, the way I see it, the reason they can't be a choice is because the vulnerabilities that they would open would take away software's choice between safety and power. If software can break out of the sandbox, the embedding environment no longer has any choice as to what rights it should have. On the other hand, granting direct access to a terminal is something that a host would have to choose to do by granting an additional right above regular terminal access. Again, that's not to say that I disagree with this sandboxed terminal being the default.

I feel like I don't really have much to offer here anymore though. My big point was that I thought an API approach was shortsighted (as Windows has shown in the past), but it seems that there's already agreement there. As for how easy it is to access the terminal directly, I'm less concerned.

[sunfishcode]

There are some ideas here that I'm interested in returning to, but for practical reasons, I think it makes sense to pursue a different approach at this time, while I'll post in #161.