ansi colors - bright colors/lscolors/dircolors - 256 color remapping

[joshtriplett]

I tried the Solarized colorscheme in both gnome-terminal and uxterm, and I ran into one major problem: Solarized seems to use several of the standard terminal colors (notably bright green and bright blue) as its "base" colors, which makes them lighter/darker versions of the base colors rather than the intended terminal colors. Notably, this means that many standard uses of terminal highlighting, such as the use of color in prompts or ls, will use the base colors rather than something distinctive.

I noticed this particularly with ls: normally, ls highlights directories in bright blue, and executables in bright green, but both of those became variants of the base color, making them much less distinctive except for the use of bold. In particular, executables prove almost completely indistinguishable from normal files, and directories appear bold but not distinguishable by color.

I like the colorscheme in general, and in particular I like that all of the same colors work in both the light and dark versions. However, the use of colors other than black and white (and their bright variants) as base colors breaks the expectations of normal terminal applications.

[altercation]

This was an intentional choice. Given that most terminal emulators allow for a specific foreground and background color in addition to the ANSI colors 0-15, there should always be a readable foreground value set to base0 in the solarized schemes.

However, you are right about the problem with lscolors/dircolors highlighting differently. I did select colors in those slots that would still be readable, even though less distinctive, in cases where there are no custom lscolors and dircolors.

So what's needed is for me to include some lscolors / dircolors that address this. Hopefully those will see the light of day in beta2 or at the latest beta3.

Consolidated into issue here:
https://github.com/altercation/solarized/issues#issue/20

[joshtriplett]

I agree that having dircolors would help. However, that doesn't address the general problem: arbitrary terminal programs expect those color slots to contain colors, not variations of the main foreground/background colors. You can fix individual programs one-by-one, but more terminal programs exist than you can fix individually. Those color slots need to contain reasonable colors.

[altercation]

josh, i'm in agreement that conforming to existing cli application expectations is best when possible, but I don't see a solution other than what I have.

Support for bright colors is all over the place, even among existing terminal applications and emulators. My heuristic was that colors 0-7 have to be fully supported and compatible, so I have acceptable, conforming hues in each of those slots. Given that this is a 16 color palette, some of the bright values will simply have to take the monotones. The only other option I see is to stick them in the 0-7 set which seems like a route with greater negative impact.

Check out the legacy bright defaults for X at: http://en.wikipedia.org/wiki/ANSI_escape_code ... there is inconsistency even there, though perhaps more out of the box contrast with fg.

I don't really see the problem with providing dircolors and lscolors that give good contrast, along with some per app colorschemes. If one doesn't want to run with the solarized colors in terminal mode I am providing 256 colorscheme versions that, while not perfect, are good replacements for the native solarized values.

I want you to know that I have thought a lot about this too, but I just don't see any other solution at this point. If you have a concrete proposal on which colors you'd like to see where, I'm certainly open to review.

[joshtriplett]

I have a few possible suggestions. First, xterm's 256-color mode supports changing colors as well, programmatically even, and since so few applications actually use 256-color mode unless explicitly configured to do so, changing the colors there would have much less impact on random terminal applications you haven't provided configuration for. So, how about taking the 256-color approximation you've provided and turning it around: remap those xterm colors to the exact solarized colors you want? Then, people can run with the 256-color "approximation" and end up with the exact solarized palette.

The following command modifies the 16 approximating colors you used from the 256-color palette, to match the solarized colors exactly. (Note that in theory xterm actually supports specifying colors as CIELab:L/a/b, since XParseColor claims to support that, but in practice that doesn't work, so you have to stick with the RGB values.)

printf '\x1B]4;234;rgb:00/2b/36;235;rgb:07/36/42;240;rgb:58/6e/75;241;rgb:65/7b/83;244;rgb:83/94/96;245;rgb:93/a1/a1;254;rgb:ee/e8/d5;230;rgb:fd/f6/e3;136;rgb:b5/89/00;166;rgb:cb/4b/16;160;rgb:dc/32/2f;125;rgb:d3/36/82;61;rgb:6c/71/c4;33;rgb:26/8b/d2;37;rgb:2a/a1/98;64;rgb:85/99/00\a'

I tested this, and it works perfectly. You can also pretty trivially generate this command from the text table in the section "The Values", since that table includes the xterm color number used for the approximation.

As for the 16-color variant, do you actually need all eight monotones for either the dark or light variants, or does having them just provide invertability? If the latter, I'd suggest providing a variant that drops the invertability of the color scheme in favor of explicit dark and light variants, which would allow replacing the monotones. If the former, any chance at all you could get by with just four monotones, and then provide colors in the "bright" slots which match the description of that slot (something greenish for bright green, something blueish for bright blue, etc). If none of those alternatives can work, I'd suggest deprecating the 16-color palette in favor of the 256-color solution above, and just providing it as an approximation for people who don't have 256-color support in their terminal.

[altercation]

Josh, good suggestions. On the iterm 256 color rewrites, I'll look into making use of that.

I'd in fact prioritize it if it were a universally applicable approach. I looked into programmatically assigning (and reading, which would be even better as I could determine ansi values and decide whether or not to fallback to 256 mode on the fly) the color table used in various terminals, but came up empty. I wasn't aware of iTerm2's support for that. I know iTerm2 can use escape codes to write the ansi colors, but where did you find the data on using CSI 4 for colortable rewriting?

Regarding the monotones, all 16 colors are in fact all used in both light and dark modes (well, 15 colors -- the distal monotone is not used) so removing them isn't an option. I'm still open to reordering if there is a better arrangement, but I did try many different arrangements in the past and this presented the fewest out of box issues across the apps I was using to test.

[joshtriplett]

See http://invisible-island.net/xterm/ctlseqs/ctlseqs.txt for documentation on xterm's control sequences. The sequence in question (OSC 4) works for both xterm and gnome-terminal, at least. I suspect it will work on all xterm-compatible terminals that support 256-color mode.

If you want to programmatically read out the colors, you can use the same control sequence, but with ? in place of the color specification. xterm will then type the control sequence back at you. However, rather than attempting to read it back in, I'd suggest just setting it.

[altercation]

Much appreciated, I should have been aware of that. I got hung up on
the proprietery esc codes in iTerm, but happy to find this is global
to xterm emulators (hopefully). I'll have to explore if other terminal
emulators have similar sequences, but I have no problem limiting this
to xterm* for now.

[joshtriplett]

I just checked, and it looks like current versions of rxvt-unicode (urxvt) can also support 256 colors (optionally), and if compiled with such support, they support the OSC 4 command on those colors.

[trapd00r]

Interesting discussion.

Changing the extended colorset works in urxvt and xterm; as long as you dont use GNU Screen or tmux.

Here's an application that gives back full hex values for all the colors: https://github.com/trapd00r/rgbterm , while https://github.com/trapd00r/colorcoke allows you to change and modify the colors in various ways; see https://github.com/trapd00r/colorcoke/wiki for some examples.

Note that, when using this approach, you cant make the changes persistent. The colors will be modified in the terminal where you changed them only.
A solution to this might be https://github.com/trapd00r/rxvt-unicode where you can define every color in .Xresources/.Xdefaults like so:

URxvt.color196: #ffff00
URxvt.color226: #ff0000

I've some templates for this here: https://github.com/trapd00r/configs/tree/master/Xresources

[altercation]

@trapd00r - another arch user I see... Interesting scripts, thanks.

This all brings me back to the issue of whether there will be a seamless and transparent solution to pushing colors via escape codes. As it stands, although we have some techniques for xterm compatible emulators, it's both temporary and requires some shell (or other) script to be executed. It's possible to call shell/perl/python scripts from various rc files but it leaves us in the same boat of having to arrange on a case by case basis support for this.

I'll continue to explore this as I find it worth looking into but I think the canonical approach at this point will continue to be "set your term colors".

[joshtriplett]

As far as I can tell, you can set the colors in the 256-color palette either via control sequences or via .Xresources, for either xterm or urxvt. Either way, this seems like it should work quite well, and seamlessly work with any terminal program for which you've provided a solarized color scheme.

[altercation]

Definitely planning on maintaining the .Xresources approach for now.
I'll keep the escape sequence push method in mind and if it looks
feasible to implement across terminals I'll look into using it as
another level of ensuring that the correct colors are present.

The real neat possibility is checking that the correct colors are set
and if not then defaulting to the failover 256 degrade mode.

Ethan Schoonover
[email protected]
http://ethanschoonover.com

On Sun, Apr 10, 2011 at 20:48, joshtriplett
[email protected]
wrote:

As far as I can tell, you can set the colors in the 256-color palette either via control sequences or via .Xresources, for either xterm or urxvt.  Either way, this seems like it should work quite well, and seamlessly work with any terminal program for which you've provided a solarized color scheme.

Reply to this email directly or view it on GitHub:
#8 (comment)

[trapd00r]

@joshtriplett: you can not set all 256 colors via .Xresources with an unmodified urxvt, nor with an xterm. See my link above.

[joshtriplett]

@trapd00r: Ah, I didn't catch that urxvt doesn't normally support .Xresources for the 256-color palette; I don't normally use urxvt. However, xterm seems to support it just fine.

[joshtriplett]

Checking the color palette proves non-trivial: you have to parse whatever output the terminal generates, and it can generate output in any format XParseColor supports, or more generally any format the terminal can accept as input. A quick glance suggests that terminals commonly output either rgb or rgba, with either 8 bits (two hex digits) or 16 bits (four hex digits) per color; they can also just output a #rrggbb string. Rather than attempting to parse those, it seems better to just always use the 256-color mode if the terminal supports it, and always set the colors unconditionally rather than checking.

[trapd00r]

@joshtriplett That was interesting. It doesn't work in xterm here; straight from the Arch repo. I'll glance through the source and configure flags to see if I'm missing something important.

[trapd00r]

Ah.

I had to set XTerm*dynamicColors to a true value, and it works indeed.

[leifwalsh]

@trapd00r I set XTerm*dynamicColors: true and ran

printf '\x1B]4;234;rgb:00/2b/36;235;rgb:07/36/42;240;rgb:58/6e/75;241;rgb:65/7b/83;244;rgb:83/94/96;245;rgb:93/a1/a1;254;rgb:ee/e8/d5;230;rgb:fd/f6/e3;136;rgb:b5/89/00;166;rgb:cb/4b/16;160;rgb:dc/32/2f;125;rgb:d3/36/82;61;rgb:6c/71/c4;33;rgb:26/8b/d2;37;rgb:2a/a1/98;64;rgb:85/99/00\a'

in a fresh xterm, and nothing happened. Did I miss something?

[trapd00r]

@adlaiff6: That works for me, both in an XTerm and in urxvt.

> perl -MTerm::ExtendedColor::Xresources=get_xterm_color -MData::Dumper -e 'print Dumper get_xterm_color({ index => [220] })'

$VAR1 = {
        '220' => {
                   'rgb' => '255/215/0',
                 }
      };


> print "\e]4;220;rgb:ff/00/00\e\\"

> perl -MTerm::ExtendedColor::Xresources=get_xterm_color -MData::Dumper -e 'print Dumper  get_xterm_color({ index => [220] })'

$VAR1 = {
          '220' => {
                     'rgb' => '255/0/0',
                   }
        };

[leifwalsh]

That works for me too, but the original line, even with the changed escapes at the beginning and end, still does nothing. Not a big deal, I'm using seebi's ls-colors, but it would be nice to get the whole 256-color palette working.

[huyz]

Just a note.

The solution of reconfiguring the 256 colors instead of the ANSI colors of course means that 8-color terminal applications such as irssi would not be able to display Solarized. So 256-color remapping is an additional choice for the user but not a replacement for the current solution of replacing the ANSI colors.

[huyz]

I've pushed a dircolors theme that will work with the current ANSI-redefinition method. So no more unexpected grays or hard-to-read colors.

https://github.com/huyz/dircolors-solarized

I will submit a pull request upstream to join the other current 256-color dircolors theme.

[seebi]

ansi themes are merged ...

[pablox-cl]

So, has something happened in this last year? There's someway we can fix this?

[Dieterbe]

+1. I'm pro modifying the 256-color palette so that the modified 256 colors resemble what ANSI intended them to be, we can use the solarized theme (at least with apps that support 256 colors, if an app only supports 16 colors they should get with the times). and the default 16-color palette would still look fine.
@trapd00r have you gotten in touch with the original urxvt authors to see what they think about your patch? of course this all depends on having the 256 color palette being actually customizeable.

[voidzero]

I'm also hoping this can be picked up again..

[adament]

http://pastebin.com/11Mw0Jsn

Based on Darsh Ranjans termcolors code and xterm's ctlseqs documentation I have written a small utility for remapping terminal palettes. It has the following features:

• Remaps 256-color palete so that approximate solarized is exact
• Remaps 16-color to solarized light or dark
• Remap colours, execute a command and after the command has finished, reset color mappings. Intended for use with 16-color applications.
• Remap default foreground, default background and textcursor colors(works only in urxvt and xterm)
• It should be easy to extend or change for arbitrary palette changes. Darsh Ranjans code is very well documented, the undocumented parts are mine.

The code is largely untested so probably it is full of bugs.

I have tested the tool in the following terminals:

• xterm - everything works
• urxvt - looks like everything works here too
• vte-based - gnome-terminal, xfce4-terminal - color remappings and resetting work, but not changing the defaults
• konsole - nothing seems to work here, I have been thinking about filing a bugreport. Since they report themselves as xterm, at least they could try supporting xterm escape sequences...