Arrow keys don't work in irb

[Tyriar]

Steps to reproduce:

1. Open vscode insiders v1.9.0 (using [email protected] with [email protected])
2. Install ruby (I used this)
3. Add to settings.json:

"terminal.integrated.shell.windows": "C:\\Ruby23-x64\\bin\\irb.cmd"

4. Launch terminal ctrl+`
5. Type something and hit the left arrow key (\e[D) to move the cursor, only the ding sound is made

Downstream issue: microsoft/vscode#14577

[rprichard]

I'll look at this next week sometime.

[rprichard]

winpty doesn't attempt to set the ENHANCED_KEY flag on keypresses, but apparently it needs to for Ruby to recognize them. I'll try to figure out how to set that flag.

[rprichard]

Ruby is using rbreadline.rb to read a command line, which uses the CRT _getch / _kbhit functions. It seems these functions return the scan code bytes, and for certain keys (like functions keys / arrow keys), the byte is preceded by 0xE0 for enhanced and 0x00 otherwise. (The preceding byte might be 0xE1 in some cases too?)

[rprichard]

FWIW: I tried entering the alternate terminal mode (DECCKM) without enabling ENABLE_VIRTUAL_TERMINAL_INPUT, and it didn't result in escape sequences used for input. (I expected this result, but I thought I should test it.)

The problem seems confined to the arrow keys and the block of 6 keys (Ins, Del, Home, End, PgUp, PgDn). I'm really only familiar with US keyboards, though, so I'm unsure.

I tested with a full-size US keyboard and noticed that, with the NumLock off, the navigation keys on the keypad do not produce ENHANCED_KEY events (and therefore use a 0x00 getch prefix rather than 0xE0.) Consequently, they don't work with Ruby. I suspect that's a Ruby rbreadline bug. Here's the affected code. (\340 is octal for 0xE0.)