Detect when the user types past the last column in the terminal

[jyn514]

This has a symptom similar to #252 and that's how I found it, but the cause I think is different. When completing a line which wraps to the next line, the completed line overwrite the suggestions:

When we backspace, it deletes the suggestions on the current line:

.

If we try to tab-complete again, it gets stranger: the cursor goes to the right of the screen and parts of the prompt get written on the new line.

This is probably caused by core/comp_ui.py not taking into account the line wrap.

[jyn514]

Related to #240

[andychu]

Yeah there are a lot of bugs in this area... I am going to fix some known ones, and then we'll have to do another round of testing to see what's left :-/

Right now OSH has --completion-display=nice or =basic. I think I have to revert to the "basic" one unfortunately. The "nice" one is more like zsh, but probably has more bugs.

Both of them have to known the terminal width, which is related to the SIGWINCH issue. I noticed that the Python interpreter, OSH, and readline are all fighting over SIGWINCH.

[andychu]

I think we fixed most of the low hanging display bugs, and I plan to make a release today or tomorrow, so we can do another round of testing.

However this one is somewhat fundamental. To restate the issue clearly:

• I'm trying to make a nicer zsh- or fish- like UI where hitting tab doesn't scroll the screen every time. I think this is a much better experience.
• We do this by calling display.EraseLines() after the user hits ENTER or Ctrl-C or Ctrl-D, etc.
• We're using GNU readline, and we don't get events on every keypress. So we don't know when the user typed past the last column.
• I tested zsh and fish, and they both have some method to detect the last column. They both have their OWN line editing library and don't use GNU readline (Look into a pure Python readline replacement #185 is a big task)
  • zsh erases the next line once the cursor spills over onto it (hm maybe readline has an option for that?)
  • fish actually wraps the command onto the newline!

Temporary workarounds:

• on our side: make the "basic" display the default. It behaves more like bash.
• on the user side: you can hit Ctrl-U to clear the line if it's really distracting ... not ideal but I am willing to do that for now :-/
• on the user side: simply type \ and ENTER before you get to the end of the line :-/ I believe I fixed lal the bugs when completing on lines other than the first (?)

[andychu]

Hm I noticed this rl_event_hook() function (which CPython also uses.) It might be possible to do something super hacky and poll for being in the last column. Doesn't sound very appealing though...

https://tiswww.case.edu/php/chet/readline/readline.html#IDX238

[jyn514]

I'm not sure this is related to going off the end of the screen actually. The problem still occurs in miniature on one line. I remember seeing an issue about extra spaces on OSX, this looks related.

[andychu]

Hooks that might help solve this problem (?)

https://lists.gnu.org/archive/html/info-gnu/2014-02/msg00011.html

k.  New application-settable variable: rl_input_available_hook; function to be
    called when readline detects there is data available on its input file
    descriptor.

l.  Readline calls an application-set event hook (rl_signal_event_hook) after
    it gets a signal while reading input (read returns -1/EINTR but readline    
    does not handle the signal immediately) to allow the application to handle
    or otherwise note it.  Not currently called for SIGHUP or SIGTERM.

[andychu]

Discussion where I referenced this:

https://oilshell.zulipchat.com/#narrow/stream/121540-oil-discuss/topic/interactive.20features

I just tried out ble.sh and it seems to do the right thing. And I'm very surprised it can do the right thing in pure bash! I want to chat with @akinomyoga about how this works, as I think this is one of the things that's keeping me from using osh!

The bug right now is if you do TAB completion in osh, and the screen isn't wide enough, the current line "draws over" the completion candidates. In ble.sh the candidates move to the next line correctly.

---

And yes the first screenshot in this issue shows this pretty much.

[andychu]

note: yash also solves this problem and appears to do something pretty smart.

[andychu]

Not essential since we're cutting the interactive shell for 2020

[andychu]

Sorta related

https://github.com/romkatv/powerlevel10k#horrific-mess-when-resizing-terminal-window

[akinomyoga]

This is a nice summary of the problem of the terminal text reflowing. I also received multiple issues related to the terminal text reflow in ble.sh: akinomyoga/ble.sh#114 (comment), akinomyoga/ble.sh#142, and akinomyoga/ble.sh#154. Every time, I need to explain that this is caused by the undefined behavior of the terminal in text reflowing, and there is no perfect solution at the line-editor side. I think I can point the user to this section of powerlevel10k README next time I receive the same report.

[andychu]

Yeah now that I look more carefully at the history of this issue, and at powerlevel10k, that is a related but distinct issue.

1. Reflowing after window resize changes the vertical distance between the cursor and the start of the prompt.
2. So does typing a long line off the end of the screen, which doesn't involve a window resize (that is what this issue is about)

I think you're saying that in case 1 it's impossible to predict what the new distance is?

But I think it's possible to handle case 2 if you control the whole line editor, which OSH doesn't. OSH is using GNU readline. (And I still haven't looked into the readline hooks I mentioned above)

So I guess to sum up this issue, we can

1. either find some GNU readline hook to fix it
2. Go back to a bash style of printing a new prompt every time you hit TAB, which means you never have to go "up" again. This is very easy to implement but I think a bad experience :-(
3. maybe: just document it as a known issue? You can sort of avoid it by having a wide enough terminal :-/

I still would like to run ble.sh but I think that is further away unfortunately

[andychu]

I guess I would add that situation #2 is more common than #1.

That is, it is common to type a long line and have a narrow terminal, causing wrapping

But if I resize my window, I'm used to having to do something to "reset" the state... i.e. using vim or tmux. Usually I tend to open new windows rather than resize existing ones, but some people may have different work habits

[akinomyoga]

I think you're saying that in case 1 it's impossible to predict what the new distance is?

Yes, it's impossible for the general terminal.

But, of course, if the terminal and its version are specified, we can test the behavior of the terminal and write the code for that specific terminal. I have once checked the behavior of several terminals on the text reflow, but the problem is that every terminal has its own behavior slightly different from any other terminal. Some terminals remember that whether the automatic line folding at the end of the line has occurred or not for each line. Some terminals do not distinguish the whitespaces at the end of the line from the empty terminal cell (the state when nothing has been written to the cell). Some terminals try to become smarter and do something complicated, for which it is hard to predict the exact behavior from the external applications, etc.

But I think it's possible to handle case 2 if you control the whole line editor,

Yeah. Anyway, full control of the line editor is needed for both cases 1 and 2.

I guess it is hard to solve the problem of this issue (case 2 for completion candidates) as far as we are using GNU readline.

I still would like to run ble.sh but I think that is further away unfortunately

We could actually once run the core part of ble.sh on osh (cf Running ble.sh With Oil) [ Note: I have recently tested it on the latest version of oil, but it became not working although I guess the adjustment is not so hard]. So, actually, I guess, after a few hooks for the user input are provided by Oil, it is not so hard for me to adjust ble.sh (and probably to submit additional small fixes to Oil) so that it works as a line editor of Oil.

However, the problem that turned out then was that the initialization and the response to the user inputs were very slow with osh which is written in Python. I think another problem was the footprint (i.e., the memory use) of Oil after loading ble.sh and related modules (which currently have more than 40k SLoC in total). I thought that these problems would be solved in oil-native. I actually have never tried oil-native, but how's the current progress of oil-native?

[andychu]

So, actually, I guess, after a few hooks for the user input are provided by Oil, it is not so hard for me to adjust ble.sh (and probably to submit additional small fixes to Oil) so that it works as a line editor of Oil.

Oh really that would be very interesting -- do you have an idea of the missing features? Maybe we can discuss more on #653 or start a new issue for "ble.sh 2022 :)"

---

oil-native is making slow progress -- as of the latest release there are 1131 tests passing, out of 1900 or so.

https://www.oilshell.org/release/0.9.6/test/spec.wwz/cpp/osh-summary.html

What is good is that we can just write code in Python and that number goes up automatically.

But what is bad is I haven't had time to work on it much since March 2021, since I've been working more on the Oil language, documentation, etc.

So right now I am trying to accelerate the project by hiring a compiler engineer. I applied for 50K euros from this open internet fund:

https://nlnet.nl/news/2021/20211201-call.html

I think we will get a response by February 1st. And then I am setting up Github Sponsors / Open Collective now so we can take donations. There have been many readers for many years, so I think at least some of them will donate.

I think it helps that the project is very concrete and it can absolutely be done with enough effort... there is no "risk".

There are even some benchmarks of oil-native on small programs here: https://www.oilshell.org/release/0.9.6/benchmarks.wwz/compute/

But it still can't run large programs.

I think it is slightly slower than bash, but it is "hilariously unoptimized". I have no doubt that we could optimize it in C++ and get 2x speedup easily, and comparable memory usage to bash (maybe a little higher because of GC), but that has to happen after the 1900 spec tests pass!

---

I noticed we stopped running ble.sh tests in our build awhile ago, let me re-enable it, which is #762

[andychu]

Hm this GNU readline configuration might let us avoid the issue? It can just scroll?

Then if people want to avoid that, they can just hit \ and enter?

horizontal-scroll-mode
    This variable can be set to either `on' or `off'. Setting it to `on' means that the text of the lines being edited will scroll horizontally on a single screen line when they are longer than the width of the screen, instead of wrapping onto a new screen line. This variable is automatically set to `on' for terminals of height 1. By default, this variable is set to `off'.

[andychu]

Horizontal scroll mode seems to do a good job, feedback welcome: https://www.oilshell.org/release/0.9.8/