Pasting is broken on Windows

[ofek]

[github-actions]

Thank you for your issue. Give us a little time to review it.

PS. You might want to check the FAQ if you haven't done so already.

This is an automated reply, generated by FAQtory

[ofek]

Upon further debugging it appears like a null byte is inserted before every capital letter...

[davep]

Given what you've provided above, it's not immediately obvious to me what I'm looking at here and how I may go about recreating the situation so I can further dive into it. To start with, would you mind providing the output of textual diagnose, just so it's possible to compare versions of things involved?

Next, do you think you could provide some instructions on how to go about recreating what you're seeing, and then explaining what you'e seeing? Presumably the above is intended to show what's in your clipboard? To start with, what do you believe you put in there, and how was it placed there, and where from? Does it matter that it looks like a file path? Do you see the same effect if the file doesn't look like a file path? Things such as that.

Also, it's not clear to me what relationship the input field and its content has to the JSON below it, which I take to be evidence of the problem given your subsequent comment.

A minimal standalone program and some instructions to reproduce would be super helpful.

[ofek]

I'm using the latest commit that has been merged and to reproduce using the following snippet you may copy text from anywhere and notice that the bug occurs for capital letters and some other characters like colon:

from textual.app import App
from textual.widgets import Footer, Header, Input, TextLog

class Configure(App):
    def compose(self):
        yield Header()
        yield Input()
        yield TextLog()
        yield Footer()

    def on_input_changed(self, event):
        text_log = self.query_one(TextLog)
        text_log.clear()
        text_log.write(repr(event.input.value))

if __name__ == "__main__":
    Configure().run()

[davep]

Thanks. I'll test here when next at my desk.

As requested, to check like-for-like (or not), can we see your textual diagnose output too please?

[ofek]

[davep]

That's a screenshot of the Textual demo. I'm asking for the output of running the textual command with the diagnose parameter.

$ textual
Usage: textual [OPTIONS] COMMAND [ARGS]...

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  borders   Explore the border styles available in Textual.
  colors    Explore the design system.
  console   Run the Textual Devtools console.
  diagnose  Print information about the Textual environment
  easing    Explore the animation easing functions available in Textual.
  keys      Show key events.
  run       Run a Textual app.

It writes handy information to stdout which you can copy and paste here. Like this:

Textual Diagnostics

Versions

Name	Value
Textual	0.10.1
Rich	13.2.0

Python

Name	Value
Version	3.10.9
Implementation	CPython
Compiler	Clang 14.0.0 (clang-1400.0.29.202)
Executable	/Users/davep/develop/python/textual-sandbox/.venv/bin/python

Operating System

Name	Value
System	Darwin
Release	22.2.0
Version	Darwin Kernel Version 22.2.0: Fri Nov 11 02:08:47 PST 2022; root:xnu-8792.61.2~4/RELEASE_X86_64

Terminal

Name	Value
Terminal Application	iTerm.app (3.4.19)
TERM	xterm-256color
COLORTERM	truecolor
FORCE_COLOR	Not set
NO_COLOR	Not set

Rich Console options

Name	Value
size	width=158, height=81
legacy_windows	False
min_width	1
max_width	158
is_terminal	True
encoding	utf-8
max_height	81
justify	None
overflow	None
no_wrap	False
highlight	None
markup	None
height	None

[ofek]

The demo is what the following command gives me:

C:\Users\ofek\AppData\Local\Programs\Python\Python311\python.exe -m textual diagnose

[davep]

Yes, python -m textual will give you the demo. I'm asking that you run the command textual diagnose.

[ofek]

I expected that to run the same entry point (as other projects do) since I have many on my PATH... but here is the output:

Textual Diagnostics

Versions

Name	Value
Textual	0.10.1
Rich	13.0.1

Python

Name	Value
Version	3.11.1
Implementation	CPython
Compiler	MSC v.1934 64 bit (AMD64)
Executable	C:\USERS\OFEK\APPDATA\LOCAL\PROGRAMS\PYTHON\PYTHON311\PYTHON.EXE

Operating System

Name	Value
System	Windows
Release	10
Version	10.0.19044

Terminal

Name	Value
Terminal Application	Windows Terminal
TERM	Not set
COLORTERM	Not set
FORCE_COLOR	Not set
NO_COLOR	Not set

Rich Console options

Name	Value
size	width=199, height=41
legacy_windows	False
min_width	1
max_width	199
is_terminal	True
encoding	utf-8
max_height	41
justify	None
overflow	None
no_wrap	False
highlight	None
markup	None
height	None

[davep]

Thank you; that'll be helpful to refer back to.

[davep]

Version: 10.0.19044

Note to self: Windows 10 rather than Windows 11. (making a note as it wouldn't be the first time I've seen mention of Windows Terminal apparently misbehaving on Windows 10 but working fine on Windows 11).

[davep]

Eliminating Input from this by testing with this:

from textual.app     import App, ComposeResult
from textual.widgets import Header, Footer, TextLog
from textual.events  import Paste

class PasteEventLog( TextLog ):

    def on_paste( self, event: Paste ) -> None:
        self.write( repr( event.text ) )
        event.stop()

class PasteExplorer( App[ None ] ):

    def compose( self ) -> ComposeResult:
        yield Header()
        yield PasteEventLog()
        yield Footer()

    def on_mount( self ) -> None:
        self.query_one( PasteEventLog ).focus()

if __name__ == "__main__":
    PasteExplorer().run()

and testing with the following three different lines as three different paste events:

C:\Foo\Bar\Baz.wibble

This Is A Test

here is some text that's all lower-case

the result (Windows 11, Windows Terminal) is this:

This is also under Parallels on macOS.

Pasting the exact same things into the same app running under macOS is fine.

[willmcgugan]

That's a weird one. Trust Windows Terminal to do some things differently.

To whomever works on this, I'd try to find the spec or PR in the Windows Terminal project that explains the weird behaviour.

Non-zero change its a bug in WIndows...

[davep]

Also worth noting that this finally explains the code for stripping out NULs in the filedrop library.

[davep]

Making a tweaked version of Textual that has some debug prints sprinkled in "xterm parser" to try and see what's going on low level. With this in place:

diff --git a/src/textual/_xterm_parser.py b/src/textual/_xterm_parser.py
index 1bbec555..dc39b9cb 100644
--- a/src/textual/_xterm_parser.py
+++ b/src/textual/_xterm_parser.py
@@ -125,6 +125,7 @@ class XTermParser(Parser[events.Event]):
             use_prior_escape = False

             if bracketed_paste:
+                print(f"PASTE - APPEND BUFFER - {character!r}")
                 paste_buffer.append(character)

             self.debug_log(f"character={character!r}")
@@ -190,11 +191,13 @@ class XTermParser(Parser[events.Event]):
                         sequence
                     )
                     if bracketed_paste_start_match is not None:
+                        print(f"PASTE - BRACKETED START - {sequence!r}")
                         bracketed_paste = True
                         break

                     bracketed_paste_end_match = _re_bracketed_paste_end.match(sequence)
                     if bracketed_paste_end_match is not None:
+                        print(f"PASTE - BRACKETED END - {sequence!r}")
                         bracketed_paste = False
                         break

on macOS I see the following sequence of events if I paste in A@A:

PASTE - BRACKETED START - '\x1b[200~'
PASTE - APPEND BUFFER - 'A'
PASTE - APPEND BUFFER - '@'
PASTE - APPEND BUFFER - 'A'
PASTE - APPEND BUFFER - '\x1b'
PASTE - BRACKETED END - '\x1b[201~'
Paste(text='A@A') >>> PasteEventLog(pseudo_classes={'focus-within', 'focus', 'hover'}) method=<PasteEventLog.on_paste>

This is what I'd expect given my reading of the code.

The same code, running the same test, on Windows, however:

Key(key='ctrl+@', character='\x00', name='ctrl_@', is_printable=False) >>> PasteEventLog(pseudo_classes={'hover', 'focus-within', 'focus'}) method=<Widget.on_key>
Key(key='ctrl+@', character='\x00', name='ctrl_@', is_printable=False) >>> Screen(id='_default', pseudo_classes={'focus-within'}) method=<Widget.on_key>
Key(key='ctrl+@', character='\x00', name='ctrl_@', is_printable=False) >>> PasteExplorer(title='PasteExplorer', classes={'-dark-mode'}) method=<App.on_key>
PASTE - BRACKETED START - '\x1b[200~'
PASTE - APPEND BUFFER - '\x00'
PASTE - APPEND BUFFER - 'A'
PASTE - APPEND BUFFER - '\x00'
PASTE - APPEND BUFFER - '@'
PASTE - APPEND BUFFER - '\x00'
PASTE - APPEND BUFFER - 'A'
PASTE - APPEND BUFFER - '\x1b'
PASTE - BRACKETED END - '\x1b[201~'
Paste(text='\x00A\x00@\x00A') >>> PasteEventLog(pseudo_classes={'hover', 'focus-within', 'focus'}) method=<PasteEventLog.on_paste>

To make clear, those Key events are extra to what we see on macOS.

[davep]

Ignoring paste for a moment: on Windows, if I press a key that requires Shift, two key events happen. Let's take pressing { for a moment. When I press that I see this:

Key(key='ctrl+@', character='\x00', name='ctrl_@', is_printable=False) >>> PasteEventLog(pseudo_classes={'focus', 'focus-within'}) method=<Widget.on_key>
Key(key='left_curly_bracket', character='{', name='left_curly_bracket', is_printable=True) >>> PasteEventLog(pseudo_classes={'focus', 'focus-within'}) method=<Widget.on_key>

The Ctrl+@ apparently corresponding to the press of Shift. On the other hand, if I press [:

Key(key='left_square_bracket', character='[', name='left_square_bracket', is_printable=True) >>> PasteEventLog(pseudo_classes={'focus', 'focus-within'}) method=<Widget.on_key>

This is not the same on macOS. There I just see this:

Key(key='left_curly_bracket', character='{', name='left_curly_bracket', is_printable=True) >>> PasteEventLog(pseudo_classes={'focus', 'focus-within'}) method=<Widget.on_key>

and this:

Key(key='left_square_bracket', character='[', name='left_square_bracket', is_printable=True) >>> PasteEventLog(pseudo_classes={'focus', 'focus-within'}) method=<Widget.on_key>

So, my hypothesis at the moment is this: Windows Terminal is feeding the pasted characters into the application as if they were keystrokes (or the app is accepting them as keystrokes), and every character that would require the Shift key to be held down as a modifier gets proceeded by a Ctrl+@.

I don't know enough about the lowest-levels of Textual and the ways in which input is processed, but this is the correlation I'm seeing: any character that would require Shift to be pressed (perhaps any modifier key actually?) ends up having a connected NUL.

[willmcgugan]

This is really feeling like a bug. Bracketed paste mode support is super new in Windows Terminal.

How about we write an independant script to capture a bracketed paste, and write out the repr. So we can raise an issue in https://github.com/microsoft/terminal/

[davep]

It's very much looking that way. Meanwhile, as a workaround, how do you feel about having the paste buffer accumulation code in the xterm parser simply skip over NULs? From what I can see this would work no worse than how Windows Terminal works in other situations, and likely actually better.

For example, if I place A^@^@^@A (where ^@ is a genuine NUL) in my clipboard, and paste it into Windows Terminal at the prompt, only A turns up -- much like it's treating it like a NUL-terminated paste. Ditto if I (for example) run up Python, evaluate input() and paste.

If I do the same into the test app I posted above on macOS the result of pasting A^@^@^@A is AA:

PASTE - BRACKETED START - '\x1b[200~'
PASTE - APPEND BUFFER - 'A'
PASTE - APPEND BUFFER - 'A'
PASTE - APPEND BUFFER - '\x1b'
PASTE - BRACKETED END - '\x1b[201~'

In other words, if we strip NULs the worst case scenario we get is it'll match what you get anyway on macOS.

The conclusion being, if we do davep@c80d933 it seems like this makes Windows work just like macOS (at least as a QoL workaround for Windows users for now). Then we can look into what's going on with Windows Terminal at our leisure.

[willmcgugan]

That seems like a reasonable workaround until we find the root cause.

Would it not be more efficient to strip the NULLs on the bracketed end? That way it becomes a single str.replace

[davep]

Aye, that seems like the more sensible way to get the same effect. I'll create a PR to that effect.

[github-actions]

Don't forget to star the repository!

Follow @textualizeio for Textual updates.