Serial console: WriteBufferFull error

[askfongjojo]

Hit this error on rack2 when trying to connect to the serial console of a running VM:

{"msg":"request completed","v":0,"name":"propolis-server","level":30,"time":"2023-10-05T23:34:49.11870469Z","hostname":"oxz_propolis-ser
ver_ca269f75-8ad3-4b0f-8c4e-fef91577180b","pid":16072,"uri":"/instance/serial?most_recent=262144","method":"GET","req_id":"69044e80-7320
-4841-97da-aeb8092fa590","remote_addr":"[fd00:1122:3344:103::3]:50289","local_addr":"[fd00:1122:3344:105::23]:12400","latency_us":3023,"
response_code":"101"}
{"msg":"Error returned from handler: WriteBufferFull(Frame(Frame { header: FrameHeader { is_final: true, rsv1: false, rsv2: false, rsv3:
 false, opcode: Data(Binary), mask: None }, payload: [27, 91, 50, 74, 27, 91, 48, 49, 59, 48, 49, 72, 27, 91, 61, 51, 104, 27, 91, 50, 7
4, 27, 91, 48, 49, 59, 48, 49, 72, 66, 100, 115, 68, 120, 101, 58, 32, 108, 111, 97, 100, 105, 110, 103, 32, 66, 111, 111, 116, 48, 48, 
48, 53, 32, 34, 117, 98, 117, 110, 116, 117, 34, 32, 102, 114, 111, 109, 32, 72, 68, 40, 49, 53, 44, 71, 80, 84, 44, 66, 51, 70, 56, 69,
 65, 65, 52, 45, 67, 66, 52, 55, 45, 52, 48, 68, 69, 45, 65, 50, 69, 69, 45, 57, 68, 52, 51, 50, 55, 68, 52, 51, 67, 55, 51, 44, 48, 120
, 50, 56, 48, 48, 44, 48, 120, 51, 53, 48, 48, 48, 41,...

https://oxide.sys.rack2.eng.oxide.computer/projects/try/instances/sysbench-mysql-14/serial-console

Instance info: sled 9 (BRM44220005), oxz_propolis-server_ca269f75-8ad3-4b0f-8c4e-fef91577180b

[jordanhendricks]

I notice that every instance of Angela's I looked at on the dogfood rack had inaccessible serial consoles. At first, I wondered if this was related to her instances having a long uptime (created August 30), but at least one instance I checked (6fea3a83 instance ID, 5cc7af7f propolis ID on BRM42220017) had the same error, and it was created today.

I then tried creating a new instance and saw the same behavior: the instance was up (I could ssh in), but I could not access the console via the web interface or the CLI, with the same error on the propolis side. For reference, this is what that looks like from the CLI:

$ ./target/release/oxide instance serial console --instance 22c9fa79-d382-4a4a-8a05-7558ea9767d3

Connection closed: Error
Server error: nexus: error in websocket connection to serial port: WebSocket protocol error: Connection reset without closing handshake

[leftwo]

I'm also seeing the same as above, but I can get the console history for my instance with
this command:

oxide instance serial history --instance base-image --project alan --max-bytes 65536

[pfmooney]

I believe this is caused by unnecessarily constrained websocket config parameters introduced in 9aa216fdd2990de767e484ef1b1c9e30df3785f9

Tracing through the tungstenite logic, if a write is issued which exceeds max_write_buffer_size (in addition to anything buffered), then a WriteBufferFull error would emitted. Sorry for messing this up when I did the update. PR coming shortly