Warning - Skipped session recording

[jakule]

I see this warning printed in Teleport logs after closing SSH session:

2022-06-17T16:52:53-04:00 DEBU [UPLOAD]    Scanned 1 uploads, started 0 in /var/lib/teleport/log/upload/streaming/default. filesessions/fileasync.go:261
2022-06-17T16:52:58-04:00 WARN [UPLOAD]    Skipped session recording 4c8bb22a-a8f4-489c-bc8e-32ecd3730608.tar. error:[session file could be corrupted or is using unsupported format: session recording 4c8bb22a-a8f4-489c-bc8e-32ecd3730608 is either corrupted or is using unsupported format, remove the file /var/lib/teleport/log/upload/streaming/default/4c8bb22a-a8f4-489c-bc8e-32ecd3730608.tar to correct the problem, remove the /var/lib/teleport/log/upload/streaming/default/4c8bb22a-a8f4-489c-bc8e-32ecd3730608.error file to retry the upload] filesessions/fileasync.go:253

Files and the content:

$ sudo ls -la /var/lib/teleport/log/upload/streaming/default/
total 208
drwxr-xr-x 3 root root   4096 Jun 17 16:49 .
drwxr-xr-x 3 root root   4096 Jun 15 17:53 ..
-rw------- 1 root root      0 Jun 17 16:22 4c8bb22a-a8f4-489c-bc8e-32ecd3730608.checkpoint
-rw------- 1 root root     94 Jun 17 16:22 4c8bb22a-a8f4-489c-bc8e-32ecd3730608.error
-rw------- 1 root root 196608 Jun 17 16:22 4c8bb22a-a8f4-489c-bc8e-32ecd3730608.tar
drwx------ 2 root root   4096 Jun 17 16:49 multi
$ sudo cat /var/lib/teleport/log/upload/streaming/default/4c8bb22a-a8f4-489c-bc8e-32ecd3730608.error
session file could be corrupted or is using unsupported format: unsupported protocol version 0

Bug details:

• Teleport version: v10 alpha
• Recreation steps: Start and stop SSH session (error is not deterministic).
• Debug logs (attached)

[Joerger]

I'm able to consistently reproduce this by terminating the Teleport Node while an SSH session is in progress, then re-starting the Teleport Node.

Edit: Interestingly, if you end the session normally but terminate the session before it manages to complete the upload, the upload will succeed on restart.

[Joerger]

@gabrielcorado This might be caused by your changes in #12916

[gabrielcorado]

This is caused because now we create an empty file when the session starts to "reserve" disk space. If the node is terminated before the complete upload is performed, this empty file is left, and when the node is started, it will fail while uploading this empty part file.

The only improvement I can see here is to update the UploadCompleter to ignore those empty part files without producing a warning message. If you have any improvements in mind, let me know.

[zmb3]

We shouldn't be leaving behind empty files. This can lead to inode exhaustion.

[zmb3]

Just a note: this happens for desktop access recordings too, so make sure we validate those when we're putting together a fix. (@ibeckermayer or I can help test)