Some fixes for NixOS

[K900]

Please tick as appropriate:

• I have tested this code on a steam deck or on a PC
• My changes generate no new errors/warnings
• This is a bugfix/hotfix
• This is a new feature

Description

Probably better viewed commit by commit. To be honest, the Python codebase isn't in the best state, and I'd be happy to help clean it up, but I wanted to get the minimal set of changes actually useful for NixOS first.

[AAGaming00]

please do not unminify that js, the more js you send to eval the higher chance steam will crash for no reason when reloading in my experience (in addition to it slightly delaying load which isn't ideal), that's why I minified it by hand to reduce whitespace but keep readability

also, a cache key is not needed, decky disables caching for it properly using a header here

decky-loader/backend/loader.py

Line 102 in 193f97d

return web.FileResponse(file, headers={"Cache-Control": "no-cache"})

[AAGaming00]

as for cleaning up the py, I'm working on adding static types everywhere to make it more maintainable, and switching communication between frontend and backend to websocket

[AAGaming00]

Also you may want to consider some way of disabling Decky's built in updater if you want to update it via packages

[K900]

please do not unminify that js, the more js you send to eval the higher chance steam will crash for no reason when reloading in my experience (in addition to it slightly delaying load which isn't ideal), that's why I minified it by hand to reduce whitespace but keep readability

Maybe we can rework it to be in an external file that's minified at runtime? (or even at build time - I'm not sure if there's any decent tool for JS minification you can easily drive from Python).

I can inline the change for now, though.

also, a cache key is not needed, decky disables caching for it properly using a header here

A cache key is unfortunately needed, because aiohttp is trying to be clever and handle If-Modified-Since, and we set all the file mtimes to epoch 0 during build for reproducibility, so Steam always hits a 304 Not Modified. I couldn't find a better way to disable this behavior (and I have a good amount of experience gazing into the aiohttp abyss), except by lying to Steam.

as for cleaning up the py, I'm working on adding static types everywhere to make it more maintainable, and switching communication between frontend and backend to websocket

A few things I'd like to do that jumped out at me:

• replace all the os.path usage with pathlib
• encode the version of the loader into the binary itself instead of putting it in an external file
• probably add some sort of linter+formatter to run in CI to catch things like unused imports and inaccurate type hints

Also you may want to consider some way of disabling Decky's built in updater if you want to update it via packages

I want to look into that in the future, yes.

[K900]

Here's what the HTTP exchange looks like on my Deck running NixOS:

Lots of text

GET /frontend/index.js HTTP/1.1
Host: localhost:1337
Connection: keep-alive
Accept: */*
Origin: https://steamloopback.host
User-Agent: Mozilla/5.0 (X11; Linux x86_64; Valve Steam Client/default/1692390949) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36
Sec-Fetch-Site: cross-site
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: script
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
If-None-Match: "3b9aca00-10d"
If-Modified-Since: Thu, 01 Jan 1970 00:00:01 GMT

HTTP/1.1 304 Not Modified
Cache-Control: no-cache
Etag: "3b9aca00-10d"
Last-Modified: Thu, 01 Jan 1970 00:00:01 GMT
Date: Tue, 22 Aug 2023 18:23:41 GMT
Server: Python/3.10 aiohttp/3.8.5
Access-Control-Expose-Headers: Etag,Server,Date
Access-Control-Allow-Origin: https://steamloopback.host
Access-Control-Allow-Credentials: true

GET /frontend/chunk-385b4644.js HTTP/1.1
Host: localhost:1337
Connection: keep-alive
Accept: */*
Origin: https://steamloopback.host
User-Agent: Mozilla/5.0 (X11; Linux x86_64; Valve Steam Client/default/1692390949) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36
Sec-Fetch-Site: cross-site
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: script
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
If-None-Match: "3b9aca00-27ce"
If-Modified-Since: Thu, 01 Jan 1970 00:00:01 GMT

HTTP/1.1 304 Not Modified
Cache-Control: no-cache
Etag: "3b9aca00-27ce"
Last-Modified: Thu, 01 Jan 1970 00:00:01 GMT
Date: Tue, 22 Aug 2023 18:23:41 GMT
Server: Python/3.10 aiohttp/3.8.5
Access-Control-Expose-Headers: Etag,Server,Date
Access-Control-Allow-Origin: https://steamloopback.host
Access-Control-Allow-Credentials: true

GET /frontend/chunk-2b92aa30.js HTTP/1.1
Host: localhost:1337
Connection: keep-alive
Accept: */*
Origin: https://steamloopback.host
User-Agent: Mozilla/5.0 (X11; Linux x86_64; Valve Steam Client/default/1692390949) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36
Sec-Fetch-Site: cross-site
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: script
Referer: http://localhost:1337/frontend/chunk-385b4644.js
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
If-None-Match: "3b9aca00-6bc1"
If-Modified-Since: Thu, 01 Jan 1970 00:00:01 GMT

HTTP/1.1 304 Not Modified
Cache-Control: no-cache
Etag: "3b9aca00-6bc1"
Last-Modified: Thu, 01 Jan 1970 00:00:01 GMT
Date: Tue, 22 Aug 2023 18:23:42 GMT
Server: Python/3.10 aiohttp/3.8.5
Access-Control-Expose-Headers: Etag,Server,Date
Access-Control-Allow-Origin: https://steamloopback.host
Access-Control-Allow-Credentials: true

[K900]

Now that I look at it, I'm pretty sure the Cache-Control header can just be removed, because it's re-requesting everything anyway.

[AAGaming00]

Maybe we can rework it to be in an external file that's minified at runtime? (or even at build time - I'm not sure if there's any decent tool for JS minification you can easily drive from Python).

but why though? this adds extra complexity for a simple bit of js still easily readable that will not change often. if you need to audit it and really can't read that, prettify it externally. we don't need to add more stuff for the loader to handle at init.

trying to be clever and handle If-Modified-Since
interesting, we can probably just null out whatever header that corresponds to, id rather do that than add a cachebust key. can you show an http exchange where it doesn't 304 so I can compare?

encoding the version into the binary is something I've always wanted to do, but idk how to do it sanely

[K900]

but why though? this adds extra complexity for a simple bit of js still easily readable that will not change often. if you need to audit it and really can't read that, prettify it externally. we don't need to add more stuff for the loader to handle at init.

I guess it's fine if it's only ever used as a preloader of sorts. I just don't like code that's hard to diff, but this may be too much work.

interesting, we can probably just null out whatever header that corresponds to, id rather do that than add a cachebust key. can you show an http exchange where it doesn't 304 so I can compare?

It's pretty deep in aiohttp: https://github.com/aio-libs/aiohttp/blob/36bc68768bcef4ab0d95d0e8ba0047565ebc6e09/aiohttp/web_fileresponse.py#L162

We'd have to patch out a big chunk of the static file handling to avoid it, or intercept and modify the request, neither of which sounds good to me.

encoding the version into the binary is something I've always wanted to do, but idk how to do it sanely

There's a few ways, I can draft up a PR if you're interested.

[K900]

(also, just as a throwaway idea for now, would you be open to replacing aiohttp with something less, uh, spaghetti flavored?)

[AAGaming00]

I just don't like code that's hard to diff, but this may be too much work.

yeah that's fair, but it's just really a bootstrapper for the main decky script and likely won't often (if ever) be changed.

replacing aiohttp

maybe eventually.

once the websocket stuff is done (which uses aiohttp but can be ported more easily to something else) we can consider it

version encoding

yeah id be interested in a PR for that, go ahead

[K900]

Force-pushed a version with minified JS.

[K900]

@AAGaming00 so what do you think of the current state of things? I'll have time to look into better version metadata this weekend probably, but this change will already help us a lot.

[K900]

Ping? We're vendoring the patches for now, so there's no real rush, but it would be nice to get this into a release.

[SkyLeite]

I haven't tested it (because there isn't really much to test) but this seems fine to me. Having to make changes to accommodate NixOS is annoying and I would usually suggest doing this kind of patching on NixOS's side (as I've unfortunately done many times), but this seems like a fairly inconsequential change

[K900]

I'd say the cache invalidation thing should definitely be in mainline, because it's not just NixOS that will zero out timestamps for reproducibility, other distros do it too, so will hit this issue too. The DECKY_VERSION thing is fundamentally a hack that can be removed with #531.

[K900]

Not going to bother rebasing this separately, see #531.