level 4 limits #662
level 4 limits #662
Conversation
bloodearnest
commented
Sep 28, 2023
bloodearnest
commented
- Use correct privacy strings in tests
- Limit level 4 file size
bloodearnest
force-pushed
the
level-4-limits
branch
from
97cb828 to
bcfcf0a
Compare
The only actual value that matters is `moderately_senstive`, which was what valid project.yaml's will have. Everything else is effectively `highly_senstive`. But hand written data in the tests was wrong, and might be confusing.
bloodearnest
force-pushed
the
level-4-limits
branch
from
bcfcf0a to
0bd7770
Compare
This adds a global config, `config.MAX_LEVEL4_FILESIZE` (which defaults to 16mb) which limits the size of files that will be copied to level 4 storage. The goal here is to prevent accidental copying of files to level 4 that should not be. Level 4 files should be aggregate data, and potentially reviewable/releaseable by output checkers. Datasets including pseudonymized patient level data should not be marked as `moderately_senstitive`, and this is one thing to prevent them being done so by mistake. When triggered, it communciates the fact that files have not been copied to users in two ways: 1) It includes the list of files not cpied in the job satus_message, which will be visible at jobs.opensafely.org 2) It writes file with the same name plus `.txt` file with a message, so it discoverable in level 4. It deletes this file if files are successfully copied. Fixes #298
bloodearnest
force-pushed
the
level-4-limits
branch
from
0bd7770 to
5c9cf6a
Compare
There was a problem hiding this comment.
This looks good to me. In particular, I like the fact that the max size is passed on the JobDefinition so we have the option to customise on a per-job basis further down the road. And I also like the fact that we have a mapping from excluded files to arbitrary error messages so we can introduce new reasons for excluding files later.
Very intentional, I see that coming...
It's almost like I could see the future... 🤣 |
