Scalability issue: Commit loads all targets metadata into memory

[ethan-lowman-dd]

Commit calls a function fileHashes which loads all targets metadata into memory. This is not scalable if there are many targets files or targets roles.

One potential fix would be to refactor Commit and fileHashes to load only the modified targets metadata into memory. However, this might be a breaking change since Commit removes targets files that are no longer signed. With partial targets metadata, it is not possible to determine what targets files are no longer signed.

[joshuagl]

We faced similar scaleability problems in python-tuf, the changes we made to the legacy implementation to support abstract files and directories might make sense in go-tuf too?

The new python-tuf implementation builds on a metadata API which enables an integration to only load the metadata files that are being used and does not perform any activity on the files other than read/write – one can create metadata in-memory and use the API to interact with it just fine.

We don't yet have a repository API implemented in new python-tuf, though doing so is a goal, but implementing repositories with the new code feels much cleaner and easier to reason about.

[rdimitrov]

Closing since the code base changed and the goal of the project would be to focus on being a library first and not so much on providing a CLI client.

There might be a CLI based on this, but it will live in a separate repository.

Thanks for raising this 👍