client refactor: remove mirrors #1368

sechkova · 2021-04-29T09:27:39Z

Addresses #1307

Description of the changes being introduced by the pull request:
Drop "mirrors" support.
Updater no longer supports downloading each metadata file from a list of user-provided mirrors. All downloads are performed from a single metadata url.
Target files are downloaded either from a single default url or an optional url is passed for each target file on download.

More info in: comment, comment

API is changed the following way:

class Updater:
     def __init__(
         self,
         repository_name: str,
-        repository_mirrors: Dict,
+        metadata_url: str,
+        default_target_url: Optional[str] = None,
         fetcher: Optional[FetcherInterface] = None,
     ):



    def download_target(
         self,
         targetinfo: Dict,
         destination_directory: str,
+        target_url: Optional[str] = None,
     ):

Edit: The exact metdata_url and target_url format needs either better documentation for the user or better normalization functions provided by TUF.

Please verify and check that the pull request fulfills the following
requirements:

The code follows the Code Style Guidelines
Tests have been added for the bug fix or new feature
Docs have been added for the bug fix or new feature

Keep the current API and mirrors configuration but use only the first mirror from the list for metadata download. Target files download remains unchanged. Signed-off-by: Teodora Sechkova <[email protected]>

Updater now uses only a single url for metadata download. Target files download use either a default url or an optional one for each file passed by the caller. Signed-off-by: Teodora Sechkova <[email protected]>

joshuagl

Apologies for quick review, but I wanted to share some quick impressions. This is a pleasing diff stat. I think we might be able to reduce it further: the _build_full_url() / try:, download(), verify() loop is repeated a few times. Might it generalise to a helper function?

joshuagl · 2021-04-29T12:10:49Z

tuf/client_rework/updater_rework.py

+        metadata_url: str,
+        default_target_url: Optional[str] = None,


nit: these are prefixes or bases, not full urls – should we call them metadata_base_url and default_target_base_url?
Or perhaps default_base_url and targets_base_url ?

joshuagl · 2021-04-29T12:12:23Z

tuf/client_rework/updater_rework.py

+        # Should we accept metadata url as a default for targets or
+        # targets_url should be provided either in this constructor
+        # or as a download_target parameter?


I think it makes sense to default to using the same host/url combination for all downloads, unless told otherwise via the targets_base_url or the optional function parameters.

I don't think it works though: metadata base url directory should contain metadata (it's like current url_prefix+metadata_path). I would be surprised if someone wants to store targets in the same directory. I think it's reasonable to expect either a default target url prefix or the individual target download url prefixes

Jussi, I agree with you, but I think Joshua simply means the following: there is a metadata_base_url parameter that must always be explicitly set. However, there is also a targets_base_url that defaults to metadata_base_url unless also explicitly set to something else.

I think I understood: I'm just wondering if anyone would want it to default to metadata_base_url?

joshuagl · 2021-04-29T12:13:12Z

tuf/client_rework/updater_rework.py

+        if default_target_url is None:
+            self._default_target_url = metadata_url
+        else:
+            self._default_target_url = default_target_url


Nit: I'd probably write this as

Suggested change

if default_target_url is None:

self._default_target_url = metadata_url

else:

self._default_target_url = default_target_url

self._default_target_url = metadata_url

if default_target_url is not None:

self._default_target_url = default_target_url

joshuagl · 2021-04-29T12:13:48Z

tuf/client_rework/updater_rework.py

+        self,
+        targetinfo: Dict,
+        destination_directory: str,
+        target_url: Optional[str] = None,


Nit: target_url_base ?

joshuagl · 2021-04-29T12:15:21Z

tuf/client_rework/updater_rework.py

+                full_url, targetinfo["fileinfo"]["length"], self._fetcher
+            )
+            _check_file_length(temp_obj, targetinfo["fileinfo"]["length"])
+            temp_obj.seek(0)


I wonder whether _check_file_length() should do the seek(0), what do you think?

joshuagl · 2021-04-29T12:15:44Z

tuf/client_rework/updater_rework.py

+                temp_obj.close()
+            # TODO: do we reraise a NoWorkingMirrorError or just
+            # let exceptions propagate?
+            raise exceptions.NoWorkingMirrorError({full_url: e}) from e


(here and below) does NoWorkingMirrorError still make sense, given we're removing mirrors?

joshuagl · 2021-04-29T12:19:42Z

tuf/client_rework/updater_rework.py

+    # Are these steps enough? Or too much? Is this the right place?
+    filepath = parse.quote(filepath)
+    # Assuming that base_url ends with a '/' character, otherwise parse.urljoin
+    # omits (correcly) the last part of the base URL path


Suggested change

# omits (correcly) the last part of the base URL path

# omits (correctly) the last part of the base URL path

jku · 2021-04-29T14:41:49Z

This is a pleasing diff stat. I think we might be able to reduce it further

Absolutely: e.g. we can now get rid of almost all manual file object handling

jku · 2021-05-04T07:54:26Z

Since Teodora is away for the week I'll take this and continue with suggested fixes (and removing loads of temp file handling). I think I'll do a new PR and close this (but will keep these commits so review can "continue")

jku · 2021-05-04T10:12:52Z

superseded by #1373

sechkova added 2 commits April 29, 2021 09:28

Download metadata from a single mirror

80ff532

Keep the current API and mirrors configuration but use only the first mirror from the list for metadata download. Target files download remains unchanged. Signed-off-by: Teodora Sechkova <[email protected]>

Drop mirrors support

576d055

Updater now uses only a single url for metadata download. Target files download use either a default url or an optional one for each file passed by the caller. Signed-off-by: Teodora Sechkova <[email protected]>

sechkova added the experimental-client Items related to the development of a new client (see milestone/8 and theexperimental-client branch) label Apr 29, 2021

sechkova added this to the Experimental client milestone Apr 29, 2021

sechkova marked this pull request as ready for review April 29, 2021 09:36

joshuagl reviewed Apr 29, 2021

View reviewed changes

jku mentioned this pull request May 4, 2021

experimental client: Remove mirrors #1373

Merged

jku closed this May 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

client refactor: remove mirrors #1368

client refactor: remove mirrors #1368

sechkova commented Apr 29, 2021 •

edited

Loading

joshuagl left a comment

joshuagl Apr 29, 2021

joshuagl Apr 29, 2021

jku Apr 29, 2021

trishankatdatadog Apr 29, 2021

jku Apr 29, 2021

joshuagl Apr 29, 2021

joshuagl Apr 29, 2021

joshuagl Apr 29, 2021

joshuagl Apr 29, 2021

joshuagl Apr 29, 2021

jku commented Apr 29, 2021

jku commented May 4, 2021

jku commented May 4, 2021

	# omits (correcly) the last part of the base URL path
	# omits (correctly) the last part of the base URL path

client refactor: remove mirrors #1368

client refactor: remove mirrors #1368

Conversation

sechkova commented Apr 29, 2021 • edited Loading

joshuagl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jku commented Apr 29, 2021

jku commented May 4, 2021

jku commented May 4, 2021

sechkova commented Apr 29, 2021 •

edited

Loading