Optimize resolve_path #6867

stsewd · 2020-04-06T23:45:25Z

This reduces: 2 queries for "normal" projects; and 4 for "complex" projects.

There are only 2 tests falling

readthedocs.org/readthedocs/rtd_tests/tests/test_resolver.py

Lines 260 to 284 in b72343f

    
           def test_resolver_no_force_translation(self): 
        
               with override_settings(USE_SUBDOMAIN=False): 
        
                   url = resolve_path( 
        
                       project=self.translation, filename='index.html', language='cz', 
        
                   ) 
        
                   self.assertEqual(url, '/docs/pip/ja/latest/index.html') 
        
               with override_settings(USE_SUBDOMAIN=True): 
        
                   url = resolve_path( 
        
                       project=self.translation, filename='index.html', language='cz', 
        
                   ) 
        
                   self.assertEqual(url, '/ja/latest/index.html') 
        
           def test_resolver_no_force_translation_with_version(self): 
        
               with override_settings(USE_SUBDOMAIN=False): 
        
                   url = resolve_path( 
        
                       project=self.translation, filename='index.html', language='cz', 
        
                       version_slug='foo', 
        
                   ) 
        
                   self.assertEqual(url, '/docs/pip/ja/foo/index.html') 
        
               with override_settings(USE_SUBDOMAIN=True): 
        
                   url = resolve_path( 
        
                       project=self.translation, filename='index.html', language='cz', 
        
                       version_slug='foo', 
        
                   ) 
        
                   self.assertEqual(url, '/ja/foo/index.html')

But can't find a place where we call the resolver with a wrong language, we usually do

readthedocs.org/readthedocs/builds/models.py

Lines 369 to 374 in 3ee8b93

    
           return self.project.get_docs_url( 
        
               version_slug=self.slug, 
        
               lang_slug=self.project.language, 
        
               private=private, 
        
               external=external, 
        
           )

Notes:

One extra query is still done for custom installations not using a subdomain
~~One 1 test pass with the old and new code, I think the other one we need to agree with what the "canonical" project is.~~ We decided to not support subprojects of translations for now.

ref #6455

stsewd · 2020-04-06T23:47:12Z

readthedocs/core/resolver.py

+        cname = (
+            cname
+            or self._use_subdomain()
+            or main_project.get_canonical_custom_domain()
+        )


I thought we were making use of the real cname, but we only use it as a boolean value to see if we need to serve from /docs instead of /.

stsewd · 2020-04-07T01:37:00Z

readthedocs/core/resolver.py

+            subproject_slug = relation.alias
+
+        return (main_project, subproject_slug)
+
    def _get_canonical_project(self, project, projects=None):


we can re-use the above method here if we don't't want to support more than 2 levels of relationships (thing that we already don't support actually...)

I don't understand this comment? re-use what method?

This method is basically the same as _get_canonical_project_data (when checking till 2 levels of relationships)

This together with: - #6867 - #6865 - #6864 reduces a lot of queries and without changing the current UI. This is a benchmark using the django debug toolbar: Before: 138 queries with 7 subprojects, 17 queries per each additional subproject With all the changes: 28 queries with 7 subprojects, 1 query per each additional subproject

ericholscher

This looks good at first, but as with all resolver changes, it scares me a bit. A few small items.

readthedocs/core/resolver.py

ericholscher · 2020-04-07T14:02:22Z

readthedocs/core/resolver.py

+            subproject_slug = relation.alias
+
+        return (main_project, subproject_slug)
+
    def _get_canonical_project(self, project, projects=None):


I don't understand this comment? re-use what method?

stsewd · 2020-04-07T16:56:13Z

but as with all resolver changes, it scares me a bit

I can split this PR into 3 PRs if that makes it more easy

ericholscher · 2020-04-09T20:48:04Z

I can split this PR into 3 PRs if that makes it more easy

I think it's just scary in general, more PRs wont make it less scary :)

ericholscher

This looks good, but I'd like to write up some QA steps in the release project before we merge it, if you have some ideas for that.

humitos · 2020-04-13T11:10:34Z

readthedocs/core/resolver.py

@@ -204,6 +188,68 @@ def resolve(
        )
        return urlunparse((protocol, domain, path, '', query_params, ''))

+    def _get_canonical_project_data(self, project):
+        """
+        Returns a tuple with (project, subproject_slug) from the canonical project of `project`.


💯 docstring!

stsewd · 2020-04-13T17:50:30Z

I have added some projects to check on the release card

stsewd added 3 commits April 6, 2020 17:52

Add tests

546f390

Optimize resolver

558b7ae

Expand docstring

ca62a45

stsewd commented Apr 6, 2020

View reviewed changes

stsewd added 2 commits April 6, 2020 18:47

Fix typo

fc9f296

Update tests

2b5c295

stsewd added the PR: work in progress Pull request is not ready for full review label Apr 7, 2020

stsewd added 3 commits April 6, 2020 19:06

Set main_language_project

93d838f

Don't support subprojects of translations

d763e88

Fix test

a4b4282

stsewd removed the PR: work in progress Pull request is not ready for full review label Apr 7, 2020

stsewd commented Apr 7, 2020

View reviewed changes

stsewd mentioned this pull request Apr 7, 2020

Don't do unnecessary queries when listing subprojects #6869

Merged

Remove tests

489d6b1

stsewd requested a review from a team April 7, 2020 01:56

ericholscher reviewed Apr 7, 2020

View reviewed changes

stsewd mentioned this pull request Apr 7, 2020

Subprojects page is throwing 502s #6455

Closed

Suggestion from review

58564a7

ericholscher approved these changes Apr 9, 2020

View reviewed changes

humitos approved these changes Apr 13, 2020

View reviewed changes

Merge branch 'master' into optimize-resolve-path

71c4ca1

stsewd merged commit ecba0f1 into master Apr 13, 2020

stsewd deleted the optimize-resolve-path branch April 13, 2020 17:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize resolve_path #6867

Optimize resolve_path #6867

stsewd commented Apr 6, 2020 •

edited

Loading

stsewd Apr 6, 2020

stsewd Apr 7, 2020

ericholscher Apr 7, 2020

stsewd Apr 7, 2020

ericholscher left a comment

ericholscher Apr 7, 2020

stsewd commented Apr 7, 2020

ericholscher commented Apr 9, 2020

ericholscher left a comment

humitos Apr 13, 2020

stsewd commented Apr 13, 2020

	def test_resolver_no_force_translation(self):
	with override_settings(USE_SUBDOMAIN=False):
	url = resolve_path(
	project=self.translation, filename='index.html', language='cz',
	)
	self.assertEqual(url, '/docs/pip/ja/latest/index.html')
	with override_settings(USE_SUBDOMAIN=True):
	url = resolve_path(
	project=self.translation, filename='index.html', language='cz',
	)
	self.assertEqual(url, '/ja/latest/index.html')

	def test_resolver_no_force_translation_with_version(self):
	with override_settings(USE_SUBDOMAIN=False):
	url = resolve_path(
	project=self.translation, filename='index.html', language='cz',
	version_slug='foo',
	)
	self.assertEqual(url, '/docs/pip/ja/foo/index.html')
	with override_settings(USE_SUBDOMAIN=True):
	url = resolve_path(
	project=self.translation, filename='index.html', language='cz',
	version_slug='foo',
	)
	self.assertEqual(url, '/ja/foo/index.html')

	return self.project.get_docs_url(
	version_slug=self.slug,
	lang_slug=self.project.language,
	private=private,
	external=external,
	)

Optimize resolve_path #6867

Optimize resolve_path #6867

Conversation

stsewd commented Apr 6, 2020 • edited Loading

stsewd Apr 6, 2020

Choose a reason for hiding this comment

stsewd Apr 7, 2020

Choose a reason for hiding this comment

ericholscher Apr 7, 2020

Choose a reason for hiding this comment

stsewd Apr 7, 2020

Choose a reason for hiding this comment

ericholscher left a comment

Choose a reason for hiding this comment

ericholscher Apr 7, 2020

Choose a reason for hiding this comment

stsewd commented Apr 7, 2020

ericholscher commented Apr 9, 2020

ericholscher left a comment

Choose a reason for hiding this comment

humitos Apr 13, 2020

Choose a reason for hiding this comment

stsewd commented Apr 13, 2020

stsewd commented Apr 6, 2020 •

edited

Loading