diff --git a/CHANGELOG.rst b/CHANGELOG.rst index f0e24d2d36c..b8c6e18632d 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -1,3 +1,39 @@ +Version 5.1.0 +------------- + +:Date: May 19, 2020 + +This release includes one major new feature which is Pageview Analytics. +This allows projects to see the pages in their docs that have been viewed in the past 30 days, +giving them an idea of what pages to focus on when updating them. + +This release also has a few small search improvements, doc updates, and other bugfixes as well. + +* `@ericholscher `__: Add a setting for storing pageviews (`#7097 `__) +* `@stsewd `__: Footer: Check for mkdocs doctype too (`#7094 `__) +* `@ericholscher `__: Fix the unresolver not working properly with root paths (`#7093 `__) +* `@stsewd `__: Privacy levels: migrate protected versions (`#7092 `__) +* `@humitos `__: Guide for Embed API (`#7089 `__) +* `@davidfischer `__: Document HSTS support (`#7083 `__) +* `@stsewd `__: Search: record queries with 0 results (`#7081 `__) +* `@stsewd `__: Search: track total results (`#7080 `__) +* `@humitos `__: Proxy embed URL (`#7079 `__) +* `@stsewd `__: Search: Little refactor (`#7076 `__) +* `@davidfischer `__: Canonical/HTTPS redirect fix (`#7075 `__) +* `@santos22 `__: Fix tests ahead of django-dynamic-fixture update (`#7073 `__) +* `@stsewd `__: Sphinx Search: don't skip indexing if one file fails (`#7071 `__) +* `@stsewd `__: Search: generate full link from the server side (`#7070 `__) +* `@ericholscher `__: Fix PR builds being marked built (`#7069 `__) +* `@ericholscher `__: Add a page about choosing between .com/.org (`#7068 `__) +* `@ericholscher `__: Release 5.0.0 (`#7064 `__) +* `@stsewd `__: Search: Index more content from sphinx (`#7063 `__) +* `@santos22 `__: Hide unbuilt versions in footer flyout (`#7056 `__) +* `@ericholscher `__: Docs: Refactor and simplify our docs (`#7052 `__) +* `@stsewd `__: Search Document: remove unused class methods (`#7035 `__) +* `@stsewd `__: Search: iterate over valid facets only (`#7034 `__) +* `@stsewd `__: RTDFacetedSearch: pass filters in one way only (`#7032 `__) +* `@dojutsu-user `__: Store Pageviews in DB (`#6121 `__) + Version 5.0.0 ------------- diff --git a/docs/_static/images/guides/sphinx-hoverxref-example.png b/docs/_static/images/guides/sphinx-hoverxref-example.png new file mode 100644 index 00000000000..fb9a90511db Binary files /dev/null and b/docs/_static/images/guides/sphinx-hoverxref-example.png differ diff --git a/docs/choosing-a-site.rst b/docs/choosing-a-site.rst new file mode 100644 index 00000000000..67ca8713ae2 --- /dev/null +++ b/docs/choosing-a-site.rst @@ -0,0 +1,47 @@ +Choosing Between Our Two Sites +============================== + +A question our users often have is what the difference is between |org_brand| and |com_brand|. +This page will lay out the functional and philosophical differences between the two sites, +which should help you choose which is a better fit for your organization. + +The features available on both platforms are the same. +The primary difference is the audience and use cases that are supported. + +Read the Docs Community +----------------------- + +|org_brand| is meant for open source projects to use for documentation hosting. +This is great for user and developer documentation for your project. + +Important points: + +* All documentation sites have advertising +* Only supports public VCS repositories +* All documentation is publicly accessible to the world +* Less build time and fewer build resources (memory & CPU) +* Documentation is organized by projects + +You can sign up for an account at https://readthedocs.org. + +Read the Docs for Business +-------------------------- + +|com_brand| is meant for companies and users who have private documentation. +It works well for product documentation as well as internal docs for your developers. + +Important points: + +* No advertising +* Allows importing private and public repositories from VCS +* Supports private versions that only your organization or people you give access to can see +* More build time and more build resources (memory & CPU) +* Documentation is organized by organization, giving more control over permissions + +You can sign up for an account at https://readthedocs.com. + +Questions? +---------- + +If you have a question about which platform would be best, +you can email us at support@readthedocs.org. diff --git a/docs/custom_domains.rst b/docs/custom_domains.rst index 07966961632..42d075fd824 100644 --- a/docs/custom_domains.rst +++ b/docs/custom_domains.rst @@ -99,6 +99,19 @@ You can also host your documentation from your own domain. .. _Amazon CAA guide: https://docs.aws.amazon.com/acm/latest/userguide/setup-caa.html +Strict Transport Security ++++++++++++++++++++++++++ + +By default, we do not return a `Strict Transport Security header`_ (HSTS) for user custom domains. +This is a conscious decision as it can be misconfigured in a not easily reversible way. +For both |org_brand| and |com_brand|, HSTS for custom domains can be set upon request. + +We always return the HSTS header with a max-age of at least one year +for our own domains including ``*.readthedocs.io``, ``*.readthedocs-hosted.com``, ``readthedocs.org`` and ``readthedocs.com``. + +.. _Strict Transport Security header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security + + Proxy SSL --------- diff --git a/docs/guides/embedding-content.rst b/docs/guides/embedding-content.rst new file mode 100644 index 00000000000..89768aaaf0f --- /dev/null +++ b/docs/guides/embedding-content.rst @@ -0,0 +1,95 @@ +Embedding Content From Your Documentation +========================================= + +Read the Docs allows you to embed content from any of the projects we host. +This allows reuse of content across sites, making sure the content is always up to date. + +There are a number of uses cases for embedding content, +so we've built our integration in a way that enables users to build on top of it. +This guide will show you some of our favorite integrations: + +.. contents:: + :local: + +Contextualized tooltips on documentation pages +---------------------------------------------- + +Tooltips on your own documentation are really useful to add more context to the current page the user is reading. +You can embed any content that is available via reference in Sphinx, including: + +* Python object references +* Full documentation pages +* Sphinx references +* Term definitions + +We built a Sphinx extension called ``sphinx-hoverxref`` on top of our Embed API +you can install in your project with minimal configuration. + +Here is an example showing a tooltip when you hover with the mouse a reference: + +.. figure:: /_static/images/guides/sphinx-hoverxref-example.png + :width: 80% + :align: center + + Tooltip shown when hovering on a reference using ``sphinx-hoverxref``. + +You can find more information about this extension, how to install and configure it in the `hoverxref documentation`_. + +.. _hoverxref documentation: https://sphinx-hoverxref.readthedocs.io/ + +Inline help on application website +---------------------------------- + +This allows us to keep the official documentation as the single source of truth, +while having great inline help in our application website as well. +On the "Automation Rules" admin page we could embed the content of our :doc:`/automation-rules` documentation +page and be sure it will be always up to date. + +.. note:: + + We recommend you point at tagged releases instead of latest. + Tags don't change over time, so you don't have to worry about the content you are embedding disappearing. + +The following example will fetch the section "Creating an automation rule" in page ``automation-rules.html`` +from our own docs and will populate the content of it into the ``#help-container`` div element. + +.. code-block:: html + + + +
+ +You can modify this example to subscribe to ``.onclick`` Javascript event, +and show a modal when the user clicks in a "Help" link. + +Calling the Embed API directly +------------------------------ + +Embed API lives under ``https://readthedocs.org/api/v2/embed/`` URL and accept two different ways of using it: + +* passing the exact URL of the section you want to embed +* sending all the attributes required as GET arguments + +The following links return exactly the same response, however the first one passes the ``url`` attribute +and the second example sends ``project``, ``version``, ``doc``, ``section`` and ``path`` as different attributes. +You can use the one that works best for your use case. + +* https://readthedocs.org/api/v2/embed/?url=https://docs.readthedocs.io/en/latest/features.html%23automatic-documentation-deployment +* https://readthedocs.org/api/v2/embed/?project=docs&version=latest&doc=features§ion=automatic-documentation-deployment&path=features.html + +You can click on these links and check the response directly in the browser. + +.. note:: + + All relative links to pages contained in the remote content will continue to point at the remote page. diff --git a/docs/guides/platform.rst b/docs/guides/platform.rst index 142829f42f8..6719af0b3a1 100644 --- a/docs/guides/platform.rst +++ b/docs/guides/platform.rst @@ -16,6 +16,7 @@ These guides will help you customize or tune aspects of Read the Docs. google-analytics hiding-a-version searching-with-readthedocs + embedding-content specifying-dependencies technical-docs-seo-guide wipe-environment diff --git a/docs/index.rst b/docs/index.rst index 7b0bd26f8f8..9cd46b86cbf 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -44,7 +44,8 @@ to help you create fantastic documentation for your project. * **Getting started**: :doc:`With Sphinx ` | :doc:`With MkDocs ` | - :doc:`Feature Overview ` + :doc:`Feature Overview ` | + :doc:`/choosing-a-site` * **Importing your existing documentation**: :doc:`Import guide ` @@ -55,11 +56,12 @@ to help you create fantastic documentation for your project. :hidden: :caption: First steps - intro/getting-started-with-sphinx - intro/getting-started-with-mkdocs + /intro/getting-started-with-sphinx + /intro/getting-started-with-mkdocs - intro/import-guide - features + /intro/import-guide + /features + /choosing-a-site Getting started with Read the Docs diff --git a/readthedocs/analytics/admin.py b/readthedocs/analytics/admin.py new file mode 100644 index 00000000000..79cdc6af7c4 --- /dev/null +++ b/readthedocs/analytics/admin.py @@ -0,0 +1,16 @@ +"""Analytics Admin classes.""" + +from django.contrib import admin + +from .models import PageView + + +class PageViewAdmin(admin.ModelAdmin): + raw_id_fields = ('project', 'version') + list_display = ('project', 'version', 'path', 'view_count', 'date') + search_fields = ('project__slug', 'version__slug', 'path') + readonly_fields = ('date',) + list_select_related = ('project', 'version', 'version__project') + + +admin.site.register(PageView, PageViewAdmin) diff --git a/readthedocs/analytics/migrations/0001_initial.py b/readthedocs/analytics/migrations/0001_initial.py new file mode 100644 index 00000000000..18ab14383df --- /dev/null +++ b/readthedocs/analytics/migrations/0001_initial.py @@ -0,0 +1,32 @@ +# Generated by Django 2.2.12 on 2020-05-19 00:45 + +import datetime +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + initial = True + + dependencies = [ + ('builds', '0022_migrate_protected_versions'), + ('projects', '0048_remove_version_privacy_field'), + ] + + operations = [ + migrations.CreateModel( + name='PageView', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('path', models.CharField(max_length=4096)), + ('view_count', models.PositiveIntegerField(default=0)), + ('date', models.DateField(db_index=True, default=datetime.date.today)), + ('project', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='page_views', to='projects.Project')), + ('version', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='page_views', to='builds.Version', verbose_name='Version')), + ], + options={ + 'unique_together': {('project', 'version', 'path', 'date')}, + }, + ), + ] diff --git a/readthedocs/analytics/migrations/__init__.py b/readthedocs/analytics/migrations/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/readthedocs/analytics/models.py b/readthedocs/analytics/models.py new file mode 100644 index 00000000000..c813b21b43e --- /dev/null +++ b/readthedocs/analytics/models.py @@ -0,0 +1,130 @@ +"""Analytics modeling to help understand the projects on Read the Docs.""" + +import datetime + +from django.db import models +from django.db.models import Sum +from django.utils import timezone +from django.utils.translation import ugettext_lazy as _ + +from readthedocs.builds.models import Version +from readthedocs.projects.models import Project + + +def _last_30_days_iter(): + """Returns iterator for previous 30 days (including today).""" + thirty_days_ago = timezone.now().date() - timezone.timedelta(days=30) + + # this includes the current day, len() = 31 + return (thirty_days_ago + timezone.timedelta(days=n) for n in range(31)) + + +class PageView(models.Model): + + """PageView counts per day for a project, version, and path.""" + + project = models.ForeignKey( + Project, + related_name='page_views', + on_delete=models.CASCADE, + ) + version = models.ForeignKey( + Version, + verbose_name=_('Version'), + related_name='page_views', + on_delete=models.CASCADE, + ) + path = models.CharField(max_length=4096) + view_count = models.PositiveIntegerField(default=0) + date = models.DateField(default=datetime.date.today, db_index=True) + + class Meta: + unique_together = ("project", "version", "path", "date") + + def __str__(self): + return f'PageView: [{self.project.slug}:{self.version.slug}] - {self.path} for {self.date}' + + @classmethod + def top_viewed_pages(cls, project, since=None): + """ + Returns top 10 pages according to view counts. + + Structure of returned data is compatible to make graphs. + Sample returned data:: + { + 'pages': ['index', 'config-file/v1', 'intro/import-guide'], + 'view_counts': [150, 120, 100] + } + This data shows that `index` is the most viewed page having 150 total views, + followed by `config-file/v1` and `intro/import-guide` having 120 and + 100 total page views respectively. + """ + if since is None: + since = timezone.now().date() - timezone.timedelta(days=30) + + qs = ( + cls.objects + .filter(project=project, date__gte=since) + .values_list('path') + .annotate(total_views=Sum('view_count')) + .values_list('path', 'total_views') + .order_by('-total_views')[:10] + ) + + pages = [] + view_counts = [] + + for data in qs.iterator(): + pages.append(data[0]) + view_counts.append(data[1]) + + final_data = { + 'pages': pages, + 'view_counts': view_counts, + } + + return final_data + + @classmethod + def page_views_by_date(cls, project_slug, since=None): + """ + Returns the total page views count for last 30 days for a particular project. + + Structure of returned data is compatible to make graphs. + Sample returned data:: + { + 'labels': ['01 Jul', '02 Jul', '03 Jul'], + 'int_data': [150, 200, 143] + } + This data shows that there were 150 page views on 01 July, + 200 page views on 02 July and 143 page views on 03 July. + """ + if since is None: + since = timezone.now().date() - timezone.timedelta(days=30) + + qs = cls.objects.filter( + project__slug=project_slug, + date__gt=since, + ).values('date').annotate(total_views=Sum('view_count')).order_by('date') + + count_dict = dict( + qs.order_by('date').values_list('date', 'total_views') + ) + + # This fills in any dates where there is no data + # to make sure we have a full 30 days of dates + count_data = [count_dict.get(date) or 0 for date in _last_30_days_iter()] + + # format the date value to a more readable form + # Eg. `16 Jul` + last_30_days_str = [ + timezone.datetime.strftime(date, '%d %b') + for date in _last_30_days_iter() + ] + + final_data = { + 'labels': last_30_days_str, + 'int_data': count_data, + } + + return final_data diff --git a/readthedocs/analytics/tasks.py b/readthedocs/analytics/tasks.py index cb58fbdf71d..17f3fd170d0 100644 --- a/readthedocs/analytics/tasks.py +++ b/readthedocs/analytics/tasks.py @@ -3,10 +3,15 @@ """Tasks for Read the Docs' analytics.""" from django.conf import settings +from django.db.models import F +from django.utils import timezone import readthedocs from readthedocs.worker import app +from readthedocs.builds.models import Version +from readthedocs.projects.models import Project +from .models import PageView from .utils import send_to_analytics @@ -70,3 +75,30 @@ def analytics_event( data.update(DEFAULT_PARAMETERS) data.update(kwargs) send_to_analytics(data) + + +@app.task(queue='web') +def increase_page_view_count(project_slug, version_slug, path): + """Increase the page view count for the given project.""" + project = Project.objects.get(slug=project_slug) + + page_view, _ = PageView.objects.get_or_create( + project=project, + version=Version.objects.get(project=project, slug=version_slug), + path=path, + date=timezone.now().date(), + ) + PageView.objects.filter(pk=page_view.pk).update( + view_count=F('view_count') + 1 + ) + + +@app.task(queue='web') +def delete_old_page_counts(): + """ + Delete page counts older than 30 days. + + This is intended to run from a periodic task daily. + """ + thirty_days_ago = timezone.now().date() - timezone.timedelta(days=30) + return PageView.objects.filter(date__lt=thirty_days_ago).delete() diff --git a/readthedocs/analytics/tests.py b/readthedocs/analytics/tests.py index eb0adcf5fa0..cd20d3880a4 100644 --- a/readthedocs/analytics/tests.py +++ b/readthedocs/analytics/tests.py @@ -1,6 +1,14 @@ -# -*- coding: utf-8 -*- +from unittest import mock + +from django_dynamic_fixture import get from django.test import TestCase, RequestFactory +from django.utils import timezone + +from readthedocs.builds.models import Version +from readthedocs.projects.models import Project +from .models import PageView +from .tasks import increase_page_view_count from .utils import ( anonymize_ip_address, anonymize_user_agent, @@ -87,3 +95,82 @@ def test_get_client_ip_with_remote_addr(self): request.META['REMOTE_ADDR'] = '203.0.113.195' client_ip = get_client_ip(request) self.assertEqual(client_ip, '203.0.113.195') + + +class AnalyticsTasksTests(TestCase): + def test_increase_page_view_count(self): + project = get( + Project, + slug='project-1', + ) + version = get(Version, slug='1.8', project=project) + + today = timezone.now() + tomorrow = timezone.now() + timezone.timedelta(days=1) + yesterday = timezone.now() - timezone.timedelta(days=1) + + assert ( + PageView.objects.all().count() == 0 + ), 'There\'s no PageView object created yet.' + + # testing for yesterday + with mock.patch('readthedocs.analytics.tasks.timezone.now') as mocked_timezone: + mocked_timezone.return_value = yesterday + + increase_page_view_count( + project_slug=project.slug, + version_slug=version.slug, + path='index', + ) + + assert ( + PageView.objects.all().count() == 1 + ), 'PageView object for path \'index\' is created' + assert ( + PageView.objects.all().first().view_count == 1 + ), '\'index\' has 1 view' + + increase_page_view_count( + project_slug=project.slug, + version_slug=version.slug, + path='index', + ) + + assert ( + PageView.objects.all().count() == 1 + ), 'PageView object for path \'index\' is already created' + assert ( + PageView.objects.all().first().view_count == 2 + ), '\'index\' has 2 views now' + + # testing for today + with mock.patch('readthedocs.analytics.tasks.timezone.now') as mocked_timezone: + mocked_timezone.return_value = today + increase_page_view_count( + project_slug=project.slug, + version_slug=version.slug, + path='index', + ) + + assert ( + PageView.objects.all().count() == 2 + ), 'PageView object for path \'index\' is created for two days (yesterday and today)' + assert ( + PageView.objects.all().order_by('-date').first().view_count == 1 + ), '\'index\' has 1 view today' + + # testing for tomorrow + with mock.patch('readthedocs.analytics.tasks.timezone.now') as mocked_timezone: + mocked_timezone.return_value = tomorrow + increase_page_view_count( + project_slug=project.slug, + version_slug=version.slug, + path='index', + ) + + assert ( + PageView.objects.all().count() == 3 + ), 'PageView object for path \'index\' is created for three days (yesterday, today & tomorrow)' + assert ( + PageView.objects.all().order_by('-date').first().view_count == 1 + ), '\'index\' has 1 view tomorrow' diff --git a/readthedocs/api/v2/proxied_urls.py b/readthedocs/api/v2/proxied_urls.py index 4a01e23a581..5ec02bb73fd 100644 --- a/readthedocs/api/v2/proxied_urls.py +++ b/readthedocs/api/v2/proxied_urls.py @@ -5,6 +5,7 @@ so they can make use of features that require to have access to their cookies. """ +from django.conf import settings from django.conf.urls import include, url from .views.proxied import ProxiedFooterHTML @@ -16,3 +17,8 @@ ] urlpatterns = api_footer_urls + +if 'readthedocsext.embed' in settings.INSTALLED_APPS: + urlpatterns += [ + url(r'embed/', include('readthedocsext.embed.urls')) + ] diff --git a/readthedocs/api/v2/utils.py b/readthedocs/api/v2/utils.py index a9fcffea6d8..e9f216cec44 100644 --- a/readthedocs/api/v2/utils.py +++ b/readthedocs/api/v2/utils.py @@ -153,7 +153,8 @@ def delete_versions(project, version_data): if to_delete_qs.count(): ret_val = {obj.slug for obj in to_delete_qs} - log.info('(Sync Versions) Deleted Versions: [%s]', ' '.join(ret_val)) + log.info('(Sync Versions) Deleted Versions: project=%s, versions=[%s]', + project.slug, ' '.join(ret_val)) to_delete_qs.delete() return ret_val return set() diff --git a/readthedocs/api/v2/views/footer_views.py b/readthedocs/api/v2/views/footer_views.py index 644d6c7f36a..73611a16d22 100644 --- a/readthedocs/api/v2/views/footer_views.py +++ b/readthedocs/api/v2/views/footer_views.py @@ -15,11 +15,13 @@ from readthedocs.builds.constants import LATEST, TAG from readthedocs.builds.models import Version from readthedocs.core.utils.extend import SettingsOverrideObject -from readthedocs.projects.models import Project +from readthedocs.projects.constants import MKDOCS, SPHINX_HTMLDIR +from readthedocs.projects.models import Project, Feature from readthedocs.projects.version_handling import ( highest_version, parse_version_failsafe, ) +from readthedocs.analytics.tasks import increase_page_view_count def get_version_compare_data(project, base_version=None): @@ -147,7 +149,7 @@ def _get_context(self): page_slug = self.request.GET.get('page', '') path = '' if page_slug and page_slug != 'index': - if version.documentation_type == 'sphinx_htmldir': + if version.documentation_type in {SPHINX_HTMLDIR, MKDOCS}: path = re.sub('/index$', '', page_slug) + '/' else: path = page_slug + '.html' @@ -221,6 +223,15 @@ def get(self, request, format=None): 'version_supported': version.supported, } + # increase the page view count for the given page + page_slug = request.GET.get('page', '') + if page_slug and project.has_feature(Feature.STORE_PAGEVIEWS): + increase_page_view_count.delay( + project_slug=context['project'].slug, + version_slug=context['version'].slug, + path=page_slug + ) + # Allow folks to hook onto the footer response for various information # collection, or to modify the resp_data. footer_response.send( diff --git a/readthedocs/api/v3/tests/test_versions.py b/readthedocs/api/v3/tests/test_versions.py index 3f112724a51..4ee2d95e95b 100644 --- a/readthedocs/api/v3/tests/test_versions.py +++ b/readthedocs/api/v3/tests/test_versions.py @@ -8,7 +8,7 @@ from readthedocs.projects.models import Project -class VerionsEndpointTests(APIEndpointMixin): +class VersionsEndpointTests(APIEndpointMixin): def test_projects_versions_list(self): self.client.credentials(HTTP_AUTHORIZATION=f'Token {self.token.key}') diff --git a/readthedocs/builds/migrations/0022_migrate_protected_versions.py b/readthedocs/builds/migrations/0022_migrate_protected_versions.py new file mode 100644 index 00000000000..c548edfc9ad --- /dev/null +++ b/readthedocs/builds/migrations/0022_migrate_protected_versions.py @@ -0,0 +1,29 @@ +# Generated by Django 2.2.12 on 2020-05-18 20:06 + +from django.db import migrations +from django.conf import settings + + +def forwards_func(apps, schema_editor): + """ + Migrate all protected versions. + + For .org, we mark them as public, + and for .com we mark them as private. + """ + Version = apps.get_model('builds', 'Version') + target_privacy_level = 'private' if settings.ALLOW_PRIVATE_REPOS else 'public' + Version.objects.filter(privacy_level='protected').update( + privacy_level=target_privacy_level, + ) + + +class Migration(migrations.Migration): + + dependencies = [ + ('builds', '0021_make_hidden_field_not_null'), + ] + + operations = [ + migrations.RunPython(forwards_func), + ] diff --git a/readthedocs/builds/querysets.py b/readthedocs/builds/querysets.py index 79f2362b6b3..ecaadd64d0e 100644 --- a/readthedocs/builds/querysets.py +++ b/readthedocs/builds/querysets.py @@ -24,7 +24,8 @@ def _add_user_repos(self, queryset, user): queryset = user_queryset | queryset return queryset - def public(self, user=None, project=None, only_active=True, include_hidden=True): + def public(self, user=None, project=None, only_active=True, + include_hidden=True, only_built=False): queryset = self.filter(privacy_level=constants.PUBLIC) if user: queryset = self._add_user_repos(queryset, user) @@ -32,6 +33,8 @@ def public(self, user=None, project=None, only_active=True, include_hidden=True) queryset = queryset.filter(project=project) if only_active: queryset = queryset.filter(active=True) + if only_built: + queryset = queryset.filter(built=True) if not include_hidden: queryset = queryset.filter(hidden=False) return queryset.distinct() diff --git a/readthedocs/core/static-src/core/js/doc-embed/search.js b/readthedocs/core/static-src/core/js/doc-embed/search.js index e16cddf4671..d3b774ce749 100644 --- a/readthedocs/core/static-src/core/js/doc-embed/search.js +++ b/readthedocs/core/static-src/core/js/doc-embed/search.js @@ -74,14 +74,7 @@ function attach_elastic_search_query(data) { } } - // Creating the result from elements - var suffix = DOCUMENTATION_OPTIONS.FILE_SUFFIX; - // Since sphinx 2.2.1 FILE_SUFFIX is .html for all builders, - // and there is a new BUILDER option. - if ('BUILDER' in DOCUMENTATION_OPTIONS && DOCUMENTATION_OPTIONS.BUILDER === 'readthedocsdirhtml') { - suffix = ''; - } - var link = doc.link + suffix + "?highlight=" + $.urlencode(query); + var link = doc.link + "?highlight=" + $.urlencode(query); var item = $('', {'href': link}); diff --git a/readthedocs/core/static/core/js/readthedocs-doc-embed.js b/readthedocs/core/static/core/js/readthedocs-doc-embed.js index 19496750196..97675bf4529 100644 --- a/readthedocs/core/static/core/js/readthedocs-doc-embed.js +++ b/readthedocs/core/static/core/js/readthedocs-doc-embed.js @@ -1 +1 @@ -!function o(s,a,l){function d(t,e){if(!a[t]){if(!s[t]){var i="function"==typeof require&&require;if(!e&&i)return i(t,!0);if(c)return c(t,!0);var n=new Error("Cannot find module '"+t+"'");throw n.code="MODULE_NOT_FOUND",n}var r=a[t]={exports:{}};s[t][0].call(r.exports,function(e){return d(s[t][1][e]||e)},r,r.exports,o,s,a,l)}return a[t].exports}for(var c="function"==typeof require&&require,e=0;e"),i("table.docutils.footnote").wrap("
"),i("table.docutils.citation").wrap("
"),i(".wy-menu-vertical ul").not(".simple").siblings("a").each(function(){var t=i(this);expand=i(''),expand.on("click",function(e){return n.toggleCurrent(t),e.stopPropagation(),!1}),t.prepend(expand)})},reset:function(){var e=encodeURI(window.location.hash)||"#";try{var t=$(".wy-menu-vertical"),i=t.find('[href="'+e+'"]');if(0===i.length){var n=$('.document [id="'+e.substring(1)+'"]').closest("div.section");0===(i=t.find('[href="#'+n.attr("id")+'"]')).length&&(i=t.find('[href="#"]'))}0this.docHeight||(this.navBar.scrollTop(i),this.winPosition=e)},onResize:function(){this.winResize=!1,this.winHeight=this.win.height(),this.docHeight=$(document).height()},hashChange:function(){this.linkScroll=!0,this.win.one("hashchange",function(){this.linkScroll=!1})},toggleCurrent:function(e){var t=e.closest("li");t.siblings("li.current").removeClass("current"),t.siblings().find("li.current").removeClass("current"),t.find("> ul li.current").removeClass("current"),t.toggleClass("current")}},"undefined"!=typeof window&&(window.SphinxRtdTheme={Navigation:t.exports.ThemeNav,StickyNav:t.exports.ThemeNav}),function(){for(var o=0,e=["ms","moz","webkit","o"],t=0;t/g,u=/"/g,h=/"/g,p=/&#([a-zA-Z0-9]*);?/gim,f=/:?/gim,g=/&newline;?/gim,m=/((j\s*a\s*v\s*a|v\s*b|l\s*i\s*v\s*e)\s*s\s*c\s*r\s*i\s*p\s*t\s*|m\s*o\s*c\s*h\s*a)\:/gi,v=/e\s*x\s*p\s*r\s*e\s*s\s*s\s*i\s*o\s*n\s*\(.*/gi,w=/u\s*r\s*l\s*\(.*/gi;function b(e){return e.replace(u,""")}function _(e){return e.replace(h,'"')}function y(e){return e.replace(p,function(e,t){return"x"===t[0]||"X"===t[0]?String.fromCharCode(parseInt(t.substr(1),16)):String.fromCharCode(parseInt(t,10))})}function x(e){return e.replace(f,":").replace(g," ")}function k(e){for(var t="",i=0,n=e.length;i/g;i.whiteList={a:["target","href","title"],abbr:["title"],address:[],area:["shape","coords","href","alt"],article:[],aside:[],audio:["autoplay","controls","loop","preload","src"],b:[],bdi:["dir"],bdo:["dir"],big:[],blockquote:["cite"],br:[],caption:[],center:[],cite:[],code:[],col:["align","valign","span","width"],colgroup:["align","valign","span","width"],dd:[],del:["datetime"],details:["open"],div:[],dl:[],dt:[],em:[],font:["color","size","face"],footer:[],h1:[],h2:[],h3:[],h4:[],h5:[],h6:[],header:[],hr:[],i:[],img:["src","alt","title","width","height"],ins:["datetime"],li:[],mark:[],nav:[],ol:[],p:[],pre:[],s:[],section:[],small:[],span:[],sub:[],sup:[],strong:[],table:["width","border","align","valign"],tbody:["align","valign"],td:["width","rowspan","colspan","align","valign"],tfoot:["align","valign"],th:["width","rowspan","colspan","align","valign"],thead:["align","valign"],tr:["rowspan","align","valign"],tt:[],u:[],ul:[],video:["autoplay","controls","loop","preload","src","height","width"]},i.getDefaultWhiteList=o,i.onTag=function(e,t,i){},i.onIgnoreTag=function(e,t,i){},i.onTagAttr=function(e,t,i){},i.onIgnoreTagAttr=function(e,t,i){},i.safeAttrValue=function(e,t,i,n){if(i=T(i),"href"===t||"src"===t){if("#"===(i=c.trim(i)))return"#";if("http://"!==i.substr(0,7)&&"https://"!==i.substr(0,8)&&"mailto:"!==i.substr(0,7)&&"tel:"!==i.substr(0,4)&&"#"!==i[0]&&"/"!==i[0])return""}else if("background"===t){if(m.lastIndex=0,m.test(i))return""}else if("style"===t){if(v.lastIndex=0,v.test(i))return"";if(w.lastIndex=0,w.test(i)&&(m.lastIndex=0,m.test(i)))return"";!1!==n&&(i=(n=n||s).process(i))}return i=E(i)},i.escapeHtml=a,i.escapeQuote=b,i.unescapeQuote=_,i.escapeHtmlEntities=y,i.escapeDangerHtml5Entities=x,i.clearNonPrintableCharacter=k,i.friendlyAttrValue=T,i.escapeAttrValue=E,i.onIgnoreTagStripAll=function(){return""},i.StripTagBody=function(o,s){"function"!=typeof s&&(s=function(){});var a=!Array.isArray(o),l=[],d=!1;return{onIgnoreTag:function(e,t,i){if(function(e){return a||-1!==c.indexOf(o,e)}(e)){if(i.isClosing){var n="[/removed]",r=i.position+n.length;return l.push([!1!==d?d:i.position,r]),d=!1,n}return d=d||i.position,"[removed]"}return s(e,t,i)},remove:function(t){var i="",n=0;return c.forEach(l,function(e){i+=t.slice(n,e[0]),n=e[1]}),i+=t.slice(n)}}},i.stripCommentTag=function(e){return e.replace(O,"")},i.stripBlankChar=function(e){var t=e.split("");return(t=t.filter(function(e){var t=e.charCodeAt(0);return 127!==t&&(!(t<=31)||(10===t||13===t))})).join("")},i.cssFilter=s,i.getDefaultCSSWhiteList=r},{"./util":5,cssfilter:10}],3:[function(e,t,i){var n=e("./default"),r=e("./parser"),o=e("./xss");for(var s in(i=t.exports=function(e,t){return new o(t).process(e)}).FilterXSS=o,n)i[s]=n[s];for(var s in r)i[s]=r[s];"undefined"!=typeof window&&(window.filterXSS=t.exports)},{"./default":2,"./parser":4,"./xss":6}],4:[function(e,t,i){var c=e("./util");function h(e){var t=c.spaceIndex(e);if(-1===t)var i=e.slice(1,-1);else i=e.slice(1,t+1);return"/"===(i=c.trim(i).toLowerCase()).slice(0,1)&&(i=i.slice(1)),"/"===i.slice(-1)&&(i=i.slice(0,-1)),i}var u=/[^a-zA-Z0-9_:\.\-]/gim;function p(e,t){for(;t"===u){n+=i(e.slice(r,o)),c=h(d=e.slice(o,a+1)),n+=t(o,n.length,c,d,"";var a=function(e){var t=b.spaceIndex(e);if(-1===t)return{html:"",closing:"/"===e[e.length-2]};var i="/"===(e=b.trim(e.slice(t+1,-1)))[e.length-1];return i&&(e=b.trim(e.slice(0,-1))),{html:e,closing:i}}(i),l=c[r],d=w(a.html,function(e,t){var i,n=-1!==b.indexOf(l,e);return _(i=p(r,e,t,n))?n?(t=g(r,e,t,v))?e+'="'+t+'"':e:_(i=f(r,e,t,n))?void 0:i:i});i="<"+r;return d&&(i+=" "+d),a.closing&&(i+=" /"),i+=">"}return _(o=h(r,i,s))?m(i):o},m);return i&&(n=i.remove(n)),n},t.exports=a},{"./default":2,"./parser":4,"./util":5,cssfilter:10}],7:[function(e,t,i){var n,r;n=this,r=function(){var T=!0;function s(i){function e(e){var t=i.match(e);return t&&1t[1][i])return 1;if(t[0][i]!==t[1][i])return-1;if(0===i)return 0}}function o(e,t,i){var n=a;"string"==typeof t&&(i=t,t=void 0),void 0===t&&(t=!1),i&&(n=s(i));var r=""+n.version;for(var o in e)if(e.hasOwnProperty(o)&&n[o]){if("string"!=typeof e[o])throw new Error("Browser version in the minVersion map should be a string: "+o+": "+String(e));return E([r,e[o]])<0}return t}return a.test=function(e){for(var t=0;t'),a=n.title;r&&r.title&&(a=R(r.title[0]));var l=DOCUMENTATION_OPTIONS.FILE_SUFFIX;"BUILDER"in DOCUMENTATION_OPTIONS&&"readthedocsdirhtml"===DOCUMENTATION_OPTIONS.BUILDER&&(l="");var d=n.link+l+"?highlight="+$.urlencode(A),c=$("",{href:d});if(c.html(a),c.find("span").addClass("highlighted"),s.append(c),n.project!==S){var u=" (from project "+n.project+")",h=$("",{text:u});s.append(h)}for(var p=0;p'),g="",m="",v="",w="",b="",y="",x="",k="",T="",E="";if("sections"===o[p].type){if(m=(g=o[p])._source.title,v=d+"#"+g._source.id,w=[g._source.content.substr(0,I)+" ..."],g.highlight&&(g.highlight["sections.title"]&&(m=R(g.highlight["sections.title"][0])),g.highlight["sections.content"])){b=g.highlight["sections.content"],w=[];for(var O=0;O<%= section_subtitle %><% for (var i = 0; i < section_content.length; ++i) { %>
<%= section_content[i] %>
<% } %>',{section_subtitle_link:v,section_subtitle:m,section_content:w})}"domains"===o[p].type&&(x=(y=o[p])._source.role_name,k=d+"#"+y._source.anchor,T=y._source.name,(E="")!==y._source.docstrings&&(E=y._source.docstrings.substr(0,I)+" ..."),y.highlight&&(y.highlight["domains.docstrings"]&&(E="... "+R(y.highlight["domains.docstrings"][0])+" ..."),y.highlight["domains.name"]&&(T=R(y.highlight["domains.name"][0]))),M(f,'
<%= domain_content %>
',{domain_subtitle_link:k,domain_subtitle:"["+x+"]: "+T,domain_content:E})),f.find("span").addClass("highlighted"),s.append(f),p!==o.length-1&&s.append($("
"))}Search.output.append(s),s.slideDown(5)}t.length?Search.status.text(_("Search finished, found %s page(s) matching the search query.").replace("%s",t.length)):(Search.query_fallback(A),console.log("Read the Docs search failed. Falling back to Sphinx search."))}).fail(function(e){Search.query_fallback(A)}).always(function(){$("#search-progress").empty(),Search.stopPulse(),Search.title.text(_("Search Results")),Search.status.fadeIn(500)}),$.ajax({url:e.href,crossDomain:!0,xhrFields:{withCredentials:!0},complete:function(e,t){return"success"!==t||void 0===e.responseJSON||0===e.responseJSON.count?n.reject():n.resolve(e.responseJSON)}}).fail(function(e,t,i){return n.reject()})}}$(document).ready(function(){"undefined"!=typeof Search&&Search.init()})}(n.get())}}},{"./../../../../../../bower_components/xss/lib/index":3,"./rtd-data":16}],18:[function(r,e,t){var o=r("./rtd-data");e.exports={init:function(){var e=o.get();if($(document).on("click","[data-toggle='rst-current-version']",function(){var e=$("[data-toggle='rst-versions']").hasClass("shift-up")?"was_open":"was_closed";"undefined"!=typeof ga?ga("rtfd.send","event","Flyout","Click",e):"undefined"!=typeof _gaq&&_gaq.push(["rtfd._setAccount","UA-17997319-1"],["rtfd._trackEvent","Flyout","Click",e])}),void 0===window.SphinxRtdTheme){var t=r("./../../../../../../bower_components/sphinx-rtd-theme/js/theme.js").ThemeNav;if($(document).ready(function(){setTimeout(function(){t.navBar||t.enable()},1e3)}),e.is_rtd_like_theme())if(!$("div.wy-side-scroll:first").length){console.log("Applying theme sidebar fix...");var i=$("nav.wy-nav-side:first"),n=$("
").addClass("wy-side-scroll");i.children().detach().appendTo(n),n.prependTo(i),t.navBar=n}}}}},{"./../../../../../../bower_components/sphinx-rtd-theme/js/theme.js":1,"./rtd-data":16}],19:[function(e,t,i){var l,d=e("./constants"),c=e("./rtd-data"),n=e("bowser"),u="#ethical-ad-placement";function h(){var e,t,i="rtd-"+(Math.random()+1).toString(36).substring(4),n=d.PROMO_TYPES.LEFTNAV,r=d.DEFAULT_PROMO_PRIORITY,o=null;return l.is_mkdocs_builder()&&l.is_rtd_like_theme()?(o="nav.wy-nav-side",e="ethical-rtd ethical-dark-theme"):l.is_rtd_like_theme()?(o="nav.wy-nav-side > div.wy-side-scroll",e="ethical-rtd ethical-dark-theme"):l.is_alabaster_like_theme()&&(o="div.sphinxsidebar > div.sphinxsidebarwrapper",e="ethical-alabaster"),o?($("
").attr("id",i).addClass(e).appendTo(o),(!(t=$("#"+i).offset())||t.top>$(window).height())&&(r=d.LOW_PROMO_PRIORITY),{div_id:i,display_type:n,priority:r}):null}function p(){var e,t,i="rtd-"+(Math.random()+1).toString(36).substring(4),n=d.PROMO_TYPES.FOOTER,r=d.DEFAULT_PROMO_PRIORITY,o=null;return l.is_rtd_like_theme()?(o=$("
").insertAfter("footer hr"),e="ethical-rtd"):l.is_alabaster_like_theme()&&(o="div.bodywrapper .body",e="ethical-alabaster"),o?($("
").attr("id",i).addClass(e).appendTo(o),(!(t=$("#"+i).offset())||t.top<$(window).height())&&(r=d.LOW_PROMO_PRIORITY),{div_id:i,display_type:n,priority:r}):null}function f(){var e="rtd-"+(Math.random()+1).toString(36).substring(4),t=d.PROMO_TYPES.FIXED_FOOTER,i=d.DEFAULT_PROMO_PRIORITY;return n&&n.mobile&&(i=d.MAXIMUM_PROMO_PRIORITY),$("
").attr("id",e).appendTo("body"),{div_id:e,display_type:t,priority:i}}function g(e){this.id=e.id,this.div_id=e.div_id||"",this.html=e.html||"",this.display_type=e.display_type||"",this.view_tracking_url=e.view_url,this.click_handler=function(){"undefined"!=typeof ga?ga("rtfd.send","event","Promo","Click",e.id):"undefined"!=typeof _gaq&&_gaq.push(["rtfd._setAccount","UA-17997319-1"],["rtfd._trackEvent","Promo","Click",e.id])}}g.prototype.display=function(){var e="#"+this.div_id,t=this.view_tracking_url;$(e).html(this.html),$(e).find('a[href*="/sustainability/click/"]').on("click",this.click_handler);function i(){$.inViewport($(e),-3)&&($("").attr("src",t).css("display","none").appendTo(e),$(window).off(".rtdinview"),$(".wy-side-scroll").off(".rtdinview"))}$(window).on("DOMContentLoaded.rtdinview load.rtdinview scroll.rtdinview resize.rtdinview",i),$(".wy-side-scroll").on("scroll.rtdinview",i),$(".ethical-close").on("click",function(){return $(e).hide(),!1}),this.post_promo_display()},g.prototype.disable=function(){$("#"+this.div_id).hide()},g.prototype.post_promo_display=function(){this.display_type===d.PROMO_TYPES.FOOTER&&($("
").insertAfter("#"+this.div_id),$("
").insertBefore("#"+this.div_id+".ethical-alabaster .ethical-footer"))},t.exports={Promo:g,init:function(){var e,t,i={format:"jsonp"},n=[],r=[],o=[],s=[p,h,f];if(l=c.get(),t=function(){var e,t="rtd-"+(Math.random()+1).toString(36).substring(4),i=d.PROMO_TYPES.LEFTNAV;return e=l.is_rtd_like_theme()?"ethical-rtd ethical-dark-theme":"ethical-alabaster",0<$(u).length?($("
").attr("id",t).addClass(e).appendTo(u),{div_id:t,display_type:i}):null}())n.push(t.div_id),r.push(t.display_type),o.push(t.priority||d.DEFAULT_PROMO_PRIORITY),!0;else{if(!l.show_promo())return;for(var a=0;a").attr("id","rtd-detection").attr("class","ethical-rtd").html(" ").appendTo("body"),0===$("#rtd-detection").height()&&(e=!0),$("#rtd-detection").remove(),e}()&&(console.log("---------------------------------------------------------------------------------------"),console.log("Read the Docs hosts documentation for tens of thousands of open source projects."),console.log("We fund our development (we are open source) and operations through advertising."),console.log("We promise to:"),console.log(" - never let advertisers run 3rd party JavaScript"),console.log(" - never sell user data to advertisers or other 3rd parties"),console.log(" - only show advertisements of interest to developers"),console.log("Read more about our approach to advertising here: https://docs.readthedocs.io/en/latest/advertising/ethical-advertising.html"),console.log("%cPlease allow our Ethical Ads or go ad-free:","font-size: 2em"),console.log("https://docs.readthedocs.io/en/latest/advertising/ad-blocking.html"),console.log("--------------------------------------------------------------------------------------"),function(){var e=h(),t=null;e&&e.div_id&&(t=$("#"+e.div_id).attr("class","keep-us-sustainable"),$("

").text("Support Read the Docs!").appendTo(t),$("

").html('Please help keep us sustainable by allowing our Ethical Ads in your ad blocker or go ad-free by subscribing.').appendTo(t),$("

").text("Thank you! ❤️").appendTo(t))}())}})}}},{"./constants":14,"./rtd-data":16,bowser:7}],20:[function(e,t,i){var o=e("./rtd-data");t.exports={init:function(e){var t=o.get();if(!e.is_highest){var i=window.location.pathname.replace(t.version,e.slug),n=$('

Note

You are not reading the most recent version of this documentation. is the latest version available.

');n.find("a").attr("href",i).text(e.slug);var r=$("div.body");r.length||(r=$("div.document")),r.prepend(n)}}}},{"./rtd-data":16}],21:[function(e,t,i){var n=e("./doc-embed/sponsorship"),r=e("./doc-embed/footer.js"),o=(e("./doc-embed/rtd-data"),e("./doc-embed/sphinx")),s=e("./doc-embed/search");$.extend(e("verge")),$(document).ready(function(){r.init(),o.init(),s.init(),n.init()})},{"./doc-embed/footer.js":15,"./doc-embed/rtd-data":16,"./doc-embed/search":17,"./doc-embed/sphinx":18,"./doc-embed/sponsorship":19,verge:13}]},{},[21]); \ No newline at end of file +!function o(s,a,l){function d(t,e){if(!a[t]){if(!s[t]){var i="function"==typeof require&&require;if(!e&&i)return i(t,!0);if(c)return c(t,!0);var n=new Error("Cannot find module '"+t+"'");throw n.code="MODULE_NOT_FOUND",n}var r=a[t]={exports:{}};s[t][0].call(r.exports,function(e){return d(s[t][1][e]||e)},r,r.exports,o,s,a,l)}return a[t].exports}for(var c="function"==typeof require&&require,e=0;e
"),i("table.docutils.footnote").wrap("
"),i("table.docutils.citation").wrap("
"),i(".wy-menu-vertical ul").not(".simple").siblings("a").each(function(){var t=i(this);expand=i(''),expand.on("click",function(e){return n.toggleCurrent(t),e.stopPropagation(),!1}),t.prepend(expand)})},reset:function(){var e=encodeURI(window.location.hash)||"#";try{var t=$(".wy-menu-vertical"),i=t.find('[href="'+e+'"]');if(0===i.length){var n=$('.document [id="'+e.substring(1)+'"]').closest("div.section");0===(i=t.find('[href="#'+n.attr("id")+'"]')).length&&(i=t.find('[href="#"]'))}0this.docHeight||(this.navBar.scrollTop(i),this.winPosition=e)},onResize:function(){this.winResize=!1,this.winHeight=this.win.height(),this.docHeight=$(document).height()},hashChange:function(){this.linkScroll=!0,this.win.one("hashchange",function(){this.linkScroll=!1})},toggleCurrent:function(e){var t=e.closest("li");t.siblings("li.current").removeClass("current"),t.siblings().find("li.current").removeClass("current"),t.find("> ul li.current").removeClass("current"),t.toggleClass("current")}},"undefined"!=typeof window&&(window.SphinxRtdTheme={Navigation:t.exports.ThemeNav,StickyNav:t.exports.ThemeNav}),function(){for(var o=0,e=["ms","moz","webkit","o"],t=0;t/g,u=/"/g,h=/"/g,p=/&#([a-zA-Z0-9]*);?/gim,f=/:?/gim,g=/&newline;?/gim,m=/((j\s*a\s*v\s*a|v\s*b|l\s*i\s*v\s*e)\s*s\s*c\s*r\s*i\s*p\s*t\s*|m\s*o\s*c\s*h\s*a)\:/gi,v=/e\s*x\s*p\s*r\s*e\s*s\s*s\s*i\s*o\s*n\s*\(.*/gi,w=/u\s*r\s*l\s*\(.*/gi;function b(e){return e.replace(u,""")}function _(e){return e.replace(h,'"')}function y(e){return e.replace(p,function(e,t){return"x"===t[0]||"X"===t[0]?String.fromCharCode(parseInt(t.substr(1),16)):String.fromCharCode(parseInt(t,10))})}function x(e){return e.replace(f,":").replace(g," ")}function k(e){for(var t="",i=0,n=e.length;i/g;i.whiteList={a:["target","href","title"],abbr:["title"],address:[],area:["shape","coords","href","alt"],article:[],aside:[],audio:["autoplay","controls","loop","preload","src"],b:[],bdi:["dir"],bdo:["dir"],big:[],blockquote:["cite"],br:[],caption:[],center:[],cite:[],code:[],col:["align","valign","span","width"],colgroup:["align","valign","span","width"],dd:[],del:["datetime"],details:["open"],div:[],dl:[],dt:[],em:[],font:["color","size","face"],footer:[],h1:[],h2:[],h3:[],h4:[],h5:[],h6:[],header:[],hr:[],i:[],img:["src","alt","title","width","height"],ins:["datetime"],li:[],mark:[],nav:[],ol:[],p:[],pre:[],s:[],section:[],small:[],span:[],sub:[],sup:[],strong:[],table:["width","border","align","valign"],tbody:["align","valign"],td:["width","rowspan","colspan","align","valign"],tfoot:["align","valign"],th:["width","rowspan","colspan","align","valign"],thead:["align","valign"],tr:["rowspan","align","valign"],tt:[],u:[],ul:[],video:["autoplay","controls","loop","preload","src","height","width"]},i.getDefaultWhiteList=o,i.onTag=function(e,t,i){},i.onIgnoreTag=function(e,t,i){},i.onTagAttr=function(e,t,i){},i.onIgnoreTagAttr=function(e,t,i){},i.safeAttrValue=function(e,t,i,n){if(i=T(i),"href"===t||"src"===t){if("#"===(i=c.trim(i)))return"#";if("http://"!==i.substr(0,7)&&"https://"!==i.substr(0,8)&&"mailto:"!==i.substr(0,7)&&"tel:"!==i.substr(0,4)&&"#"!==i[0]&&"/"!==i[0])return""}else if("background"===t){if(m.lastIndex=0,m.test(i))return""}else if("style"===t){if(v.lastIndex=0,v.test(i))return"";if(w.lastIndex=0,w.test(i)&&(m.lastIndex=0,m.test(i)))return"";!1!==n&&(i=(n=n||s).process(i))}return i=E(i)},i.escapeHtml=a,i.escapeQuote=b,i.unescapeQuote=_,i.escapeHtmlEntities=y,i.escapeDangerHtml5Entities=x,i.clearNonPrintableCharacter=k,i.friendlyAttrValue=T,i.escapeAttrValue=E,i.onIgnoreTagStripAll=function(){return""},i.StripTagBody=function(o,s){"function"!=typeof s&&(s=function(){});var a=!Array.isArray(o),l=[],d=!1;return{onIgnoreTag:function(e,t,i){if(function(e){return a||-1!==c.indexOf(o,e)}(e)){if(i.isClosing){var n="[/removed]",r=i.position+n.length;return l.push([!1!==d?d:i.position,r]),d=!1,n}return d=d||i.position,"[removed]"}return s(e,t,i)},remove:function(t){var i="",n=0;return c.forEach(l,function(e){i+=t.slice(n,e[0]),n=e[1]}),i+=t.slice(n)}}},i.stripCommentTag=function(e){return e.replace(A,"")},i.stripBlankChar=function(e){var t=e.split("");return(t=t.filter(function(e){var t=e.charCodeAt(0);return 127!==t&&(!(t<=31)||(10===t||13===t))})).join("")},i.cssFilter=s,i.getDefaultCSSWhiteList=r},{"./util":5,cssfilter:10}],3:[function(e,t,i){var n=e("./default"),r=e("./parser"),o=e("./xss");for(var s in(i=t.exports=function(e,t){return new o(t).process(e)}).FilterXSS=o,n)i[s]=n[s];for(var s in r)i[s]=r[s];"undefined"!=typeof window&&(window.filterXSS=t.exports)},{"./default":2,"./parser":4,"./xss":6}],4:[function(e,t,i){var c=e("./util");function h(e){var t=c.spaceIndex(e);if(-1===t)var i=e.slice(1,-1);else i=e.slice(1,t+1);return"/"===(i=c.trim(i).toLowerCase()).slice(0,1)&&(i=i.slice(1)),"/"===i.slice(-1)&&(i=i.slice(0,-1)),i}var u=/[^a-zA-Z0-9_:\.\-]/gim;function p(e,t){for(;t"===u){n+=i(e.slice(r,o)),c=h(d=e.slice(o,a+1)),n+=t(o,n.length,c,d,"";var a=function(e){var t=b.spaceIndex(e);if(-1===t)return{html:"",closing:"/"===e[e.length-2]};var i="/"===(e=b.trim(e.slice(t+1,-1)))[e.length-1];return i&&(e=b.trim(e.slice(0,-1))),{html:e,closing:i}}(i),l=c[r],d=w(a.html,function(e,t){var i,n=-1!==b.indexOf(l,e);return _(i=p(r,e,t,n))?n?(t=g(r,e,t,v))?e+'="'+t+'"':e:_(i=f(r,e,t,n))?void 0:i:i});i="<"+r;return d&&(i+=" "+d),a.closing&&(i+=" /"),i+=">"}return _(o=h(r,i,s))?m(i):o},m);return i&&(n=i.remove(n)),n},t.exports=a},{"./default":2,"./parser":4,"./util":5,cssfilter:10}],7:[function(e,t,i){var n,r;n=this,r=function(){var T=!0;function s(i){function e(e){var t=i.match(e);return t&&1t[1][i])return 1;if(t[0][i]!==t[1][i])return-1;if(0===i)return 0}}function o(e,t,i){var n=a;"string"==typeof t&&(i=t,t=void 0),void 0===t&&(t=!1),i&&(n=s(i));var r=""+n.version;for(var o in e)if(e.hasOwnProperty(o)&&n[o]){if("string"!=typeof e[o])throw new Error("Browser version in the minVersion map should be a string: "+o+": "+String(e));return E([r,e[o]])<0}return t}return a.test=function(e){for(var t=0;t'),a=n.title;r&&r.title&&(a=O(r.title[0]));var l=n.link+"?highlight="+$.urlencode(A),d=$("",{href:l});if(d.html(a),d.find("span").addClass("highlighted"),s.append(d),n.project!==S){var c=" (from project "+n.project+")",u=$("",{text:c});s.append(u)}for(var h=0;h'),f="",g="",m="",v="",w="",b="",y="",x="",k="",T="";if("sections"===o[h].type){if(g=(f=o[h])._source.title,m=l+"#"+f._source.id,v=[f._source.content.substr(0,C)+" ..."],f.highlight&&(f.highlight["sections.title"]&&(g=O(f.highlight["sections.title"][0])),f.highlight["sections.content"])){w=f.highlight["sections.content"],v=[];for(var E=0;E<%= section_subtitle %>
<% for (var i = 0; i < section_content.length; ++i) { %>
<%= section_content[i] %>
<% } %>',{section_subtitle_link:m,section_subtitle:g,section_content:v})}"domains"===o[h].type&&(y=(b=o[h])._source.role_name,x=l+"#"+b._source.anchor,k=b._source.name,(T="")!==b._source.docstrings&&(T=b._source.docstrings.substr(0,C)+" ..."),b.highlight&&(b.highlight["domains.docstrings"]&&(T="... "+O(b.highlight["domains.docstrings"][0])+" ..."),b.highlight["domains.name"]&&(k=O(b.highlight["domains.name"][0]))),M(p,'
<%= domain_content %>
',{domain_subtitle_link:x,domain_subtitle:"["+y+"]: "+k,domain_content:T})),p.find("span").addClass("highlighted"),s.append(p),h!==o.length-1&&s.append($("
"))}Search.output.append(s),s.slideDown(5)}t.length?Search.status.text(_("Search finished, found %s page(s) matching the search query.").replace("%s",t.length)):(Search.query_fallback(A),console.log("Read the Docs search failed. Falling back to Sphinx search."))}).fail(function(e){Search.query_fallback(A)}).always(function(){$("#search-progress").empty(),Search.stopPulse(),Search.title.text(_("Search Results")),Search.status.fadeIn(500)}),$.ajax({url:e.href,crossDomain:!0,xhrFields:{withCredentials:!0},complete:function(e,t){return"success"!==t||void 0===e.responseJSON||0===e.responseJSON.count?n.reject():n.resolve(e.responseJSON)}}).fail(function(e,t,i){return n.reject()})}}$(document).ready(function(){"undefined"!=typeof Search&&Search.init()})}(n.get())}}},{"./../../../../../../bower_components/xss/lib/index":3,"./rtd-data":16}],18:[function(r,e,t){var o=r("./rtd-data");e.exports={init:function(){var e=o.get();if($(document).on("click","[data-toggle='rst-current-version']",function(){var e=$("[data-toggle='rst-versions']").hasClass("shift-up")?"was_open":"was_closed";"undefined"!=typeof ga?ga("rtfd.send","event","Flyout","Click",e):"undefined"!=typeof _gaq&&_gaq.push(["rtfd._setAccount","UA-17997319-1"],["rtfd._trackEvent","Flyout","Click",e])}),void 0===window.SphinxRtdTheme){var t=r("./../../../../../../bower_components/sphinx-rtd-theme/js/theme.js").ThemeNav;if($(document).ready(function(){setTimeout(function(){t.navBar||t.enable()},1e3)}),e.is_rtd_like_theme())if(!$("div.wy-side-scroll:first").length){console.log("Applying theme sidebar fix...");var i=$("nav.wy-nav-side:first"),n=$("
").addClass("wy-side-scroll");i.children().detach().appendTo(n),n.prependTo(i),t.navBar=n}}}}},{"./../../../../../../bower_components/sphinx-rtd-theme/js/theme.js":1,"./rtd-data":16}],19:[function(e,t,i){var l,d=e("./constants"),c=e("./rtd-data"),n=e("bowser"),u="#ethical-ad-placement";function h(){var e,t,i="rtd-"+(Math.random()+1).toString(36).substring(4),n=d.PROMO_TYPES.LEFTNAV,r=d.DEFAULT_PROMO_PRIORITY,o=null;return l.is_mkdocs_builder()&&l.is_rtd_like_theme()?(o="nav.wy-nav-side",e="ethical-rtd ethical-dark-theme"):l.is_rtd_like_theme()?(o="nav.wy-nav-side > div.wy-side-scroll",e="ethical-rtd ethical-dark-theme"):l.is_alabaster_like_theme()&&(o="div.sphinxsidebar > div.sphinxsidebarwrapper",e="ethical-alabaster"),o?($("
").attr("id",i).addClass(e).appendTo(o),(!(t=$("#"+i).offset())||t.top>$(window).height())&&(r=d.LOW_PROMO_PRIORITY),{div_id:i,display_type:n,priority:r}):null}function p(){var e,t,i="rtd-"+(Math.random()+1).toString(36).substring(4),n=d.PROMO_TYPES.FOOTER,r=d.DEFAULT_PROMO_PRIORITY,o=null;return l.is_rtd_like_theme()?(o=$("
").insertAfter("footer hr"),e="ethical-rtd"):l.is_alabaster_like_theme()&&(o="div.bodywrapper .body",e="ethical-alabaster"),o?($("
").attr("id",i).addClass(e).appendTo(o),(!(t=$("#"+i).offset())||t.top<$(window).height())&&(r=d.LOW_PROMO_PRIORITY),{div_id:i,display_type:n,priority:r}):null}function f(){var e="rtd-"+(Math.random()+1).toString(36).substring(4),t=d.PROMO_TYPES.FIXED_FOOTER,i=d.DEFAULT_PROMO_PRIORITY;return n&&n.mobile&&(i=d.MAXIMUM_PROMO_PRIORITY),$("
").attr("id",e).appendTo("body"),{div_id:e,display_type:t,priority:i}}function g(e){this.id=e.id,this.div_id=e.div_id||"",this.html=e.html||"",this.display_type=e.display_type||"",this.view_tracking_url=e.view_url,this.click_handler=function(){"undefined"!=typeof ga?ga("rtfd.send","event","Promo","Click",e.id):"undefined"!=typeof _gaq&&_gaq.push(["rtfd._setAccount","UA-17997319-1"],["rtfd._trackEvent","Promo","Click",e.id])}}g.prototype.display=function(){var e="#"+this.div_id,t=this.view_tracking_url;$(e).html(this.html),$(e).find('a[href*="/sustainability/click/"]').on("click",this.click_handler);function i(){$.inViewport($(e),-3)&&($("").attr("src",t).css("display","none").appendTo(e),$(window).off(".rtdinview"),$(".wy-side-scroll").off(".rtdinview"))}$(window).on("DOMContentLoaded.rtdinview load.rtdinview scroll.rtdinview resize.rtdinview",i),$(".wy-side-scroll").on("scroll.rtdinview",i),$(".ethical-close").on("click",function(){return $(e).hide(),!1}),this.post_promo_display()},g.prototype.disable=function(){$("#"+this.div_id).hide()},g.prototype.post_promo_display=function(){this.display_type===d.PROMO_TYPES.FOOTER&&($("
").insertAfter("#"+this.div_id),$("
").insertBefore("#"+this.div_id+".ethical-alabaster .ethical-footer"))},t.exports={Promo:g,init:function(){var e,t,i={format:"jsonp"},n=[],r=[],o=[],s=[p,h,f];if(l=c.get(),t=function(){var e,t="rtd-"+(Math.random()+1).toString(36).substring(4),i=d.PROMO_TYPES.LEFTNAV;return e=l.is_rtd_like_theme()?"ethical-rtd ethical-dark-theme":"ethical-alabaster",0<$(u).length?($("
").attr("id",t).addClass(e).appendTo(u),{div_id:t,display_type:i}):null}())n.push(t.div_id),r.push(t.display_type),o.push(t.priority||d.DEFAULT_PROMO_PRIORITY),!0;else{if(!l.show_promo())return;for(var a=0;a").attr("id","rtd-detection").attr("class","ethical-rtd").html(" ").appendTo("body"),0===$("#rtd-detection").height()&&(e=!0),$("#rtd-detection").remove(),e}()&&(console.log("---------------------------------------------------------------------------------------"),console.log("Read the Docs hosts documentation for tens of thousands of open source projects."),console.log("We fund our development (we are open source) and operations through advertising."),console.log("We promise to:"),console.log(" - never let advertisers run 3rd party JavaScript"),console.log(" - never sell user data to advertisers or other 3rd parties"),console.log(" - only show advertisements of interest to developers"),console.log("Read more about our approach to advertising here: https://docs.readthedocs.io/en/latest/advertising/ethical-advertising.html"),console.log("%cPlease allow our Ethical Ads or go ad-free:","font-size: 2em"),console.log("https://docs.readthedocs.io/en/latest/advertising/ad-blocking.html"),console.log("--------------------------------------------------------------------------------------"),function(){var e=h(),t=null;e&&e.div_id&&(t=$("#"+e.div_id).attr("class","keep-us-sustainable"),$("

").text("Support Read the Docs!").appendTo(t),$("

").html('Please help keep us sustainable by allowing our Ethical Ads in your ad blocker or go ad-free by subscribing.').appendTo(t),$("

").text("Thank you! ❤️").appendTo(t))}())}})}}},{"./constants":14,"./rtd-data":16,bowser:7}],20:[function(e,t,i){var o=e("./rtd-data");t.exports={init:function(e){var t=o.get();if(!e.is_highest){var i=window.location.pathname.replace(t.version,e.slug),n=$('

Note

You are not reading the most recent version of this documentation. is the latest version available.

');n.find("a").attr("href",i).text(e.slug);var r=$("div.body");r.length||(r=$("div.document")),r.prepend(n)}}}},{"./rtd-data":16}],21:[function(e,t,i){var n=e("./doc-embed/sponsorship"),r=e("./doc-embed/footer.js"),o=(e("./doc-embed/rtd-data"),e("./doc-embed/sphinx")),s=e("./doc-embed/search");$.extend(e("verge")),$(document).ready(function(){r.init(),o.init(),s.init(),n.init()})},{"./doc-embed/footer.js":15,"./doc-embed/rtd-data":16,"./doc-embed/search":17,"./doc-embed/sphinx":18,"./doc-embed/sponsorship":19,verge:13}]},{},[21]); \ No newline at end of file diff --git a/readthedocs/core/unresolver.py b/readthedocs/core/unresolver.py index e2b35502001..bd5a6da9f14 100644 --- a/readthedocs/core/unresolver.py +++ b/readthedocs/core/unresolver.py @@ -55,9 +55,15 @@ def unresolve(self, url): filename=kwargs.get('filename', ''), ) + # Handle our backend storage not supporting directory indexes, + # so we need to append index.html when appropriate. + if not filename or filename.endswith('/'): + # We need to add the index.html to find this actual file + filename += 'index.html' + log.info( - 'Unresolver parsed:' - 'url=%s, project=%s lang_slug=%s version_slug=%s filename=%s', + 'Unresolver parsed: ' + 'url=%s project=%s lang_slug=%s version_slug=%s filename=%s', url, final_project.slug, lang_slug, version_slug, filename ) return UnresolvedObject(final_project, lang_slug, version_slug, filename, parsed.fragment) diff --git a/readthedocs/projects/models.py b/readthedocs/projects/models.py index d20cd8048f1..40af8e6fc44 100644 --- a/readthedocs/projects/models.py +++ b/readthedocs/projects/models.py @@ -881,6 +881,7 @@ def ordered_active_versions(self, **kwargs): { 'project': self, 'only_active': True, + 'only_built': True, }, ) versions = ( @@ -1242,9 +1243,6 @@ def get_processed_json_sphinx(self): Both lead to `foo/index.html` https://github.com/rtfd/readthedocs.org/issues/5368 """ - file_path = None - storage = get_storage_class(settings.RTD_BUILD_MEDIA_STORAGE)() - fjson_paths = [] basename = os.path.splitext(self.path)[0] fjson_paths.append(basename + '.fjson') @@ -1252,22 +1250,23 @@ def get_processed_json_sphinx(self): new_basename = re.sub(r'\/index$', '', basename) fjson_paths.append(new_basename + '.fjson') + storage = get_storage_class(settings.RTD_BUILD_MEDIA_STORAGE)() storage_path = self.project.get_storage_path( type_='json', version_slug=self.version.slug, include_file=False ) - try: - for fjson_path in fjson_paths: - file_path = storage.join(storage_path, fjson_path) - if storage.exists(file_path): - return process_file(file_path) - except Exception: - log.warning( - 'Unhandled exception during search processing file: %s', - file_path, - ) + for fjson_path in fjson_paths: + try: + fjson_storage_path = storage.join(storage_path, fjson_path) + if storage.exists(fjson_storage_path): + return process_file(fjson_storage_path) + except Exception: + log.warning( + 'Unhandled exception during search processing file: %s', + fjson_path, + ) return { - 'path': file_path, + 'path': self.path, 'title': '', 'sections': [], 'domain_data': {}, @@ -1469,6 +1468,7 @@ def add_features(sender, **kwargs): FORCE_SPHINX_FROM_VENV = 'force_sphinx_from_venv' LIST_PACKAGES_INSTALLED_ENV = 'list_packages_installed_env' VCS_REMOTE_LISTING = 'vcs_remote_listing' + STORE_PAGEVIEWS = 'store_pageviews' FEATURES = ( (USE_SPHINX_LATEST, _('Use latest version of Sphinx')), @@ -1558,6 +1558,10 @@ def add_features(sender, **kwargs): VCS_REMOTE_LISTING, _('Use remote listing in VCS (e.g. git ls-remote) if supported for sync versions'), ), + ( + STORE_PAGEVIEWS, + _('Store pageviews for this project'), + ), ) projects = models.ManyToManyField( diff --git a/readthedocs/projects/tasks.py b/readthedocs/projects/tasks.py index 06ccde4300f..79df64cfb6d 100644 --- a/readthedocs/projects/tasks.py +++ b/readthedocs/projects/tasks.py @@ -783,10 +783,7 @@ def run_build(self, record): epub=bool(outcomes['epub']), ) - # Finalize build and update web servers - # We upload EXTERNAL version media files to blob storage - # We should have this check here to make sure - # the files don't get re-uploaded on web. + # TODO: Remove this function and just update the DB and index search directly self.update_app_instances( html=bool(outcomes['html']), search=bool(outcomes['search']), @@ -1268,6 +1265,7 @@ def fileify(version_pk, commit, build): This is so we have an idea of what files we have in the database. """ version = Version.objects.get_object_or_log(pk=version_pk) + # Don't index external version builds for now if not version or version.type == EXTERNAL: return project = version.project diff --git a/readthedocs/projects/urls/private.py b/readthedocs/projects/urls/private.py index 5ce38078392..1db0be47af3 100644 --- a/readthedocs/projects/urls/private.py +++ b/readthedocs/projects/urls/private.py @@ -26,6 +26,7 @@ IntegrationExchangeDetail, IntegrationList, IntegrationWebhookSync, + TrafficAnalyticsView, ProjectAdvancedUpdate, ProjectAdvertisingUpdate, ProjectDashboard, @@ -139,6 +140,10 @@ SearchAnalytics.as_view(), name='projects_search_analytics', ), + url( + r'^(?P[-\w]+)/traffic-analytics/$', + TrafficAnalyticsView.as_view(), name='projects_traffic_analytics', + ), ] domain_urls = [ diff --git a/readthedocs/projects/views/private.py b/readthedocs/projects/views/private.py index 34e59e30c7f..b23a4ef3654 100644 --- a/readthedocs/projects/views/private.py +++ b/readthedocs/projects/views/private.py @@ -6,7 +6,6 @@ from allauth.socialaccount.models import SocialAccount from django.conf import settings from django.contrib import messages -from django.contrib.auth.decorators import login_required from django.db.models import Count from django.http import ( Http404, @@ -33,6 +32,7 @@ UpdateView, ) +from readthedocs.analytics.models import PageView from readthedocs.builds.forms import RegexAutomationRuleForm, VersionForm from readthedocs.builds.models import ( RegexAutomationRule, @@ -41,7 +41,6 @@ ) from readthedocs.core.mixins import ( ListViewWithForm, - LoginRequiredMixin, PrivateViewMixin, ) from readthedocs.core.utils import broadcast, trigger_build @@ -977,15 +976,6 @@ class RegexAutomationRuleUpdate(RegexAutomationRuleMixin, UpdateView): pass -@login_required -def search_analytics_view(request, project_slug): - """View for search analytics.""" - project = get_object_or_404( - Project.objects.for_admin_user(request.user), - slug=project_slug, - ) - - class SearchAnalytics(ProjectAdminMixin, PrivateViewMixin, TemplateView): template_name = 'projects/projects_search_analytics.html' @@ -1013,7 +1003,7 @@ def get_context_data(self, **kwargs): qs.values('query') .annotate(count=Count('id')) .order_by('-count', 'query') - .values_list('query', 'count') + .values_list('query', 'count', 'total_results') ) # only show top 100 queries @@ -1040,7 +1030,7 @@ def _search_analytics_csv_data(self): created__date__lte=now, ) .order_by('-created') - .values_list('created', 'query') + .values_list('created', 'query', 'total_results') ) file_name = '{project_slug}_from_{start}_to_{end}.csv'.format( @@ -1052,8 +1042,8 @@ def _search_analytics_csv_data(self): file_name = '-'.join([text for text in file_name.split() if text]) csv_data = ( - [timezone.datetime.strftime(time, '%Y-%m-%d %H:%M:%S'), query] - for time, query in data + [timezone.datetime.strftime(time, '%Y-%m-%d %H:%M:%S'), query, total_results] + for time, query, total_results in data ) pseudo_buffer = Echo() writer = csv.writer(pseudo_buffer) @@ -1063,3 +1053,32 @@ def _search_analytics_csv_data(self): ) response['Content-Disposition'] = f'attachment; filename="{file_name}"' return response + + +class TrafficAnalyticsView(ProjectAdminMixin, PrivateViewMixin, TemplateView): + + template_name = 'projects/project_traffic_analytics.html' + http_method_names = ['get'] + + def get_context_data(self, **kwargs): + context = super().get_context_data(**kwargs) + project = self.get_project() + + # Count of views for top pages over the month + top_pages = PageView.top_viewed_pages(project) + top_viewed_pages = zip( + top_pages['pages'], + top_pages['view_counts'] + ) + + # Aggregate pageviews grouped by day + page_data = PageView.page_views_by_date( + project_slug=project.slug, + ) + + context.update({ + 'top_viewed_pages': top_viewed_pages, + 'page_data': page_data, + }) + + return context diff --git a/readthedocs/proxito/middleware.py b/readthedocs/proxito/middleware.py index 5493de7f33d..d95eba84df5 100644 --- a/readthedocs/proxito/middleware.py +++ b/readthedocs/proxito/middleware.py @@ -56,7 +56,10 @@ def map_host_to_project_slug(request): # pylint: disable=too-many-return-statem project_slug = host_parts[0] request.subdomain = True log.debug('Proxito Public Domain: host=%s', host) - if Domain.objects.filter(project__slug=project_slug).filter(canonical=True).exists(): + if Domain.objects.filter(project__slug=project_slug).filter( + canonical=True, + https=True, + ).exists(): log.debug('Proxito Public Domain -> Canonical Domain Redirect: host=%s', host) request.canonicalize = 'canonical-cname' return project_slug diff --git a/readthedocs/proxito/tests/base.py b/readthedocs/proxito/tests/base.py index 4ed02d93b74..daee545b30f 100644 --- a/readthedocs/proxito/tests/base.py +++ b/readthedocs/proxito/tests/base.py @@ -21,7 +21,6 @@ def setUp(self): Project, slug='project', privacy_level=PUBLIC, - version_privacy_level=PUBLIC, users=[self.eric], main_language_project=None, ) diff --git a/readthedocs/proxito/tests/test_middleware.py b/readthedocs/proxito/tests/test_middleware.py index a7296ab7bfb..a71d404cce0 100644 --- a/readthedocs/proxito/tests/test_middleware.py +++ b/readthedocs/proxito/tests/test_middleware.py @@ -50,7 +50,7 @@ def test_proper_cname_https_upgrade(self): self.assertEqual(request.canonicalize, 'https') def test_canonical_cname_redirect(self): - """Requests to the public domain URL should redirect to the custom domain only if the domain is canonical.""" + """Requests to the public domain URL should redirect to the custom domain if the domain is canonical/https.""" cname = 'docs.random.com' domain = get(Domain, project=self.pip, domain=cname, canonical=False, https=False) @@ -59,8 +59,9 @@ def test_canonical_cname_redirect(self): self.assertIsNone(res) self.assertFalse(hasattr(request, 'canonicalize')) - # Make the domain canonical and make sure we redirect + # Make the domain canonical/https and make sure we redirect domain.canonical = True + domain.https = True domain.save() for url in (self.url, '/subdir/'): request = self.request(url, HTTP_HOST='pip.dev.readthedocs.io') diff --git a/readthedocs/proxito/urls.py b/readthedocs/proxito/urls.py index 413731268d5..e657edc0e7b 100644 --- a/readthedocs/proxito/urls.py +++ b/readthedocs/proxito/urls.py @@ -82,6 +82,7 @@ ), # Serve proxied API + # /_/api/v2/ url( r'^{DOC_PATH_PREFIX}api/v2/'.format( DOC_PATH_PREFIX=DOC_PATH_PREFIX, diff --git a/readthedocs/rtd_tests/tests/test_api.py b/readthedocs/rtd_tests/tests/test_api.py index 61a673de274..cb3106572ab 100644 --- a/readthedocs/rtd_tests/tests/test_api.py +++ b/readthedocs/rtd_tests/tests/test_api.py @@ -354,7 +354,9 @@ def test_update_build_without_permission(self): client = APIClient() api_user = get(User, is_staff=False, password='test') client.force_authenticate(user=api_user) - build = get(Build, project_id=1, version_id=1, state='cloning') + project = Project.objects.get(pk=1) + version = project.versions.first() + build = get(Build, project=project, version=version, state='cloning') resp = client.put( '/api/v2/build/{}/'.format(build.pk), { @@ -373,7 +375,9 @@ def test_make_build_protected_fields(self): Super users should be able to read/write the `builder` property, but we don't expose this to end users via the API """ - build = get(Build, project_id=1, version_id=1, builder='foo') + project = Project.objects.get(pk=1) + version = project.versions.first() + build = get(Build, project=project, version=version, builder='foo') client = APIClient() api_user = get(User, is_staff=False, password='test') @@ -423,7 +427,9 @@ def test_make_build_commands(self): self.assertEqual(build['commands'][0]['description'], 'foo') def test_get_raw_log_success(self): - build = get(Build, project_id=1, version_id=1, builder='foo') + project = Project.objects.get(pk=1) + version = project.versions.first() + build = get(Build, project=project, version=version, builder='foo') get( BuildCommandResult, build=build, @@ -462,8 +468,10 @@ def test_get_raw_log_success(self): ) def test_get_raw_log_building(self): + project = Project.objects.get(pk=1) + version = project.versions.first() build = get( - Build, project_id=1, version_id=1, + Build, project=project, version=version, builder='foo', success=False, exit_code=1, state='building', ) @@ -506,8 +514,10 @@ def test_get_raw_log_building(self): ) def test_get_raw_log_failure(self): + project = Project.objects.get(pk=1) + version = project.versions.first() build = get( - Build, project_id=1, version_id=1, + Build, project=project, version=version, builder='foo', success=False, exit_code=1, ) get( @@ -562,8 +572,12 @@ def test_build_filter_by_commit(self): Should return the list of builds according to the commit query params """ - get(Build, project_id=1, version_id=1, builder='foo', commit='test') - get(Build, project_id=2, version_id=1, builder='foo', commit='other') + project1 = Project.objects.get(pk=1) + project2 = Project.objects.get(pk=2) + version1 = project1.versions.first() + version2 = project2.versions.first() + get(Build, project=project1, version=version1, builder='foo', commit='test') + get(Build, project=project2, version=version2, builder='foo', commit='other') client = APIClient() api_user = get(User, is_staff=False, password='test') client.force_authenticate(user=api_user) @@ -635,7 +649,7 @@ def test_project_pagination(self): def test_remote_repository_pagination(self): account = get(SocialAccount, provider='github') - user = get(User, socialaccount_set=[account]) + user = get(User) for _ in range(20): get(RemoteRepository, users=[user], account=account) @@ -649,7 +663,7 @@ def test_remote_repository_pagination(self): def test_remote_organization_pagination(self): account = get(SocialAccount, provider='github') - user = get(User, socialaccount_set=[account]) + user = get(User) for _ in range(30): get(RemoteOrganization, users=[user], account=account) @@ -719,9 +733,9 @@ def test_permissions(self): account_a = get(SocialAccount, provider='github') account_b = get(SocialAccount, provider='github') account_c = get(SocialAccount, provider='github') - user_a = get(User, password='test', socialaccount_set=[account_a]) - user_b = get(User, password='test', socialaccount_set=[account_b]) - user_c = get(User, password='test', socialaccount_set=[account_c]) + user_a = get(User, password='test') + user_b = get(User, password='test') + user_c = get(User, password='test') org_a = get(RemoteOrganization, users=[user_a], account=account_a) repo_a = get( RemoteRepository, diff --git a/readthedocs/rtd_tests/tests/test_core_utils.py b/readthedocs/rtd_tests/tests/test_core_utils.py index 8c3430a6133..62305e949e4 100644 --- a/readthedocs/rtd_tests/tests/test_core_utils.py +++ b/readthedocs/rtd_tests/tests/test_core_utils.py @@ -174,7 +174,6 @@ def test_trigger_build_rounded_time_limit(self, update_docs): 'commit': None } options = { - 'queue': mock.ANY, 'time_limit': 3, 'soft_time_limit': 3, 'priority': CELERY_HIGH, @@ -241,7 +240,6 @@ def test_trigger_external_build_low_priority(self, update_docs): 'commit': None } options = { - 'queue': mock.ANY, 'time_limit': mock.ANY, 'soft_time_limit': mock.ANY, 'priority': CELERY_LOW, @@ -265,7 +263,6 @@ def test_trigger_build_translation_medium_priority(self, update_docs): 'commit': None } options = { - 'queue': mock.ANY, 'time_limit': mock.ANY, 'soft_time_limit': mock.ANY, 'priority': CELERY_MEDIUM, diff --git a/readthedocs/rtd_tests/tests/test_footer.py b/readthedocs/rtd_tests/tests/test_footer.py index a77d4cbb1f0..9f3e0902176 100644 --- a/readthedocs/rtd_tests/tests/test_footer.py +++ b/readthedocs/rtd_tests/tests/test_footer.py @@ -27,7 +27,7 @@ def setUp(self): privacy_level=PUBLIC, main_language_project=None, ) - self.pip.versions.update(privacy_level=PUBLIC) + self.pip.versions.update(privacy_level=PUBLIC, built=True) self.latest = self.pip.versions.get(slug=LATEST) self.url = ( @@ -224,6 +224,62 @@ def test_hidden_versions(self): self.assertIn('/en/latest/', response.data['html']) self.assertNotIn('/en/2.0/', response.data['html']) + def test_built_versions(self): + built_version = get( + Version, + slug='2.0', + active=True, + built=True, + privacy_level=PUBLIC, + project=self.pip, + ) + + # The built versions appears on the footer + self.url = ( + reverse('footer_html') + + f'?project={self.pip.slug}&version={self.latest.slug}&page=index&docroot=/' + ) + response = self.render() + self.assertIn('/en/latest/', response.data['html']) + self.assertIn('/en/2.0/', response.data['html']) + + # We can access the built version, and it appears on the footer + self.url = ( + reverse('footer_html') + + f'?project={self.pip.slug}&version={built_version.slug}&page=index&docroot=/' + ) + response = self.render() + self.assertIn('/en/latest/', response.data['html']) + self.assertIn('/en/2.0/', response.data['html']) + + def test_not_built_versions(self): + not_built_version = get( + Version, + slug='2.0', + active=True, + built=False, + privacy_level=PUBLIC, + project=self.pip, + ) + + # The un-built version doesn't appear on the footer + self.url = ( + reverse('footer_html') + + f'?project={self.pip.slug}&version={self.latest.slug}&page=index&docroot=/' + ) + response = self.render() + self.assertIn('/en/latest/', response.data['html']) + self.assertNotIn('/en/2.0/', response.data['html']) + + # We can access the unbuilt version, but it doesn't appear on the footer + self.url = ( + reverse('footer_html') + + f'?project={self.pip.slug}&version={not_built_version.slug}&page=index&docroot=/' + ) + response = self.render() + self.assertIn('/en/latest/', response.data['html']) + self.assertNotIn('/en/2.0/', response.data['html']) + class TestFooterHTML(BaseTestFooterHTML, TestCase): @@ -362,7 +418,7 @@ class TestFooterPerformance(APITestCase): # The expected number of queries for generating the footer # This shouldn't increase unless we modify the footer API - EXPECTED_QUERIES = 13 + EXPECTED_QUERIES = 14 def setUp(self): self.pip = Project.objects.get(slug='pip') @@ -378,30 +434,37 @@ def render(self): def test_version_queries(self): # The number of Versions shouldn't impact the number of queries - with self.assertNumQueries(self.EXPECTED_QUERIES): - response = self.render() - self.assertContains(response, '0.8.1') - - for patch in range(3): - identifier = '0.99.{}'.format(patch) - self.pip.versions.create( - verbose_name=identifier, - identifier=identifier, - type=TAG, - active=True, - ) - - with self.assertNumQueries(self.EXPECTED_QUERIES): - response = self.render() - self.assertContains(response, '0.99.0') + with mock.patch('readthedocs.api.v2.views.footer_views.increase_page_view_count') as mocked: + mocked.side_effect = None + + with self.assertNumQueries(self.EXPECTED_QUERIES): + response = self.render() + self.assertContains(response, '0.8.1') + + for patch in range(3): + identifier = '0.99.{}'.format(patch) + self.pip.versions.create( + verbose_name=identifier, + identifier=identifier, + type=TAG, + active=True, + built=True + ) + + with self.assertNumQueries(self.EXPECTED_QUERIES): + response = self.render() + self.assertContains(response, '0.99.0') def test_domain_queries(self): # Setting up a custom domain shouldn't impact the number of queries - self.pip.domains.create( - domain='http://docs.foobar.com', - canonical=True, - ) + with mock.patch('readthedocs.api.v2.views.footer_views.increase_page_view_count') as mocked: + mocked.side_effect = None + + self.pip.domains.create( + domain='http://docs.foobar.com', + canonical=True, + ) - with self.assertNumQueries(self.EXPECTED_QUERIES): - response = self.render() - self.assertContains(response, 'docs.foobar.com') + with self.assertNumQueries(self.EXPECTED_QUERIES): + response = self.render() + self.assertContains(response, 'docs.foobar.com') diff --git a/readthedocs/rtd_tests/tests/test_privacy_urls.py b/readthedocs/rtd_tests/tests/test_privacy_urls.py index 5c834abad55..2a2a4ddbf69 100644 --- a/readthedocs/rtd_tests/tests/test_privacy_urls.py +++ b/readthedocs/rtd_tests/tests/test_privacy_urls.py @@ -150,7 +150,7 @@ class ProjectMixin(URLAccessMixin): def setUp(self): super().setUp() - self.build = get(Build, project=self.pip) + self.build = get(Build, project=self.pip, version=self.pip.versions.first()) self.tag = get(Tag, slug='coolness') self.subproject = get( Project, slug='sub', language='ja', @@ -352,7 +352,7 @@ class APIMixin(URLAccessMixin): def setUp(self): super().setUp() - self.build = get(Build, project=self.pip) + self.build = get(Build, project=self.pip, version=self.pip.versions.first()) self.build_command_result = get(BuildCommandResult, build=self.build) self.domain = get(Domain, domain='docs.foobar.com', project=self.pip) self.social_account = get(SocialAccount) diff --git a/readthedocs/rtd_tests/tests/test_unresolver.py b/readthedocs/rtd_tests/tests/test_unresolver.py index cb17daa6d29..c86a57599b6 100644 --- a/readthedocs/rtd_tests/tests/test_unresolver.py +++ b/readthedocs/rtd_tests/tests/test_unresolver.py @@ -47,7 +47,7 @@ def test_unresolver_domain(self): self.assertEqual(parts.project.slug, 'pip') self.assertEqual(parts.language_slug, 'en') self.assertEqual(parts.version_slug, 'latest') - self.assertEqual(parts.filename, '') + self.assertEqual(parts.filename, 'index.html') def test_unresolver_single_version(self): self.pip.single_version = True @@ -55,7 +55,7 @@ def test_unresolver_single_version(self): self.assertEqual(parts.project.slug, 'pip') self.assertEqual(parts.language_slug, None) self.assertEqual(parts.version_slug, None) - self.assertEqual(parts.filename, '') + self.assertEqual(parts.filename, 'index.html') def test_unresolver_subproject_alias(self): relation = self.pip.subprojects.first() @@ -65,7 +65,7 @@ def test_unresolver_subproject_alias(self): self.assertEqual(parts.project.slug, 'sub') self.assertEqual(parts.language_slug, 'ja') self.assertEqual(parts.version_slug, 'latest') - self.assertEqual(parts.filename, '') + self.assertEqual(parts.filename, 'index.html') def test_unresolver_external_version(self): ver = self.pip.versions.first() @@ -75,7 +75,7 @@ def test_unresolver_external_version(self): self.assertEqual(parts.project.slug, 'pip') self.assertEqual(parts.language_slug, 'en') self.assertEqual(parts.version_slug, '10') - self.assertEqual(parts.filename, '') + self.assertEqual(parts.filename, 'index.html') def test_unresolver_unknown_host(self): parts = unresolve('http://random.stuff.com/en/latest/') diff --git a/readthedocs/rtd_tests/tests/test_views.py b/readthedocs/rtd_tests/tests/test_views.py index 188147866c3..f9fbb5e8159 100644 --- a/readthedocs/rtd_tests/tests/test_views.py +++ b/readthedocs/rtd_tests/tests/test_views.py @@ -286,15 +286,15 @@ def test_top_queries(self): test_time.return_value = self.test_time expected_result = [ - ('hello world', 5), - ('documentation', 4), - ('read the docs', 4), - ('advertising', 3), - ('elasticsearch', 2), - ('sphinx', 2), - ('github', 1), - ('hello', 1), - ('search', 1), + ('hello world', 5, 0), + ('documentation', 4, 0), + ('read the docs', 4, 0), + ('advertising', 3, 0), + ('elasticsearch', 2, 0), + ('sphinx', 2, 0), + ('github', 1, 0), + ('hello', 1, 0), + ('search', 1, 0), ] resp = self.client.get(self.analyics_page) diff --git a/readthedocs/search/api.py b/readthedocs/search/api.py index d5e123d2863..d36a1ab8e5d 100644 --- a/readthedocs/search/api.py +++ b/readthedocs/search/api.py @@ -1,5 +1,6 @@ import itertools import logging +import re from django.shortcuts import get_object_or_404 from django.utils import timezone @@ -9,6 +10,7 @@ from readthedocs.api.v2.permissions import IsAuthorizedToViewVersion from readthedocs.builds.models import Version +from readthedocs.projects.constants import MKDOCS, SPHINX_HTMLDIR from readthedocs.projects.models import HTMLFile, Project from readthedocs.search import tasks, utils from readthedocs.search.faceted_search import PageSearch @@ -27,15 +29,28 @@ class PageSearchSerializer(serializers.Serializer): version = serializers.CharField() title = serializers.CharField() path = serializers.CharField() + full_path = serializers.CharField() link = serializers.SerializerMethodField() highlight = serializers.SerializerMethodField() inner_hits = serializers.SerializerMethodField() def get_link(self, obj): - projects_url = self.context.get('projects_url') - if projects_url: - docs_url = projects_url[obj.project] - return docs_url + obj.path + project_data = self.context['projects_data'].get(obj.project) + if not project_data: + return None + + docs_url, doctype = project_data + path = obj.full_path + + # Generate an appropriate link for the doctypes that use htmldir, + # and always end it with / so it goes directly to proxito. + if doctype in {SPHINX_HTMLDIR, MKDOCS}: + new_path = re.sub('(^|/)index.html$', '/', path) + # docs_url already ends with /, + # so path doesn't need to start with /. + path = new_path.lstrip('/') + + return docs_url + path def get_highlight(self, obj): highlight = getattr(obj.meta, 'highlight', None) @@ -117,19 +132,24 @@ def get_queryset(self): # Validate all the required params are there self.validate_query_params() query = self.request.query_params.get('q', '') - kwargs = {'filter_by_user': False, 'filters': {}} - kwargs['filters']['project'] = [p.slug for p in self.get_all_projects()] - kwargs['filters']['version'] = self._get_version().slug - # Check to avoid searching all projects in case project is empty. - if not kwargs['filters']['project']: + filters = {} + filters['project'] = [p.slug for p in self.get_all_projects()] + filters['version'] = self._get_version().slug + + # Check to avoid searching all projects in case these filters are empty. + if not filters['project']: log.info("Unable to find a project to search") return HTMLFile.objects.none() - if not kwargs['filters']['version']: + if not filters['version']: log.info("Unable to find a version to search") return HTMLFile.objects.none() - user = self.request.user + queryset = PageSearch( - query=query, user=user, **kwargs + query=query, + filters=filters, + user=self.request.user, + # We use a permission class to control authorization + filter_by_user=False, ) return queryset @@ -155,7 +175,7 @@ def validate_query_params(self): def get_serializer_context(self): context = super().get_serializer_context() - context['projects_url'] = self.get_all_projects_url() + context['projects_data'] = self.get_all_projects_data() return context def get_all_projects(self): @@ -183,29 +203,44 @@ def get_all_projects(self): all_projects.append(version.project) return all_projects - def get_all_projects_url(self): + def get_all_projects_data(self): """ - Return a dict containing the project slug and its version URL. + Return a dict containing the project slug and its version URL and version's doctype. - The dictionary contains the project and its subprojects . Each project's - slug is used as a key and the documentation URL for that project and - version as the value. - - Example: + The dictionary contains the project and its subprojects. Each project's + slug is used as a key and a tuple with the documentation URL and doctype + from the version. Example: { - "requests": "https://requests.readthedocs.io/en/latest/", - "requests-oauth": "https://requests-oauth.readthedocs.io/en/latest/", + "requests": ( + "https://requests.readthedocs.io/en/latest/", + "sphinx", + ), + "requests-oauth": ( + "https://requests-oauth.readthedocs.io/en/latest/", + "sphinx_htmldir", + ), } :rtype: dict """ all_projects = self.get_all_projects() version_slug = self._get_version().slug - projects_url = {} + project_urls = {} for project in all_projects: - projects_url[project.slug] = project.get_docs_url(version_slug=version_slug) - return projects_url + project_urls[project.slug] = project.get_docs_url(version_slug=version_slug) + + versions_doctype = ( + Version.objects + .filter(project__slug__in=project_urls.keys(), slug=version_slug) + .values_list('project__slug', 'documentation_type') + ) + + projects_data = { + project_slug: (project_urls[project_slug], doctype) + for project_slug, doctype in versions_doctype + } + return projects_data def list(self, request, *args, **kwargs): """Overriding ``list`` method to record query in database.""" diff --git a/readthedocs/search/documents.py b/readthedocs/search/documents.py index 68dd80ee8d3..55efe0b4ca5 100644 --- a/readthedocs/search/documents.py +++ b/readthedocs/search/documents.py @@ -2,12 +2,10 @@ from django.conf import settings from django_elasticsearch_dsl import DocType, Index, fields - from elasticsearch import Elasticsearch from readthedocs.projects.models import HTMLFile, Project - project_conf = settings.ES_INDEXES['project'] project_index = Index(project_conf['name']) project_index.settings(**project_conf['settings']) @@ -50,19 +48,6 @@ class Meta: fields = ('name', 'slug', 'description') ignore_signals = True - @classmethod - def faceted_search(cls, query, user, language=None): - from readthedocs.search.faceted_search import ProjectSearch - kwargs = { - 'user': user, - 'query': query, - } - - if language: - kwargs['filters'] = {'language': language} - - return ProjectSearch(**kwargs) - @page_index.doc_type class PageDocument(RTDDocTypeMixin, DocType): @@ -148,28 +133,6 @@ def prepare_domains(self, html_file): return all_domains - @classmethod - def faceted_search( - cls, query, user, projects_list=None, versions_list=None, - filter_by_user=True - ): - from readthedocs.search.faceted_search import PageSearch - kwargs = { - 'user': user, - 'query': query, - 'filter_by_user': filter_by_user, - } - - filters = {} - if projects_list is not None: - filters['project'] = projects_list - if versions_list is not None: - filters['version'] = versions_list - - kwargs['filters'] = filters - - return PageSearch(**kwargs) - def get_queryset(self): """Overwrite default queryset to filter certain files to index.""" queryset = super().get_queryset() diff --git a/readthedocs/search/faceted_search.py b/readthedocs/search/faceted_search.py index 912e1ac99d9..e0aea1b6b68 100644 --- a/readthedocs/search/faceted_search.py +++ b/readthedocs/search/faceted_search.py @@ -1,18 +1,13 @@ import logging +from django.conf import settings from elasticsearch import Elasticsearch from elasticsearch_dsl import FacetedSearch, TermsFacet from elasticsearch_dsl.faceted_search import NestedFacet -from elasticsearch_dsl.query import Bool, SimpleQueryString, Nested, Match - -from django.conf import settings +from elasticsearch_dsl.query import Bool, Match, Nested, SimpleQueryString from readthedocs.core.utils.extend import SettingsOverrideObject -from readthedocs.search.documents import ( - PageDocument, - ProjectDocument, -) - +from readthedocs.search.documents import PageDocument, ProjectDocument log = logging.getLogger(__name__) @@ -21,7 +16,11 @@ class RTDFacetedSearch(FacetedSearch): - def __init__(self, user, **kwargs): + """Custom wrapper around FacetedSearch.""" + + operators = [] + + def __init__(self, query=None, filters=None, user=None, **kwargs): """ Pass in a user in order to filter search results by privacy. @@ -33,23 +32,21 @@ def __init__(self, user, **kwargs): self.user = user self.filter_by_user = kwargs.pop('filter_by_user', True) - # Set filters properly - for facet in self.facets: - if facet in kwargs: - kwargs.setdefault('filters', {})[facet] = kwargs.pop(facet) - - # Don't pass along unnecessary filters - for f in ALL_FACETS: - if f in kwargs: - del kwargs[f] - # Hack a fix to our broken connection pooling # This creates a new connection on every request, # but actually works :) log.info('Hacking Elastic to fix search connection pooling') self.using = Elasticsearch(**settings.ELASTICSEARCH_DSL['default']) - super().__init__(**kwargs) + filters = filters or {} + + # We may recieve invalid filters + valid_filters = { + k: v + for k, v in filters.items() + if k in self.facets + } + super().__init__(query=query, filters=valid_filters, **kwargs) def query(self, search, query): """ @@ -57,7 +54,7 @@ def query(self, search, query): Also: - * Adds SimpleQueryString instead of default query. + * Adds SimpleQueryString with `self.operators` instead of default query. * Adds HTML encoding of results to avoid XSS issues. """ search = search.highlight_options(encoder='html', number_of_fragments=3) diff --git a/readthedocs/search/migrations/0002_add_total_results_field.py b/readthedocs/search/migrations/0002_add_total_results_field.py new file mode 100644 index 00000000000..12704622569 --- /dev/null +++ b/readthedocs/search/migrations/0002_add_total_results_field.py @@ -0,0 +1,18 @@ +# Generated by Django 2.2.12 on 2020-05-14 17:29 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('search', '0001_initial'), + ] + + operations = [ + migrations.AddField( + model_name='searchquery', + name='total_results', + field=models.IntegerField(default=0, null=True, verbose_name='Total results'), + ), + ] diff --git a/readthedocs/search/models.py b/readthedocs/search/models.py index 53fedecaf96..a7cf408fe73 100644 --- a/readthedocs/search/models.py +++ b/readthedocs/search/models.py @@ -11,6 +11,7 @@ from readthedocs.builds.models import Version from readthedocs.projects.models import Project from readthedocs.projects.querysets import RelatedProjectQuerySet +from readthedocs.search.utils import _last_30_days_iter class SearchQuery(TimeStampedModel): @@ -32,6 +33,12 @@ class SearchQuery(TimeStampedModel): _('Query'), max_length=4092, ) + total_results = models.IntegerField( + _('Total results'), + default=0, + # TODO: to avoid downtime, remove later. + null=True, + ) objects = RelatedProjectQuerySet.as_manager() class Meta: @@ -58,9 +65,6 @@ def generate_queries_count_of_one_month(cls, project_slug): today = timezone.now().date() last_30th_day = timezone.now().date() - timezone.timedelta(days=30) - # this includes the current day also - last_31_days_iter = [last_30th_day + timezone.timedelta(days=n) for n in range(31)] - qs = cls.objects.filter( project__slug=project_slug, created__date__lte=today, @@ -77,17 +81,17 @@ def generate_queries_count_of_one_month(cls, project_slug): .values_list('created_date', 'count') ) - count_data = [count_dict.get(date) or 0 for date in last_31_days_iter] + count_data = [count_dict.get(date) or 0 for date in _last_30_days_iter()] # format the date value to a more readable form # Eg. `16 Jul` - last_31_days_str = [ + last_30_days_str = [ timezone.datetime.strftime(date, '%d %b') - for date in last_31_days_iter + for date in _last_30_days_iter() ] final_data = { - 'labels': last_31_days_str, + 'labels': last_30_days_str, 'int_data': count_data, } diff --git a/readthedocs/search/parse_json.py b/readthedocs/search/parse_json.py index 715bd81c866..051ac19a2e8 100644 --- a/readthedocs/search/parse_json.py +++ b/readthedocs/search/parse_json.py @@ -2,19 +2,29 @@ import logging from urllib.parse import urlparse -import orjson as json +import orjson as json from django.conf import settings from django.core.files.storage import get_storage_class - from selectolax.parser import HTMLParser - log = logging.getLogger(__name__) -def generate_page_sections(body, fjson_storage_path): - """Generate section dicts for each section.""" +def generate_page_sections(page_title, body, fjson_storage_path): + """ + Generate section dicts for each section for sphinx. + + In Sphinx sub-sections are nested, so they are children of the outer section, + and sections with the same level are neighbors. + We index the content under a section till before the next one. + + We can have pages that have content before the first title or that don't have a title, + we index that content first under the title of the original page (`page_title`). + + Contents that are likely to be a sphinx domain are deleted, + since we already index those in another step. + """ # Removing all
tags to prevent duplicate indexing with Sphinx Domains. nodes_to_be_removed = [] @@ -38,55 +48,51 @@ def generate_page_sections(body, fjson_storage_path): for node in nodes_to_be_removed: node.decompose() - # Capture text inside h1 before the first h2 - h1_section = body.css('.section > h1') - if h1_section: - h1_section = h1_section[0] - div = h1_section.parent - h1_title = h1_section.text().replace('¶', '').strip() - h1_id = div.attributes.get('id', '') - h1_content = '' - next_p = body.css_first('h1').next - while next_p: - if next_p.tag == 'div' and 'class' in next_p.attributes: - if 'section' in next_p.attributes['class']: - break - - text = parse_content(next_p.text(), remove_first_line=False) - - if h1_content: - if text: - h1_content = f'{h1_content} {text}' - else: - h1_content = text - - next_p = next_p.next - - if h1_content: - yield { - 'id': h1_id, - 'title': h1_title, - 'content': h1_content, - } + # Index content for pages that don't start with a title. + content = _get_content_from_tag(body.body.child) + if content: + yield { + 'id': '', + 'title': page_title, + 'content': content, + } - # Capture text inside h2's - section_list = body.css('.section > h2') - for tag in section_list: - div = tag.parent - title = tag.text().replace('¶', '').strip() - section_id = div.attributes.get('id', '') + # Index content from h1 to h6 headers. + for head_level in range(1, 7): + tags = body.css(f'.section > h{head_level}') + for tag in tags: + title = tag.text().replace('¶', '').strip() - content = div.text() - content = parse_content(content, remove_first_line=True) + div = tag.parent + section_id = div.attributes.get('id', '') - if content: yield { 'id': section_id, 'title': title, - 'content': content, + 'content': _get_content_from_tag(tag.next), } +def _get_content_from_tag(tag): + """Gets the content from tag till before a new section.""" + contents = [] + next_tag = tag + while next_tag and not _is_section(next_tag): + content = parse_content(next_tag.text()) + if content: + contents.append(content) + next_tag = next_tag.next + return ' '.join(contents) + + +def _is_section(tag): + """Check if `tag` is a sphinx section (linkeable header).""" + return ( + tag.tag == 'div' and + 'section' in tag.attributes.get('class', []) + ) + + def process_file(fjson_storage_path): """Read the fjson file from disk and parse it into a structured dict.""" storage = get_storage_class(settings.RTD_BUILD_MEDIA_STORAGE)() @@ -111,10 +117,20 @@ def process_file(fjson_storage_path): else: log.info('Unable to index file due to no name %s', fjson_storage_path) + if 'title' in data: + title = data['title'] + title = HTMLParser(title).text().replace('¶', '').strip() + else: + log.info('Unable to index title for: %s', fjson_storage_path) + if data.get('body'): body = HTMLParser(data['body']) body_copy = HTMLParser(data['body']) - sections = generate_page_sections(body, fjson_storage_path) + sections = generate_page_sections( + page_title=title, + body=body, + fjson_storage_path=fjson_storage_path, + ) # pass a copy of `body` so that the removed # nodes in the original don't reflect here. @@ -122,16 +138,10 @@ def process_file(fjson_storage_path): else: log.info('Unable to index content for: %s', fjson_storage_path) - if 'title' in data: - title = data['title'] - title = HTMLParser(title).text().replace('¶', '').strip() - else: - log.info('Unable to index title for: %s', fjson_storage_path) - return { 'path': path, 'title': title, - 'sections': tuple(sections), + 'sections': list(sections), 'domain_data': domain_data, } @@ -189,14 +199,15 @@ def _get_text_for_domain_data(desc): def parse_content(content, remove_first_line=False): """Removes new line characters and ¶.""" content = content.replace('¶', '').strip() + content = content.split('\n') # removing the starting text of each - content = content.split('\n') if remove_first_line and len(content) > 1: content = content[1:] - # converting newlines to ". " - content = ' '.join(text.strip() for text in content if text) + # Convert all new lines to " " + content = (text.strip() for text in content) + content = ' '.join(text for text in content if text) return content diff --git a/readthedocs/search/tasks.py b/readthedocs/search/tasks.py index 98687f33fff..cd82f8fe25e 100644 --- a/readthedocs/search/tasks.py +++ b/readthedocs/search/tasks.py @@ -144,7 +144,7 @@ def delete_old_search_queries_from_db(): @app.task(queue='web') def record_search_query(project_slug, version_slug, query, total_results, time_string): - """Record/update search query in database.""" + """Record/update a search query for analytics.""" if not project_slug or not version_slug or not query: log.debug( 'Not recording the search query. Passed arguments: ' @@ -162,24 +162,15 @@ def record_search_query(project_slug, version_slug, query, total_results, time_s modified__gte=before_10_sec, ).order_by('-modified') - # check if partial query exists, - # if yes, then just update the object. + # If a partial query exists, + # then just update that object. for partial_query in partial_query_qs.iterator(): if query.startswith(partial_query.query): partial_query.query = query + partial_query.total_results = total_results partial_query.save() return - # don't record query with zero results. - if not total_results: - log.debug( - 'Not recording search query because of zero results. Passed arguments: ' - 'project_slug: %s, version_slug: %s, query: %s, total_results: %s, time: %s' % ( - project_slug, version_slug, query, total_results, time - ) - ) - return - project = Project.objects.filter(slug=project_slug).first() if not project: log.debug( @@ -190,9 +181,9 @@ def record_search_query(project_slug, version_slug, query, total_results, time_s ) return - version_qs = Version.objects.filter(project=project, slug=version_slug) + version = Version.objects.filter(project=project, slug=version_slug).first() - if not version_qs.exists(): + if not version: log.debug( 'Not recording the search query because version does not exist. ' 'project_slug: %s, version_slug: %s' % ( @@ -201,11 +192,10 @@ def record_search_query(project_slug, version_slug, query, total_results, time_s ) return - version = version_qs.first() - - # make a new SearchQuery object. + # Create a new SearchQuery object. SearchQuery.objects.create( project=project, version=version, query=query, + total_results=total_results, ) diff --git a/readthedocs/search/tests/conftest.py b/readthedocs/search/tests/conftest.py index 1705118b380..97bba15dc45 100644 --- a/readthedocs/search/tests/conftest.py +++ b/readthedocs/search/tests/conftest.py @@ -4,7 +4,7 @@ import pytest from django.core.management import call_command -from django_dynamic_fixture import G +from django_dynamic_fixture import get from readthedocs.projects.constants import PUBLIC from readthedocs.projects.models import HTMLFile, Project @@ -28,7 +28,7 @@ def all_projects(es_index, mock_processed_json, db, settings): settings.ELASTICSEARCH_DSL_AUTOSYNC = True projects_list = [] for project_slug in ALL_PROJECTS: - project = G( + project = get( Project, slug=project_slug, name=project_slug, @@ -41,7 +41,13 @@ def all_projects(es_index, mock_processed_json, db, settings): # file_basename in config are without extension so add html extension file_name = file_basename + '.html' version = project.versions.all()[0] - html_file = G(HTMLFile, project=project, version=version, name=file_name) + html_file = get( + HTMLFile, + project=project, + version=version, + name=file_name, + path=file_name, + ) # creating sphinx domain test objects file_path = get_json_file_path(project.slug, file_basename) @@ -54,7 +60,7 @@ def all_projects(es_index, mock_processed_json, db, settings): domain_role_name = domain_data.pop('role_name') domain, type_ = domain_role_name.split(':') - G( + get( SphinxDomain, project=project, version=version, diff --git a/readthedocs/search/tests/data/docs/guides/index.json b/readthedocs/search/tests/data/docs/guides/index.json new file mode 100644 index 00000000000..5ca9b20764a --- /dev/null +++ b/readthedocs/search/tests/data/docs/guides/index.json @@ -0,0 +1,13 @@ +{ + "path": "guides/index", + "title": "Guides", + "sections": [ + { + "id": "guides", + "title": "Guides", + "content": "Content from guides/index" + } + ], + "domains": [], + "domain_data": {} +} diff --git a/readthedocs/search/tests/data/docs/index.json b/readthedocs/search/tests/data/docs/index.json new file mode 100644 index 00000000000..c85846aa5bb --- /dev/null +++ b/readthedocs/search/tests/data/docs/index.json @@ -0,0 +1,13 @@ +{ + "path": "index", + "title": "Index", + "sections": [ + { + "id": "title", + "title": "Title", + "content": "Some content from index" + } + ], + "domains": [], + "domain_data": {} +} diff --git a/readthedocs/search/tests/data/sphinx/in/no-title.html b/readthedocs/search/tests/data/sphinx/in/no-title.html new file mode 100644 index 00000000000..5af77ac2e7f --- /dev/null +++ b/readthedocs/search/tests/data/sphinx/in/no-title.html @@ -0,0 +1,7 @@ +

A page without a title.

+

Only content.

+
    +
  • One
    • +
  • Two
    • +
  • Three
    • +
diff --git a/readthedocs/search/tests/data/sphinx/in/no-title.json b/readthedocs/search/tests/data/sphinx/in/no-title.json new file mode 100644 index 00000000000..ae78faddcef --- /dev/null +++ b/readthedocs/search/tests/data/sphinx/in/no-title.json @@ -0,0 +1,59 @@ +{ + "parents": [ + { + "link": "../", + "title": "Guides" + } + ], + "prev": { + "link": "../conda/", + "title": "Conda Support" + }, + "next": { + "link": "../feature-flags/", + "title": "Feature Flags" + }, + "title": "<no title>", + "meta": {}, + "body": "", + "metatags": "", + "rellinks": [ + [ + "genindex", + "General Index", + "I", + "index" + ], + [ + "http-routingtable", + "HTTP Routing Table", + "", + "routing table" + ], + [ + "guides/feature-flags", + "Feature Flags", + "N", + "next" + ], + [ + "guides/conda", + "Conda Support", + "P", + "previous" + ] + ], + "sourcename": "guides/environment-variables.rst.txt", + "toc": "
    \n
\n", + "display_toc": false, + "page_source_suffix": ".rst", + "current_page_name": "guides/environment-variables", + "sidebars": [ + "localtoc.html", + "relations.html", + "sourcelink.html", + "searchbox.html" + ], + "customsidebar": null, + "alabaster_version": "0.7.12" +} diff --git a/readthedocs/search/tests/data/sphinx/in/page.html b/readthedocs/search/tests/data/sphinx/in/page.html new file mode 100644 index 00000000000..dcdcbac030e --- /dev/null +++ b/readthedocs/search/tests/data/sphinx/in/page.html @@ -0,0 +1,22 @@ +

Content at the beginning.

+ +
+

I Need Secrets (or Environment Variables) in my Build

+

It may happen that your documentation depends on an authenticated service to be built properly.

+
+ +
+

Title One

+

This is another H1 title.

+ +
+

Sub-title one

+

Sub title

+ +
+

Subsub title

+

This is a H3 title.

+
+
+
diff --git a/readthedocs/search/tests/data/sphinx/in/page.json b/readthedocs/search/tests/data/sphinx/in/page.json new file mode 100644 index 00000000000..464200077c7 --- /dev/null +++ b/readthedocs/search/tests/data/sphinx/in/page.json @@ -0,0 +1,59 @@ +{ + "parents": [ + { + "link": "../", + "title": "Guides" + } + ], + "prev": { + "link": "../conda/", + "title": "Conda Support" + }, + "next": { + "link": "../feature-flags/", + "title": "Feature Flags" + }, + "title": "I Need Secrets (or Environment Variables) in my Build", + "meta": {}, + "body": "", + "metatags": "", + "rellinks": [ + [ + "genindex", + "General Index", + "I", + "index" + ], + [ + "http-routingtable", + "HTTP Routing Table", + "", + "routing table" + ], + [ + "guides/feature-flags", + "Feature Flags", + "N", + "next" + ], + [ + "guides/conda", + "Conda Support", + "P", + "previous" + ] + ], + "sourcename": "guides/environment-variables.rst.txt", + "toc": "\n", + "display_toc": true, + "page_source_suffix": ".rst", + "current_page_name": "guides/environment-variables", + "sidebars": [ + "localtoc.html", + "relations.html", + "sourcelink.html", + "searchbox.html" + ], + "customsidebar": null, + "alabaster_version": "0.7.12" +} diff --git a/readthedocs/search/tests/data/sphinx/out/no-title.json b/readthedocs/search/tests/data/sphinx/out/no-title.json new file mode 100644 index 00000000000..5ad8077ab2e --- /dev/null +++ b/readthedocs/search/tests/data/sphinx/out/no-title.json @@ -0,0 +1,12 @@ +{ + "title": "", + "path": "guides/environment-variables", + "sections": [ + { + "id": "", + "title": "", + "content": "A page without a title. Only content. One Two Three" + } + ], + "domain_data": {} +} diff --git a/readthedocs/search/tests/data/sphinx/out/page.json b/readthedocs/search/tests/data/sphinx/out/page.json new file mode 100644 index 00000000000..1996fb2d976 --- /dev/null +++ b/readthedocs/search/tests/data/sphinx/out/page.json @@ -0,0 +1,31 @@ +{ + "title": "I Need Secrets (or Environment Variables) in my Build", + "path": "guides/environment-variables", + "sections": [ + { + "id": "", + "title": "I Need Secrets (or Environment Variables) in my Build", + "content": "Content at the beginning." + }, + { + "id": "i-need-secrets-or-environment-variables-in-my-build", + "title": "I Need Secrets (or Environment Variables) in my Build", + "content": "It may happen that your documentation depends on an authenticated service to be built properly." + }, + { + "content": "This is another H1 title.", + "id": "title-one", + "title": "Title One" + }, + { + "content": "Sub title", + "id": "sub-title-one", + "title": "Sub-title one"}, + { + "content": "This is a H3 title.", + "id": "subsub-title", + "title": "Subsub title" + } + ], + "domain_data": {} +} diff --git a/readthedocs/search/tests/dummy_data.py b/readthedocs/search/tests/dummy_data.py index 8c1cc9e5951..3b998c17474 100644 --- a/readthedocs/search/tests/dummy_data.py +++ b/readthedocs/search/tests/dummy_data.py @@ -1,7 +1,7 @@ PROJECT_DATA_FILES = { 'pipeline': ['installation', 'signals'], 'kuma': ['documentation', 'docker'], - 'docs': ['support', 'wiping'], + 'docs': ['support', 'wiping', 'index', 'guides/index'], } ALL_PROJECTS = PROJECT_DATA_FILES.keys() diff --git a/readthedocs/search/tests/test_api.py b/readthedocs/search/tests/test_api.py index 72d9b403edc..73025856f37 100644 --- a/readthedocs/search/tests/test_api.py +++ b/readthedocs/search/tests/test_api.py @@ -6,7 +6,14 @@ from django_dynamic_fixture import G from readthedocs.builds.models import Version -from readthedocs.projects.constants import PUBLIC +from readthedocs.projects.constants import ( + MKDOCS, + MKDOCS_HTML, + PUBLIC, + SPHINX, + SPHINX_HTMLDIR, + SPHINX_SINGLEHTML, +) from readthedocs.projects.models import HTMLFile, Project from readthedocs.search.api import PageSearchAPIView from readthedocs.search.documents import PageDocument @@ -324,6 +331,114 @@ def test_doc_search_hidden_versions(self, api_client, all_projects): first_result = data[0] assert first_result['project'] == subproject.slug + @pytest.mark.parametrize('doctype', [SPHINX, SPHINX_SINGLEHTML, MKDOCS_HTML]) + def test_search_correct_link_for_normal_page_html_projects(self, api_client, doctype): + project = Project.objects.get(slug='docs') + project.versions.update(documentation_type=doctype) + version = project.versions.all().first() + + search_params = { + 'project': project.slug, + 'version': version.slug, + 'q': 'Support', + } + resp = self.get_search(api_client, search_params) + assert resp.status_code == 200 + + result = resp.data['results'][0] + assert result['project'] == project.slug + assert result['link'].endswith('en/latest/support.html') + + @pytest.mark.parametrize('doctype', [SPHINX, SPHINX_SINGLEHTML, MKDOCS_HTML]) + def test_search_correct_link_for_index_page_html_projects(self, api_client, doctype): + project = Project.objects.get(slug='docs') + project.versions.update(documentation_type=doctype) + version = project.versions.all().first() + + search_params = { + 'project': project.slug, + 'version': version.slug, + 'q': 'Some content from index', + } + resp = self.get_search(api_client, search_params) + assert resp.status_code == 200 + + result = resp.data['results'][0] + assert result['project'] == project.slug + assert result['link'].endswith('en/latest/index.html') + + @pytest.mark.parametrize('doctype', [SPHINX, SPHINX_SINGLEHTML, MKDOCS_HTML]) + def test_search_correct_link_for_index_page_subdirectory_html_projects(self, api_client, doctype): + project = Project.objects.get(slug='docs') + project.versions.update(documentation_type=doctype) + version = project.versions.all().first() + + search_params = { + 'project': project.slug, + 'version': version.slug, + 'q': 'Some content from guides/index', + } + resp = self.get_search(api_client, search_params) + assert resp.status_code == 200 + + result = resp.data['results'][0] + assert result['project'] == project.slug + assert result['link'].endswith('en/latest/guides/index.html') + + @pytest.mark.parametrize('doctype', [SPHINX_HTMLDIR, MKDOCS]) + def test_search_correct_link_for_normal_page_htmldir_projects(self, api_client, doctype): + project = Project.objects.get(slug='docs') + project.versions.update(documentation_type=doctype) + version = project.versions.all().first() + + search_params = { + 'project': project.slug, + 'version': version.slug, + 'q': 'Support', + } + resp = self.get_search(api_client, search_params) + assert resp.status_code == 200 + + result = resp.data['results'][0] + assert result['project'] == project.slug + assert result['link'].endswith('en/latest/support.html') + + @pytest.mark.parametrize('doctype', [SPHINX_HTMLDIR, MKDOCS]) + def test_search_correct_link_for_index_page_htmldir_projects(self, api_client, doctype): + project = Project.objects.get(slug='docs') + project.versions.update(documentation_type=doctype) + version = project.versions.all().first() + + search_params = { + 'project': project.slug, + 'version': version.slug, + 'q': 'Some content from index', + } + resp = self.get_search(api_client, search_params) + assert resp.status_code == 200 + + result = resp.data['results'][0] + assert result['project'] == project.slug + assert result['link'].endswith('en/latest/') + + @pytest.mark.parametrize('doctype', [SPHINX_HTMLDIR, MKDOCS]) + def test_search_correct_link_for_index_page_subdirectory_htmldir_projects(self, api_client, doctype): + project = Project.objects.get(slug='docs') + project.versions.update(documentation_type=doctype) + version = project.versions.all().first() + + search_params = { + 'project': project.slug, + 'version': version.slug, + 'q': 'Some content from guides/index', + } + resp = self.get_search(api_client, search_params) + assert resp.status_code == 200 + + result = resp.data['results'][0] + assert result['project'] == project.slug + assert result['link'].endswith('en/latest/guides/') + class TestDocumentSearch(BaseTestDocumentSearch): diff --git a/readthedocs/search/tests/test_faceted_search.py b/readthedocs/search/tests/test_faceted_search.py index 74112321682..79a9c43b900 100644 --- a/readthedocs/search/tests/test_faceted_search.py +++ b/readthedocs/search/tests/test_faceted_search.py @@ -1,6 +1,6 @@ import pytest -from readthedocs.search.documents import PageDocument +from readthedocs.search.faceted_search import PageSearch @pytest.mark.django_db @@ -21,7 +21,7 @@ def test_search_exact_match(self, client, project, case): cased_query = getattr(query_text, case) query = cased_query() - page_search = PageDocument.faceted_search(query=query, user='') + page_search = PageSearch(query=query) results = page_search.execute() assert len(results) == 1 @@ -37,7 +37,7 @@ def test_search_combined_result(self, client, project): - Where `Foo` or `Bar` is present """ query = 'Elasticsearch Query' - page_search = PageDocument.faceted_search(query=query, user='') + page_search = PageSearch(query=query) results = page_search.execute() assert len(results) == 3 diff --git a/readthedocs/search/tests/test_parse_json.py b/readthedocs/search/tests/test_parse_json.py index 3d6b3860041..e5ab1927da3 100644 --- a/readthedocs/search/tests/test_parse_json.py +++ b/readthedocs/search/tests/test_parse_json.py @@ -7,7 +7,7 @@ from django_dynamic_fixture import get from readthedocs.builds.storage import BuildMediaFileSystemStorage -from readthedocs.projects.constants import MKDOCS +from readthedocs.projects.constants import MKDOCS, SPHINX from readthedocs.projects.models import HTMLFile, Project data_path = Path(__file__).parent.resolve() / 'data' @@ -103,3 +103,57 @@ def test_mkdocs_old_version(self, storage_open, storage_exists): ] expected_json = json.load(open(data_path / 'mkdocs/out/search_index_old.json')) assert parsed_json == expected_json + + @mock.patch.object(BuildMediaFileSystemStorage, 'exists') + @mock.patch.object(BuildMediaFileSystemStorage, 'open') + def test_sphinx(self, storage_open, storage_exists): + json_file = data_path / 'sphinx/in/page.json' + html_content = data_path / 'sphinx/in/page.html' + + json_content = json.load(json_file.open()) + json_content['body'] = html_content.open().read() + storage_open.side_effect = self._mock_open( + json.dumps(json_content) + ) + storage_exists.return_value = True + + self.version.documentation_type = SPHINX + self.version.save() + + page_file = get( + HTMLFile, + project=self.project, + version=self.version, + path='page.html', + ) + + parsed_json = page_file.processed_json + expected_json = json.load(open(data_path / 'sphinx/out/page.json')) + assert parsed_json == expected_json + + @mock.patch.object(BuildMediaFileSystemStorage, 'exists') + @mock.patch.object(BuildMediaFileSystemStorage, 'open') + def test_sphinx_page_without_title(self, storage_open, storage_exists): + json_file = data_path / 'sphinx/in/no-title.json' + html_content = data_path / 'sphinx/in/no-title.html' + + json_content = json.load(json_file.open()) + json_content['body'] = html_content.open().read() + storage_open.side_effect = self._mock_open( + json.dumps(json_content) + ) + storage_exists.return_value = True + + self.version.documentation_type = SPHINX + self.version.save() + + page_file = get( + HTMLFile, + project=self.project, + version=self.version, + path='no-title.html', + ) + + parsed_json = page_file.processed_json + expected_json = json.load(open(data_path / 'sphinx/out/no-title.json')) + assert parsed_json == expected_json diff --git a/readthedocs/search/tests/test_search_tasks.py b/readthedocs/search/tests/test_search_tasks.py index 92501dfd367..2e13066fef4 100644 --- a/readthedocs/search/tests/test_search_tasks.py +++ b/readthedocs/search/tests/test_search_tasks.py @@ -6,8 +6,6 @@ from django.urls import reverse from django.utils import timezone -from readthedocs.projects.models import Project -from readthedocs.builds.models import Version from readthedocs.search.models import SearchQuery from readthedocs.search import tasks @@ -87,8 +85,8 @@ def test_partial_queries_are_not_recorded(self, api_client): SearchQuery.objects.all().first().query == 'stack overflow' ), 'one SearchQuery should be there because partial queries gets updated' - def test_search_query_not_recorded_when_results_are_zero(self, api_client): - """Test that search queries are not recorded when they have zero results.""" + def test_search_query_recorded_when_results_are_zero(self, api_client): + """Test that search queries are recorded when they have zero results.""" assert ( SearchQuery.objects.all().count() == 0 @@ -102,10 +100,8 @@ def test_search_query_not_recorded_when_results_are_zero(self, api_client): } resp = api_client.get(self.url, search_params) - assert (resp.data['count'] == 0) - assert ( - SearchQuery.objects.all().count() == 0 - ), 'there should be 0 obj since there were no results.' + assert resp.data['count'] == 0 + assert SearchQuery.objects.all().count() == 1 def test_delete_old_search_queries_from_db(self, project): """Test that the old search queries are being deleted.""" diff --git a/readthedocs/search/tests/test_xss.py b/readthedocs/search/tests/test_xss.py index ed5d674f668..e2ca97d9b80 100644 --- a/readthedocs/search/tests/test_xss.py +++ b/readthedocs/search/tests/test_xss.py @@ -1,6 +1,6 @@ import pytest -from readthedocs.search.documents import PageDocument +from readthedocs.search.faceted_search import PageSearch @pytest.mark.django_db @@ -9,7 +9,7 @@ class TestXSS: def test_facted_page_xss(self, client, project): query = 'XSS' - page_search = PageDocument.faceted_search(query=query, user='') + page_search = PageSearch(query=query) results = page_search.execute() expected = """ <h3>XSS exploit</h3> diff --git a/readthedocs/search/utils.py b/readthedocs/search/utils.py index c6d370e7a79..70b0e5f0e80 100644 --- a/readthedocs/search/utils.py +++ b/readthedocs/search/utils.py @@ -3,13 +3,13 @@ import logging from operator import attrgetter +from django.utils import timezone from django.shortcuts import get_object_or_404 from django_elasticsearch_dsl.apps import DEDConfig from django_elasticsearch_dsl.registries import registry from readthedocs.builds.models import Version from readthedocs.projects.models import HTMLFile, Project -from readthedocs.search.documents import PageDocument log = logging.getLogger(__name__) @@ -117,6 +117,7 @@ def _indexing_helper(html_objs_qs, wipe=False): If ``wipe`` is set to False, html_objs are deleted from the ES index, else, html_objs are indexed. """ + from readthedocs.search.documents import PageDocument from readthedocs.search.tasks import index_objects_to_es, delete_objects_in_es if html_objs_qs: @@ -153,3 +154,20 @@ def _get_sorted_results(results, source_key='_source'): ] return sorted_results + + +def _last_30_days_iter(): + """Returns iterator for previous 30 days (including today).""" + thirty_days_ago = timezone.now().date() - timezone.timedelta(days=30) + + # this includes the current day, len() = 31 + return (thirty_days_ago + timezone.timedelta(days=n) for n in range(31)) + + +def _get_last_30_days_str(date_format='%Y-%m-%d'): + """Returns the list of dates in string format for previous 30 days (including today).""" + last_30_days_str = [ + timezone.datetime.strftime(date, date_format) + for date in _last_30_days_iter() + ] + return last_30_days_str diff --git a/readthedocs/search/views.py b/readthedocs/search/views.py index ba04dfa20d1..89e2e8ddaa5 100644 --- a/readthedocs/search/views.py +++ b/readthedocs/search/views.py @@ -8,13 +8,12 @@ from readthedocs.builds.constants import LATEST from readthedocs.projects.models import Project +from readthedocs.search import utils from readthedocs.search.faceted_search import ( ALL_FACETS, PageSearch, ProjectSearch, ) -from readthedocs.search import utils - log = logging.getLogger(__name__) LOG_TEMPLATE = '(Elastic Search) [%(user)s:%(type)s] [%(project)s:%(version)s:%(language)s] %(msg)s' @@ -69,15 +68,15 @@ def elastic_search(request, project_slug=None): facets = {} if user_input.query: - kwargs = {} + filters = {} for avail_facet in ALL_FACETS: value = getattr(user_input, avail_facet, None) if value: - kwargs[avail_facet] = value + filters[avail_facet] = value search = search_facets[user_input.type]( - query=user_input.query, user=request.user, **kwargs + query=user_input.query, filters=filters, user=request.user, ) results = search[:50].execute() facets = results.facets @@ -95,12 +94,10 @@ def elastic_search(request, project_slug=None): ) # Make sure our selected facets are displayed even when they return 0 results - for avail_facet in ALL_FACETS: - value = getattr(user_input, avail_facet, None) - if not value or avail_facet not in facets: - continue - if value not in [val[0] for val in facets[avail_facet]]: - facets[avail_facet].insert(0, (value, 0, True)) + for facet in facets: + value = getattr(user_input, facet, None) + if value and value not in (val[0] for val in facets[facet]): + facets[facet].insert(0, (value, 0, True)) if results: diff --git a/readthedocs/settings/base.py b/readthedocs/settings/base.py index 0dace8774a4..7cecf4089a7 100644 --- a/readthedocs/settings/base.py +++ b/readthedocs/settings/base.py @@ -1,6 +1,5 @@ # pylint: disable=missing-docstring -import getpass import os import subprocess @@ -350,7 +349,12 @@ def USE_PROMOS(self): # noqa 'task': 'readthedocs.search.tasks.delete_old_search_queries_from_db', 'schedule': crontab(minute=0, hour=0), 'options': {'queue': 'web'}, - } + }, + 'every-day-delete-old-page-views': { + 'task': 'readthedocs.analytics.tasks.delete_old_page_counts', + 'schedule': crontab(minute=0, hour=1), + 'options': {'queue': 'web'}, + }, } MULTIPLE_APP_SERVERS = [CELERY_DEFAULT_QUEUE] MULTIPLE_BUILD_SERVERS = [CELERY_DEFAULT_QUEUE] diff --git a/readthedocs/templates/projects/project_edit_base.html b/readthedocs/templates/projects/project_edit_base.html index 629985e0bf2..2199bd0005d 100644 --- a/readthedocs/templates/projects/project_edit_base.html +++ b/readthedocs/templates/projects/project_edit_base.html @@ -27,6 +27,7 @@
  • {% trans "Environment Variables" %}
  • {% trans "Automation Rules" %}
  • {% trans "Notifications" %}
    • +
  • {% trans "Traffic Analytics" %}
  • {% trans "Search Analytics" %}
    • {% if USE_PROMOS %}
  • {% trans "Advertising" %}
    • diff --git a/readthedocs/templates/projects/project_traffic_analytics.html b/readthedocs/templates/projects/project_traffic_analytics.html new file mode 100644 index 00000000000..bf4a89e1466 --- /dev/null +++ b/readthedocs/templates/projects/project_traffic_analytics.html @@ -0,0 +1,74 @@ +{% extends "projects/project_edit_base.html" %} + +{% load i18n %} +{% load static %} + +{% block title %}{% trans "Traffic Analytics" %}{% endblock %} + +{% block nav-dashboard %} class="active"{% endblock %} + +{% block project-traffic-analytics-active %}active{% endblock %} +{% block project_edit_content_header %}{% trans "Traffic Analytics" %}{% endblock %} + +{% block project_edit_content %} +

    {% trans "Top viewed pages of the past month" %}

    +

    + This beta feature can be enabled by asking support. +

    +
    +
    +
      + {% for page, count in top_viewed_pages %} +
    • + {{ page }} + {{ count }} +
      • + {% empty %} +
    • +

      + {% trans 'No date available.' %} +

      +
      • + {% endfor %} +
    +
    +
    + +
    + +

    {% trans "Overview of the past month" %}

    + + +{% endblock %} + +{% block extra_scripts %} + + +{% endblock %} + +{% block extra_links %} + + +{% endblock %} + +{% block footerjs %} + var line_chart = document.getElementById("page-views-per-page").getContext("2d"); + + {# Using |safe here is ok since this is just integers and formatted dates #} + var line_chart_labels = {{ page_data.labels|safe }}; + var line_chart_data = {{ page_data.int_data|safe }}; + + var line_chart = new Chart(line_chart, { + type: "line", + data: { + labels: line_chart_labels, + datasets: [{ + label: "# of views", + data: line_chart_data, + fill: false, + borderColor: "rgba(75, 192, 192, 1)", + pointBorderColor: "rgba(75, 192, 192, 1)", + }] + }, + }); +{% endblock %} diff --git a/readthedocs/templates/projects/projects_search_analytics.html b/readthedocs/templates/projects/projects_search_analytics.html index 93fa412f4ad..b02635fd549 100644 --- a/readthedocs/templates/projects/projects_search_analytics.html +++ b/readthedocs/templates/projects/projects_search_analytics.html @@ -16,9 +16,10 @@

    {% trans "Top queries" %}

      - {% for query, count in queries %} + {% for query, count, total_results in queries %}
    • {{ query }} + ({{ total_results }} result{{ total_results|pluralize:"s" }}) {{ count }} search{{ count|pluralize:"es" }} @@ -36,7 +37,7 @@

      {% trans "Top queries" %}


      {% if query_count_of_1_month.labels and query_count_of_1_month.int_data %} -

      {% trans "Overview of the past 1 month:" %}

      +

      {% trans "Overview of the past month" %}

      {% endif %} diff --git a/requirements/testing.txt b/requirements/testing.txt index dff25b09e2d..97adde92c7c 100644 --- a/requirements/testing.txt +++ b/requirements/testing.txt @@ -1,7 +1,7 @@ -r pip.txt -r local-docs-build.txt -django-dynamic-fixture==2.0.0 +django-dynamic-fixture==3.1.0 pytest==5.2.2 pytest-custom-exit-code==0.3.0 pytest-django==3.6.0 diff --git a/setup.cfg b/setup.cfg index f00f192f884..534e38d8c67 100644 --- a/setup.cfg +++ b/setup.cfg @@ -1,6 +1,6 @@ [metadata] name = readthedocs -version = 5.0.0 +version = 5.1.0 license = MIT description = Read the Docs builds and hosts documentation author = Read the Docs, Inc