Implement mkdocs key from v2 config #4486

stsewd · 2018-08-07T16:50:36Z

This is on top of #4482, it should be merged after.

New changes are from 3f311ef

Ref #4219

humitos · 2018-08-15T18:58:13Z

I will wait until the base PR gets merged to review this.

agjohnson · 2018-09-07T16:55:56Z

Base PR is merged, care to rebase this?

stsewd · 2018-09-07T16:56:07Z

I'm rebasing this now

stsewd · 2018-09-10T18:51:11Z

I had to use an empty commit, travis wasn't being executed, even forcing the push :/

stsewd · 2018-09-10T20:48:31Z

I'll be testing this with a couple of mkdocs projects and see how it goes

stsewd

So, I realize that we use the doctype from the api and resolver, we don't have a config object there. We need to figure out how to pass the doctype there.

We can save the config as metadata in each version
We can save only the doctype in each version
We can write some values to a file as we do to detect an old environment.

stsewd · 2018-09-11T05:19:29Z

readthedocs/projects/tasks.py

@@ -470,10 +470,6 @@ def run_build(self, docker, record):

        # Environment used for building code, usually with Docker
        with self.build_env:
-
-            if self.project.documentation_type == 'auto':
-                self.update_documentation_type()


I move this logic to the save method of the project model

stsewd · 2018-09-11T05:20:49Z

readthedocs/projects/tasks.py


 # Web tasks
 @app.task(queue='web')
-def sync_files(project_pk, version_pk, hostname=None, html=False,
+def sync_files(project_pk, version_pk, config, hostname=None, html=False,


We need the current doctype, so I'm passing it to all tasks

stsewd · 2018-09-11T05:41:57Z

Search tests are failing, but before fixing them, I'd like to resolve #4486 (review)

davidfischer · 2018-09-11T15:36:13Z

readthedocs/doc_builder/backends/mkdocs.py

@@ -198,6 +199,8 @@ def build(self):
            '--site-dir', self.build_dir,
            '--config-file', self.yaml_file,
        ]
+        if self.config.mkdocs.fail_on_warning:
+            build_command.append('--strict')


Personally, I'd let users configure this in their mkdocs.yml rather than adding an extra Read the Docs option.

I'm +1 for this, I didn't know about that, Sphinx doesn't have something like that (I mean, using the conf.py file).

I think you could define setup() function in conf.py and override config.warningiserror = True and I think it should work.

stsewd · 2018-09-11T19:33:53Z

We can use https://docs.djangoproject.com/en/1.9/ref/contrib/postgres/fields/#jsonfield to store metadata on the build/version object.

stsewd · 2018-09-11T20:07:47Z

We can't use the readthedocs-env.json file, because it is saved on the builders (those are deleted frequently)

stsewd · 2018-09-11T20:13:54Z

So, what I think:

use a json type on the build object (we can get this from version object, querying the latest build of that version).
Or we can just add a new field (documentation_type) to the Version model.

I prefer 1), as we can save all the metadata used to build the docs, and query that data (like show the exact configuration used to build the docs). One downside is that we need to figure out how to make it work on development with sqlite.

Edit: We can use https://pypi.org/project/jsonfield/ or https://pypi.org/project/django-jsonfield/ if we want to keep the code database agnostic. Or maybe recommend using docker to new contributors (time to setup a docker-compose environment for dev?).

stsewd · 2018-09-17T18:08:54Z

readthedocs/doc_builder/python_environments.py

@@ -236,7 +236,7 @@ def install_core_requirements(self):
            'recommonmark==0.4.0',
        ]

-        if self.project.documentation_type == 'mkdocs':
+        if self.config.doctype == 'mkdocs':


I'm leaving the doctype access in some places where we always use the config object, it doesn't break anything anyway. Just keeping the same pattern.

stsewd · 2018-09-18T18:18:37Z

readthedocs/config/config.py

@@ -138,7 +138,7 @@ def __init__(self, env_config, raw_config, source_file, source_position):
    def error(self, key, message, code):
        """Raise an error related to ``key``."""
        source = '{file} [{pos}]'.format(
-            file=self.source_file,
+            file=os.path.relpath(self.source_file, self.base_path),


We were showing the full path here (/rtd/builds/...)

stsewd · 2018-09-18T18:19:43Z

readthedocs/config/config.py

@@ -566,6 +573,8 @@ def validate(self):
        self.validate_doc_types()
        self._config['mkdocs'] = self.validate_mkdocs()
        self._config['sphinx'] = self.validate_sphinx()
+        # TODO: remove later
+        self.validate_final_doc_type()


We need to remove this later #4638 (there is a note with the commit there)

stsewd · 2018-09-18T18:22:23Z

readthedocs/projects/models.py

-    def is_type_mkdocs(self):
-        """Is project type Mkdocs."""
-        return 'mkdocs' in self.documentation_type
-


Removing these is a first step to deprecate Project.doctype

agjohnson

Noted a couple small changes, but i think this looks good otherwise!

agjohnson · 2018-09-18T18:53:31Z

readthedocs/config/config.py

+            self.error(
+                key,
+                'Your documentation type should match '
+                'the one from the admin panel of your project.',


nitpick: "project admin dashboard" instead of "admin panel of your panel"

Also, perhaps instead mention the expected value here?

"Your project is configured as '{ project.documentation_type }' in your project admin dashboard, but there was no "sphinx" key specified in the configuration."

Something like that

agjohnson · 2018-09-18T18:55:45Z

readthedocs/config/tests/test_config.py

@@ -1333,6 +1336,7 @@ def test_sphinx_builder_default(self):
        build.validate()
        build.sphinx.builder == 'sphinx'

+    @pytest.mark.skip


Why is this skipped?

Perhaps we should get in the habit of code commenting why a test is xfail, skip, etc. Or i think you can do:

@pytest.mark.skip("Skipped because ...")

Those tests are incompatible with the new behavior of checking the same doctype, all those changes are part of #4638 (comment)

agjohnson · 2018-09-18T19:31:41Z

readthedocs/rtd_tests/tests/test_config_integration.py

        self.project.save()

        update_docs = self.get_update_docs_task()
        update_docs.build_docs_html()

        get_builder_class.assert_called_with(result)

+    @pytest.mark.skip


Another spot to note why we're skipping

stsewd · 2018-09-19T16:13:28Z

@agjohnson done!

agjohnson

One last update on the error copy and I think we're good to merge.

agjohnson · 2018-09-20T15:27:55Z

readthedocs/config/config.py

-                'Your documentation type should match '
-                'the one from the admin panel of your project.',
+                'Your project is configured as "{doctype}" in your admin '
+                'dashboard, but there was a "{key}" key specified.'.format(


One last nitpick. The error is sort of inverted i feel -- we don't actually tell the user the they need a "sphinx" key, just that they can't have a "mkdocs" key. I feel we need to be clearer here, ie:

Your project is configured as "Sphinx" in your project admin dashboard, but there is no "sphinx" key specified.

Also, for some of the odd documentation types, it looks like we're putting the slug in there, so:

Your project is configured as "sphinx_htmldir" ...

I feel like we want the verbose name instead. This shouldn't be a huge change, just using the model field to resolve the verbose name, but if it is a large change, it's not horribly important. This piece is temporary and it's probably close enough for most users.

Mmm, I'm not sure about accessing the db from the config module (we don't do that there). I'm changing the error message.

stsewd · 2018-09-20T16:17:58Z

@agjohnson done

agjohnson

Looks like the last fix wasn't correct.

agjohnson · 2018-09-20T16:23:02Z

readthedocs/config/config.py

+            )
+            self.error(
+                key,
+                'Your project is configured as "{doctype}" in your admin '


doctype here is the interpolation that could be displayed as sphinx_htmldir. key in this string can only be sphinx or mkdocs.

agjohnson · 2018-09-20T16:28:22Z

readthedocs/config/config.py

+                'Your project is configured as "{doctype}" in your admin '
+                'dashboard, but there is no "{key}" key specified.'.format(
+                    doctype=dashboard_doctype,
+                    key=needed_doctype,


I think what you meant was:

doctype=needed_doctype, key=key,

This was on purpose, as we say that the project is configured as sphinx_htmldir in the admin dashboard, and the user is missing a sphinx key in their config file.

key represents the current doctype from the config file.

Dashboard -> sphinx
Config -> mkdocs

Error: Your project is configured as "sphinx" in your admin dashboard, but there is no "sphinx" key specified.

The key is used to indicate where is the error something like File, key: error message

so, in the previous example, key would be mkdocs We can put the key as . if that's looks confusing

Also, we need the dashboard_doctype, otherwise we will have some weird errors like:

Dashboard -> sphinx_htmldir
Config -> sphinx.html

Your project is configured as "sphinx" in your admin dashboard, but there is no "sphinx" key specified

The current message for this case looks like:

Your project is configured as "sphinx_htmldir" in your admin dashboard, but there is no "sphinx" key specified

Isn't perfect neither... but better than the above p:

What about Your project is configured as "sphinx_htmldir" in your admin dashboard, it doesn't match the documentation type from the configuration

Isn't perfect neither... but better than the above p:

Exposing our internal slug to users isn't great UX, so i actually prefer the first. I wanted to originally use the verbose option name, but this is maybe annoying to get from the choice list -- for example, if i was using the model, i'd use the get_documentation_type_display() automatic function.

The _get_FIELD_display method is very small, we can likely replicate it very easily:
https://github.com/django/django/blob/cc79c7ee637e65c8da27e56d746c87903d5ec901/django/db/models/base.py#L902-L904

That way, we give the user an error that matches what the user sees in our admin, not what we see in our database. So, ideally:

Your project is configured as Sphinx HTMLdir in your admin dashboard, but there is no "sphinx" key specified

stsewd · 2018-09-21T18:22:32Z

readthedocs/config/config.py

+        'sphinx': 'Sphinx Html',
+        'sphinx_htmldir': 'Sphinx HtmlDir',
+        'sphinx_singlehtml': 'Sphinx Single Page HTML',
+    }


I didn't want to bring these from the db, as the config module is isolated from the other rtd code

I'm probably -1 on maintaining this in the config module separately. Was this to ensure we could move the config module out of rtd core?

I'd say either rely on the model field choices to report messages like ".. your project is configured as Sphinx HTMLdir", or skip all mentions of the Sphinx builder. I think what I was trying to describe in an earlier review was that we should skip mentioning the exact builder entirely if this complicates things too much (which seems like it is).

Yeah, that was the reason. I think we can pair haha

agjohnson

Sorry! More review on the error reporting. I think I might have suggested changes that complicate this PR further. If it helps to pair on this really quick, let me know!

agjohnson · 2018-09-24T15:40:04Z

readthedocs/config/config.py

+        'sphinx': 'Sphinx Html',
+        'sphinx_htmldir': 'Sphinx HtmlDir',
+        'sphinx_singlehtml': 'Sphinx Single Page HTML',
+    }


I'm probably -1 on maintaining this in the config module separately. Was this to ensure we could move the config module out of rtd core?

I'd say either rely on the model field choices to report messages like ".. your project is configured as Sphinx HTMLdir", or skip all mentions of the Sphinx builder. I think what I was trying to describe in an earlier review was that we should skip mentioning the exact builder entirely if this complicates things too much (which seems like it is).

agjohnson · 2018-09-24T15:56:09Z

readthedocs/config/config.py

-                code=INVALID_KEYS_COMBINATION,
-            )
+            error_msg = (
+                'Your project is configured as "{}" in your admin dashboard.'


The error message shouldn't end with a period, it results in two sentence fragments:

Your project is configured as "Mkdocs" in your admin dashboard. But there is no "mkdocs" key specified.

It would be better as:

... in your admin, but there is no ..

agjohnson · 2018-09-25T15:04:20Z

readthedocs/config/config.py

+                    'mkdocs' if dashboard_doctype == 'mkdocs' else 'sphinx'
+                )
+            else:
+                error_msg += ' But your "sphinx.builder" key does not match.'


We're no longer catching the case where the user has no sphinx key at all. I still think this needs to be a separate check on top of checking if there is a sphinx key.

The check for no sphinx key is above

agjohnson · 2018-09-27T14:36:39Z

Thanks for repeatedly taking my nitpick abuse! 🙂

Looks good!

stsewd · 2018-09-27T15:05:11Z

No worries, I'm a nitpick person too haha

humitos · 2018-10-01T09:50:06Z

I miss something here. I understood that you said that we were returning documentation_type in the API response. What are we returning now? If I'm reading correctly, we didn't remove that field from our DB and we are returning it as usual, but just added a validation on the config file that fails when the doctype from the YAML object differs from the one set in the admin. Am I right?

stsewd · 2018-10-01T16:03:07Z

@humitos yes, we keep the documentation_type here, we opened #4638 to track the complete remove

stsewd mentioned this pull request Aug 15, 2018

Match v1 config interface to new one #4456

Merged

agjohnson added PR: work in progress Pull request is not ready for full review Status: blocked Issue is blocked on another issue and removed PR: work in progress Pull request is not ready for full review labels Aug 27, 2018

agjohnson added this to the YAML File Completion milestone Aug 27, 2018

stsewd force-pushed the build-config-v2-mkdocs branch 2 times, most recently from 8ad9594 to fe5de39 Compare September 10, 2018 17:45

stsewd removed the Status: blocked Issue is blocked on another issue label Sep 10, 2018

stsewd force-pushed the build-config-v2-mkdocs branch 2 times, most recently from 1cb4d95 to 7892ec3 Compare September 10, 2018 18:31

stsewd commented Sep 11, 2018

View reviewed changes

davidfischer reviewed Sep 11, 2018

View reviewed changes

stsewd self-assigned this Sep 12, 2018

stsewd force-pushed the build-config-v2-mkdocs branch from a8db61a to 463f879 Compare September 13, 2018 15:10

agjohnson mentioned this pull request Sep 17, 2018

Deprecate Project.documentation_type #4638

Closed

6 tasks

stsewd force-pushed the build-config-v2-mkdocs branch 2 times, most recently from 9ddbb78 to ff1a8df Compare September 17, 2018 17:27

stsewd commented Sep 17, 2018

View reviewed changes

stsewd added 3 commits September 18, 2018 11:51

Implement mkdocs key

c1d791e

Add mkdocs key to v1

5918ce0

Unskip tests

019fc27

stsewd added 7 commits September 18, 2018 11:51

Use config object in python env

8b077e9

Move auto detection for doctype

050c974

Use config object to get doctype

c469d64

Remove mock

ce16fe7

Revert changes on documentation_type/doctype

9caace0

Fix test

d9d2f25

Validate same doctype (web and config file)

b0d23a6

stsewd force-pushed the build-config-v2-mkdocs branch from 80e3904 to b0d23a6 Compare September 18, 2018 16:51

Use relative path to shown error

3962715

stsewd commented Sep 18, 2018

View reviewed changes

agjohnson reviewed Sep 19, 2018

View reviewed changes

stsewd added 2 commits September 19, 2018 11:04

Better error message

1aa8126

Add reason on skipped tests

ca60dca

agjohnson reviewed Sep 20, 2018

View reviewed changes

Better error message

96f6241

agjohnson reviewed Sep 20, 2018

View reviewed changes

Better error message

a03da2b

stsewd commented Sep 21, 2018

View reviewed changes

agjohnson reviewed Sep 25, 2018

View reviewed changes

Better error msg

357f034

agjohnson merged commit df715de into readthedocs:master Sep 27, 2018

stsewd deleted the build-config-v2-mkdocs branch September 27, 2018 15:05

stsewd mentioned this pull request Oct 26, 2018

Support passing configuration file name to mkdocs #2275

Closed

Implement mkdocs key from v2 config #4486

Implement mkdocs key from v2 config #4486

Conversation

stsewd commented Aug 7, 2018 • edited Loading

humitos commented Aug 15, 2018

agjohnson commented Sep 7, 2018

stsewd commented Sep 7, 2018

stsewd commented Sep 10, 2018

stsewd commented Sep 10, 2018

stsewd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsewd commented Sep 11, 2018

Choose a reason for hiding this comment

stsewd Sep 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsewd commented Sep 11, 2018

stsewd commented Sep 11, 2018

stsewd commented Sep 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agjohnson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsewd commented Sep 19, 2018

agjohnson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsewd commented Sep 20, 2018

agjohnson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stsewd Sep 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agjohnson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agjohnson commented Sep 27, 2018

stsewd commented Sep 27, 2018

humitos commented Oct 1, 2018

stsewd commented Oct 1, 2018

stsewd commented Aug 7, 2018 •

edited

Loading

stsewd Sep 11, 2018 •

edited

Loading

stsewd commented Sep 11, 2018 •

edited

Loading

stsewd Sep 20, 2018 •

edited

Loading