Getting a KeyError: 'rows' error with Discourse 2.8.0.beta7

[alexseymer]

Hi,

first, thank you for the great work and for sharing it!

I use Discourse 2.8.0.beta7 and run a flask app containing only the python module.

Running the app in debug mode, I get the following output:

[2021-11-08 13:49:12 +0100] [6548] [INFO] Starting gunicorn 20.0.4
[2021-11-08 13:49:12 +0100] [6548] [INFO] Listening at: http://127.0.0.1:8000 (6548)
[2021-11-08 13:49:12 +0100] [6548] [INFO] Using worker: sync
[2021-11-08 13:49:12 +0100] [6549] [INFO] Booting worker with pid: 6549
track API called when there is no active context, data will not be stored
[2021-11-08 13:49:15,204] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/home/alex/.local/lib/python3.9/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/alex/.local/lib/python3.9/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/alex/.local/lib/python3.9/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/alex/.local/lib/python3.9/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/alex/.local/lib/python3.9/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/alex/.local/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/alex/.local/lib/python3.9/site-packages/canonicalwebteam/discourse/app.py", line 238, in document_view
    document = self.parser.parse_topic(self.parser.index_topic)
  File "/home/alex/.local/lib/python3.9/site-packages/canonicalwebteam/discourse/parsers/tutorials.py", line 41, in parse_topic
    self.tutorials = self._get_tutorials_topics()
  File "/home/alex/.local/lib/python3.9/site-packages/canonicalwebteam/discourse/parsers/tutorials.py", line 112, in _get_tutorials_topics
    response = self.api.get_topics(topics)
  File "/home/alex/.local/lib/python3.9/site-packages/canonicalwebteam/discourse/models.py", line 64, in get_topics
    return response.json()["rows"]
KeyError: 'rows'

If I understand it properly, the troubles come from line 57 to 62, where the API call takes place:

        response = self.session.post(
            f"{self.base_url}/admin/plugins/explorer/"
            f"queries/{self.get_topics_query_id}/run",
            headers=headers,
            data={"params": f'{{"topics":"{topics}"}}'},
        )

The respective logs from my server (404 Code) indicate that the call is unsuccessful:

X.X.X.X - - [08/Nov/2021:14:10:38 +0100] "POST /admin/plugins/explorer/queries/None/run HTTP/1.0" 404 99 "-" "python-requests/2.26.0"

If I am not mistaken, I require the Data Explorer Plugin for discourse to make it work. I installed it and would guess I need to create a query for Data Explorer? Could somebody help me out with a template or something?

I would suspect, that the answer is somewhere in the TutorialParser (tutorial.py), but I cannot get my head around it.

Thank you in advance!

[carkod]

Hi, thanks for trying out our module!

You need a very specific table with a specific format. Like this one:

The navigation header must exist and have those headers. The previous example is here

[alexseymer]

Hi, thank you for your swift reply. I forgot to mention that I use the tutorial module and used an identical table like in the ubuntu tutorials discourse section.

I used the table in the past with the older version of the python module, which worked flawlessly. Feel free to take a look here.

Does that mean, I do not need a specific discourse query for Data Explorer?

[alexseymer]

I took some time to play around with Data Explorer and managed to build a SQL query, which should satisfy the requirements:

-- [params]
-- int_list :topics

SELECT cooked
FROM posts
WHERE posts.topic_id IN (:topics)
GROUP BY cooked

In my understanding, the python module reads out all topics with there respective topic_id from the table @carkod mentioned. These ids get joined in a string separated by commas and handed over to the SQL query defined in the Data Explorer plugin on the discourse server. So I used a string like 16,12,60 as example for the :topics argument and could extract the relevant data. I could download the output as json and the json contains the relevant information in the "rows" attribute as requested in the model.py:

def get_topics(self, topic_ids):
        """
        This endpoint returns multiple topics HTML cooked content.
        This is possible with the Data Explorer plugin for Discourse
        we are using it to obtain multiple Tutorials content without
        doing multiple API calls.
        """
        headers = {
            "Accept": "application/json",
            "Content-Type": "multipart/form-data;",
        }

        # Run query on Data Explorer with topic IDs
        topics = ",".join([str(i) for i in topic_ids])

        response = self.session.post(
            f"{self.base_url}/admin/plugins/explorer/"
            f"queries/{self.get_topics_query_id}/run",
            headers=headers,
            data={"params": f'{{"topics":"{topics}"}}'},
        )

        return response.json()["rows"]

Unfortunately, I still get the same error in my app server. Only the error on the discourse server change from code 404 to 442.

Any ideas on where I am wrong? Thank you!

[alexseymer]

Another minor addition to the issue. I run two additional tests to identify the cause of the problem:

1. I switched from my discourse server to discourse.ubuntu.com as reference in the app;
2. I run the local development setup from ubuntu.com, which runs smoothly except for the tutorials. The tutorials stop with the exactly same KeyError.

Both tests yield the same result - KeyError.

[alexseymer]

Just to briefly follow up with how I dealt with the issue. I downgrade to Discourse 2.6.7 and my old setup worked again. I assume it relates to #101 as Discourse changed how headers are treated.