Problem: autoescape not getting passed to urlize_quoted_links filter

[dkliban]

Solution: set needs_autoescape=True when registering the filter

Without this patch, the disabling autoescape in the template does not work.

refs #6201

[carltongibson]

Hi @dkliban. Can you please add a failing test case (with your example from #6201) to go with this change? Thanks.

[dkliban]

@carltongibson I've added a test. It fails when I remove my patch and passes with it. Can you think of a better name for the test?

[rpkilby]

Thanks @dkliban. Confirmed on my end that the test does fail without the fix.

[carltongibson]

Yep. Super. Thanks!

[zyv]

@dkliban unfortunately, after your fix the browsable API became unusable if the response contains HTML, because it is now unescaped, as autoescape is off in base.html, but if autoescape is turned on in the template, then the content is escaped twice - first in the function, and then outside.

class TestViewSet(GenericViewSet):
    def list(self, request):
        return Response({"a": "<h1>badmarker</h1> - https://www.google.com"})

is now rendered like this (wrong)

{
    "a": "<h1>badmarker</h1> - <a href="https://www.google.com" rel="nofollow">https://www.google.com</a>"
}

instead of this (right)

{
    &quot;a&quot;: &quot;&lt;h1&gt;badmarker&lt;/h1&gt; - <a href="https://www.google.com" rel="nofollow">https://www.google.com</a>&quot;
}

I don't quite understand from your test, what was your motivation to change this function to respect autoescape behaviour? As the other test shows, before your change it was working correctly with browsable API...

/cc @carltongibson

[carltongibson]

@zyv can you open a PR adding your test case? It may be we need to revert this, but let’s have a look at it.

[dkliban]

@zyv I added the patch because the filter was not adhering to the API. In my code[0] the filter is chained with other filters. The filter that processed the string after 'urlize_quoted_urls' was receiving data that was escaped when autoescape was supposed to be disabled.

[0] https://github.com/pulp/pulp/blob/master/pulpcore/pulpcore/app/templates/rest_framework/api.html#L176

[xordoquy]

@dkliban thanks for the feedback. This fix may only highlight another issue. We'll have a closer look at it with a failing test case.

[zyv]

@dkliban thank you for the explanation! I currently believe that you were misled by the parameter name and it was never supposed to adhere to the API in the first place, it's just that the parameter name is confusing, but that's only my working theory. I will hopefully have a look into your case as time permits.

@carltongibson thanks for chiming in, I have opened a new PR with a test that demonstrates the issue.

[sloria]

We ran into the issue @zyv described above (HTML getting rendered in the browseable API) and have to downgrade to 3.8.2. Perhaps this change should get reverted for now, until a proper fix for the OP is found.

[zyv]

I'm a bit worried that there haven't been any visible progress on #6330 so far, given that it's a gaping XSS hole for anyone using Browsable API and returning user-entered data as a part of the response.

I second the opinion of @sloria, that maybe if a proper fix has proved to be challenging, it would be great to merge #6330, revert #6191 and make a 3.9.1 bugfix release... Thanks!

[carltongibson]

Thanks for your concern @zyv. The issue is just time to examine it properly rather than it looking particularly difficult. Reverting is one option, but we need to make sure that’s the correct option before just rushing on. And that needs capacity, which can take a while in an open source project.

You’re welcome to take it on yourself if you like.

Or, if you’re in a desperate rush, you can momentarily fork and revert, and then deploy your fork, using Pip’s excellent VCS support.

Failing either of those, we’ll get to it.

[zyv]

Alright, since we've already started migration to 3.9.x, and the only thing holding us back is this XSS issue, I've decided to follow @carltongibson suggestion and made a custom unofficial DRF 3.9.0.post1 release with this commit reverted.

If anyone is interested, the pre-built wheel that we'll be using is available at https://moneymeets.github.io/django/djangorestframework/ and the corresponding source at
https://github.com/moneymeets/django-rest-framework/tree/moneymeets/3.9.0.post1 , but, of course, keep in mind that I'm just a random guy on the Internet, this is no official DRF release and if something goes awry you keep the pieces...

[rpkilby]

Ach - sorry for the long response time. I had started writing something out but then got distracted.

I've replied to the PR with some thoughts.

[zyv]

I think I've found a correct solution to the problem in #6330 and I would appreciate your feedback @sloria @dkliban .