Columns starting with an underscore behave poorly in filters

[simonw]

Similar bug to #1525 (and #1506 before it). Start on https://latest.datasette.io/fixtures/facetable?_facet=_neighborhood - then select a neighborhood - then try to remove that filter using the little "x" and submitting the form again.

[simonw]

I think the root cause is this hidden form field on https://latest.datasette.io/fixtures/facetable?_facet=_neighborhood&_neighborhood__exact=Downtown

            <input type="hidden" name="_facet" value="_neighborhood">
            <input type="hidden" name="_neighborhood__exact" value="Downtown">
        <input type="submit" value="Apply">

[simonw]

I started fiddling with a test for this which extracts the <select> options and ALL of the <input> fields, but I probably won't use it:

def test_exact_parameter_results_in_correct_hidden_fields(app_client):
    # https://github.com/simonw/datasette/issues/1527
    response = app_client.get(
        "/fixtures/facetable?_facet=_neighborhood&_neighborhood__exact=Downtown"
    )
    # In this case we should NOT have a hidden _neighborhood__exact=Downtown field
    form = Soup(response.body, "html.parser").find("form")
    selects = [
        {
            "name": select["name"],
            "value": select.select("option[selected]")[0].text
            if select.select("option[selected]")
            else "",
        }
        for select in form.findAll("select")
    ]
    inputs = [input.attrs for input in form.findAll("input")]
    # Turn those both into a {name: (value, type)} array
    form_inputs = {}
    form_inputs.update(
        {select["name"]: (select["value"], "select") for select in selects}
    )
    form_inputs.update(
        {
            input["name"]: (input.get("value"), input["type"])
            for input in inputs
            if input.get("name")
        }
    )
    assert form_inputs == {
        "_filter_column_1": ("_neighborhood", "select"),
        "_filter_op_1": ("=", "select"),
        "_filter_value_1": ("Downtown", "text"),
        "_filter_column": ("", "select"),
        "_filter_op": ("", "select"),
        "_filter_value": (None, "text"),
        "_sort": ("Sort by pk", "select"),
        "_sort_by_desc": (None, "checkbox"),
        "_facet": ("_neighborhood", "hidden"),
        "_neighborhood__exact": ("Downtown", "hidden"),
    }

The problem is that last hidden field, _neighborhood__exact=Downtown - which should not be there.

[simonw]

Manually tested this too, looks like that fixed it.

[simonw]

Found a new case of this bug: click the "Apply" button on https://latest.datasette.io/fixtures/facetable?_sort=pk&_city_id__gt=1

[simonw]

That's because that page has this unnecessary hidden form field:

<input type="hidden" name="_city_id__gt" value="1">

That field is added by this bit in the template:

datasette/datasette/templates/table.html

Lines 119 to 122 in 515f8d3

	{% for key, value in form_hidden_args %}
	<input type="hidden" name="{{ key }}" value="{{ value }}">
	{% endfor %}
	<input type="submit" value="Apply">

Which is populated here:

datasette/datasette/views/table.py

Lines 813 to 821 in ace8656

	form_hidden_args = []
	for key in request.args:
	if (
	key.startswith("_")
	and key not in ("_sort", "_search", "_next")
	and not key.endswith("__exact")
	):
	for value in request.args.getlist(key):
	form_hidden_args.append((key, value))

[simonw]

Oh! This is because _city_id has a leading underscore (for testing purposes). I think I need to filter out any keys that contain __ in that case.

What happens to columns that contain a __? They shouldn't be reflected in the hidden arguments either - this code is really only supposed to catch things like _where and _m2m_through= and _col - basically most of the list on https://docs.datasette.io/en/stable/json_api.html#special-table-arguments

[simonw]

Bug is fixed on https://latest.datasette.io/fixtures/facetable?_sort=pk&_city_id__gt=1