Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix and test for include_total not working #4448

Merged
merged 5 commits into from
Sep 14, 2018
Merged

Conversation

davidread
Copy link
Contributor

@davidread davidread commented Sep 7, 2018

include_total wasn't working because of the default() validator was broken, so I fixed that.

Fixes #4446

Proposed fixes:

Features:

  • includes tests covering changes
  • includes updated documentation
  • includes user-visible changes
  • includes API changes
  • includes bugfix for possible backport

Please [X] all the boxes above that apply

assert res_dict['success'] is True
result = res_dict['result']
assert 'total' not in result

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you add this to the class at the top instead of with the legacy tests and like those tests use helpers.call_action to skip the request processing bits.

@@ -143,7 +143,7 @@ def result_page(offs, lim):
'offset': offs,
'sort': '_id',
'records_format': records_format,
'include_total': 'false', # XXX: default() is broken
'include_total': 'false',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can change this to False now 🙌

@davidread
Copy link
Contributor Author

@wardi thanks for the review. I've made those two improvements now :)

@wardi wardi self-assigned this Sep 11, 2018
@wardi
Copy link
Contributor

wardi commented Sep 13, 2018

@amercader mentioned that the change to default should be added to the changelog

@davidread
Copy link
Contributor Author

I've added these things to the changelog.

I've also removed an unnecessary total/SELECT COUNT(*) call by resource_view_get_fields(). I think it hangs well with this same PR since it is the same topic and is safe one-liner.

This PR now gets rid of 2 of the 4 SELECT COUNT(*) calls done when the user views the resource page. Which should make a difference to server load for large resources (e.g. 1 million rows / 100MB +) - see ckan/ckanext-xloader#41

CHANGELOG.rst Outdated
Minor changes:

* For navl schemas, the 'default' validator no longer applies the default when
the value is False, (), [] or {} (#4448)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remove () (shouldn't appear as part of a json payload) and add 0 to this list.

@@ -74,6 +74,8 @@ def ignore(key, data, errors, context):
raise StopOnError

def default(default_value):
'''When key is missing or value is 'empty', set it with a default value.
See the code for how 'empty' is defined.'''
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it hurts to repeat ourselves here.

When key is missing or value is an empty string or None, replace it with a default value

One day we could include the validator docstrings in the proper docs.

@@ -2280,7 +2280,8 @@ def resource_view_get_fields(resource):

data = {
'resource_id': resource['id'],
'limit': 0
'limit': 0,
'include_total': False,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good!

@davidread
Copy link
Contributor Author

Comments addressed!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

In datastore_search, parameter include_total=False doesn't work
3 participants