Adds support for accuracyThreshold #1703

oliverroick · 2017-08-08T09:24:59Z

Proposed changes in this pull request

Implements #597: Adds support for XLSForm's accuracyThreshold to set the minimum GPS accuracy required when collecting coordinates in GeoODK.

Adds questionnaires.validators.validate_accuracy, which checks if a given value is a positive float. This is used in various places to validate input for accuracyThreshold.
Refactor questionnaires.models.Question.TYPE_CHOICES into questionnaires.choices.QUESTION_TYPES to resolve a circular import between models and validators
Adds the new field gps_accuracy to questionnaires.models.Question.
Extends questionnaires.managers.QuestionManager.create_from_dict so it reads and stores accuracyThreshold from XLSForms.
Extends xforms.renderers.XFormRenderer so accuracyThreshold is added to rendered XForms, which are used by GeoODK.
Adds field gps_accuracy to questionnaires.serializers.QuestionSerializer so the value is read and written via the JSON API.
Extends questionnaires.validators.QUESTION_SCHEMA so gps_accuracy is validated using questionnaires.validators.validate_accuracy. questionnaires.validators.validate_schema is also extended, so fields can be validated using validation functions.

When should this PR be merged

Should go into the sprint 20 release.

Risks

None.

Follow-up actions

The rendering of accuracyThreshold into XForms needs to be verified using GeoODK.

Checklist (for reviewing)

General

Is this PR explained thoroughly? All code changes must be accounted for in the PR description.
- Review 1
- Review 2
Is the PR labeled correctly? It should have the migration label if a new migration is added.
- Review 1
- Review 2
Is the risk level assessment sufficient? The risks section should contain all risks that might be introduced with the PR and which actions we need to take to mitigate these risks. Possible risks are database migrations, new libraries that need to be installed or changes to deployment scripts.
- Review 1
- Review 2

Functionality

Are all requirements met? Compare implemented functionality with the requirements specification.
- Review 1
- Review 2
Does the UI work as expected? There should be no Javascript errors in the console; all resources should load. There should be no unexpected errors. Deliberately try to break the feature to find out if there are corner cases that are not handled.
- Review 1
- Review 2

Code

Do you fully understand the introduced changes to the code? If not ask for clarification, it might uncover ways to solve a problem in a more elegant and efficient way.
- Review 1
- Review 2
Does the PR introduce any inefficient database requests? Use the debug server to check for duplicate requests.
- Review 1
- Review 2
Are all necessary strings marked for translation? All strings that are exposed to users via the UI must be marked for translation.
- Review 1
- Review 2
Is the code documented sufficiently? Large and complex classes, functions or methods must be annotated with comments following our code-style guidelines.
- Review 1
- Review 2
Has the scalability of this change been evaluated?
- Review 1
- Review 2
Is there a maintenance plan in place?
- Review 1
- Review 2

Tests

Are there sufficient test cases? Ensure that all components are tested individually; models, forms, and serializers should be tested in isolation even if a test for a view covers these components.
- Review 1
- Review 2
If this is a bug fix, are tests for the issue in place? There must be a test case for the bug to ensure the issue won’t regress. Make sure that the tests break without the new code to fix the issue.
- Review 1
- Review 2
If this is a new feature or a significant change to an existing feature? has the manual testing spreadsheet been updated with instructions for manual testing?
- Review 1
- Review 2

Security

Confirm this PR doesn't commit any keys, passwords, tokens, usernames, or other secrets.
- Review 1
- Review 2
Are all UI and API inputs run through forms or serializers?
- Review 1
- Review 2
Are all external inputs validated and sanitized appropriately?
- Review 1
- Review 2
Does all branching logic have a default case?
- Review 1
- Review 2
Does this solution handle outliers and edge cases gracefully?
- Review 1
- Review 2
Are all external communications secured and restricted to SSL?
- Review 1
- Review 2

Documentation

Are changes to the UI documented in the platform docs? If this PR introduces new platform site functionality or changes existing ones, the changes must be documented in the Cadasta Platform Documentation.
- Review 1
- Review 2
Are changes to the API documented in the API docs? If this PR introduces new API functionality or changes existing ones, the changes must be documented in the API docs.
- Review 1
- Review 2
Are reusable components documented? If this PR introduces components that are relevant to other developers (for instance a mixin for a view or a generic form) they should be documented in the Wiki.
- Review 1
- Review 2

seav

This is a solid piece of work! I only have one inline code comment. Aside from that I have the following questions:

Does pyxform validate that accuracyThreshold is only present on geo fields? I don't think it makes sense for us to reject XLSForms that have invalid values for accuracyThreshold on non-geo fields. If pyxform doesn't support this validation, should we build it ourselves?
For the manual test cases, I suppose we should add test cases for attempting to upload XLSForms with valid and various invalid values (like -1.5, 0, and STRING).

seav · 2017-08-09T06:23:46Z

cadasta/questionnaires/tests/test_validators.py

@@ -45,6 +63,10 @@ def test_validate_id_string_contains_whitespace(self):
            "'id_string' cannot be blank or contain whitespace.")


+def positive(val):
+    return val > 1


The function name doesn't seem to match the condition. Should this be val > 0? (But this is admittedly inconsequential in the grand scheme of things.)

Changed that.

oliverroick · 2017-08-09T15:59:39Z

Does pyxform validate that accuracyThreshold is only present on geo fields? I don't think it makes sense for us to reject XLSForms that have invalid values for accuracyThreshold on non-geo fields. If pyxform doesn't support this validation, should we build it ourselves?

pyxform does not check if accuracyThreshold is only present on geo fields. I agree that it probably makes sense to only validate those for geometry fields. I'll work on it tomorrow.

For the manual test cases, I suppose we should add test cases for attempting to upload XLSForms with valid and various invalid values (like -1.5, 0, and STRING).

Sounds about right.

oliverroick · 2017-08-10T09:52:49Z

@seav I added code to make sure accuracyThreshold is only validated on geometry fields and ignored for all other fields.

seav

Just a couple of spelling errors and this should be good to go from my end. :)

seav · 2017-08-11T04:24:05Z

cadasta/questionnaires/tests/test_managers.py

+        assert model.gps_accuracy == 1.5
+
+    def test_create_from_dict_ingnore_accuracy_threshold(self):
+        """For non-geomtrey fields accuracy should be ignored"""


[MINOR] "geomtrey" should be ""geometry".

seav · 2017-08-11T04:26:06Z

cadasta/questionnaires/tests/test_validators.py

@@ -21,6 +21,12 @@ def test_wrong_type(self):
        assert validators.validate_accuracy('Something') is False


+class GpsRelevvantTest(TestCase):


[MINOR] Relevvant should be Relevant.

seav · 2017-08-11T04:37:21Z

(Weird. I check items on the review checklist but they aren't persistent...)

oliverroick · 2017-08-11T08:18:55Z

(Weird. I check items on the review checklist but they aren't persistent...)

It seems to only work for bugs for some reason...

alukach · 2017-08-11T16:13:38Z

Sorry about the delay, will attempt to get to this over the weekend.

alukach · 2017-08-14T16:47:01Z

cadasta/questionnaires/validators.py

+    return json.get('type') in XFORM_GEOM_FIELDS
+
+
+def validate_id_string(json):


nitpick: json doesn't feel like an appropriate name for this field as JSON is a str and these are obviously dict instances. Maybe just data? 🤷‍♂️

I see that this is done elsewhere. Maybe it's our convention and we should leave as is.

oliverroick added PR: migration PR: needs review PR: needs testing labels Aug 8, 2017

oliverroick self-assigned this Aug 8, 2017

oliverroick requested review from seav and alukach August 8, 2017 09:25

seav suggested changes Aug 9, 2017

View reviewed changes

seav suggested changes Aug 11, 2017

View reviewed changes

seav approved these changes Aug 11, 2017

View reviewed changes

alukach reviewed Aug 14, 2017

View reviewed changes

alukach approved these changes Aug 14, 2017

View reviewed changes

oliverroick added PR: approved and removed PR: needs review labels Aug 15, 2017

oliverroick added 10 commits August 15, 2017 23:44

Add accuracy validator

0986857

Refactor Question.TYPE_CHOICES

5248170

Add gps_accuracy field to Question model

cda1289

Read accuracyThreshold from XLSForm

7900154

Add accuracyThreshold to XForm renderer

048e486

Add gps_accuracy to QuestionSerializer

0466179

Add accuracy validator to questionnaire schema validation

ec34d0c

positive now returns val > 0

f711914

Validate accuracyThreshold only for geomtry fields

ddb1bac

Fix typos

4d9fd1e

amplifi force-pushed the feature/gps-accuracy branch from 28a97ba to 4d9fd1e Compare August 15, 2017 22:44

amplifi merged commit 0cf53b3 into master Aug 15, 2017

amplifi deleted the feature/gps-accuracy branch August 15, 2017 23:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds support for accuracyThreshold #1703

Adds support for accuracyThreshold #1703

oliverroick commented Aug 8, 2017

seav left a comment

seav Aug 9, 2017

oliverroick Aug 9, 2017

oliverroick commented Aug 9, 2017

oliverroick commented Aug 10, 2017

seav left a comment

seav Aug 11, 2017

oliverroick Aug 11, 2017

seav Aug 11, 2017

oliverroick Aug 11, 2017

seav commented Aug 11, 2017

oliverroick commented Aug 11, 2017

alukach commented Aug 11, 2017

alukach Aug 14, 2017

alukach Aug 14, 2017

		@@ -21,6 +21,12 @@ def test_wrong_type(self):
		assert validators.validate_accuracy('Something') is False


		class GpsRelevvantTest(TestCase):

		return json.get('type') in XFORM_GEOM_FIELDS


		def validate_id_string(json):

Adds support for accuracyThreshold #1703

Adds support for accuracyThreshold #1703

Conversation

oliverroick commented Aug 8, 2017

Proposed changes in this pull request

When should this PR be merged

Risks

Follow-up actions

Checklist (for reviewing)

General

Functionality

Code

Tests

Security

Documentation

seav left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oliverroick commented Aug 9, 2017

oliverroick commented Aug 10, 2017

seav left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seav commented Aug 11, 2017

oliverroick commented Aug 11, 2017

alukach commented Aug 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment