Fixes #1575, #1804 -- Increase max_length for fields

[oliverroick]

Proposed changes in this pull request

• Fixes Relevant conditions in XLSForms are capped to 100 chars #1804: Increase the max_length of Question.relevant and QuestionGroup.relevant to 300 to accomodate for longer, more complex relevant clauses.
• Fixes Custom values (location_type, tenure_type) cannot have more than 10 chars - Error should be raised when uploading the xlsform questionnaire #1575: Incrases the max_length of SpatialUnit.type and TenureRelationship.tenure_type to 100 to match maximimum length allowed for custom relationship and location types.

When should this PR be merged

After release 1.13.0 is out.

Risks

None

Follow-up actions

We should improve error messaging for questionnaire uploads to notify users that the length for some fields is capped.
API docs may also need to be updated.

Checklist (for reviewing)

General

Is this PR explained thoroughly? All code changes must be accounted for in the PR description.

• Review 1
• Review 2

Is the PR labeled correctly? It should have the migration label if a new migration is added.

• Review 1
• Review 2

Is the risk level assessment sufficient? The risks section should contain all risks that might be introduced with the PR and which actions we need to take to mitigate these risks. Possible risks are database migrations, new libraries that need to be installed or changes to deployment scripts.

• Review 1
• Review 2

Functionality

Are all requirements met? Compare implemented functionality with the requirements specification.

• Review 1
• Review 2

Does the UI work as expected? There should be no Javascript errors in the console; all resources should load. There should be no unexpected errors. Deliberately try to break the feature to find out if there are corner cases that are not handled.

• Review 1
• Review 2

Code

Do you fully understand the introduced changes to the code? If not ask for clarification, it might uncover ways to solve a problem in a more elegant and efficient way.

• Review 1
• Review 2

Does the PR introduce any inefficient database requests? Use the debug server to check for duplicate requests.

• Review 1
• Review 2

Are all necessary strings marked for translation? All strings that are exposed to users via the UI must be marked for translation.

• Review 1
• Review 2

Is the code documented sufficiently? Large and complex classes, functions or methods must be annotated with comments following our code-style guidelines.

• Review 1
• Review 2

Has the scalability of this change been evaluated?

• Review 1
• Review 2

Is there a maintenance plan in place?

• Review 1
• Review 2

Tests

Are there sufficient test cases? Ensure that all components are tested individually; models, forms, and serializers should be tested in isolation even if a test for a view covers these components.

• Review 1
• Review 2

If this is a bug fix, are tests for the issue in place? There must be a test case for the bug to ensure the issue won’t regress. Make sure that the tests break without the new code to fix the issue.

• Review 1
• Review 2

If this is a new feature or a significant change to an existing feature? has the manual testing spreadsheet been updated with instructions for manual testing?

• Review 1
• Review 2

Security

Confirm this PR doesn't commit any keys, passwords, tokens, usernames, or other secrets.

• Review 1
• Review 2

Are all UI and API inputs run through forms or serializers?

• Review 1
• Review 2

Are all external inputs validated and sanitized appropriately?

• Review 1
• Review 2

Does all branching logic have a default case?

• Review 1
• Review 2

Does this solution handle outliers and edge cases gracefully?

• Review 1
• Review 2

Are all external communications secured and restricted to SSL?

• Review 1
• Review 2

Documentation

Are changes to the UI documented in the platform docs? If this PR introduces new platform site functionality or changes existing ones, the changes must be documented in the Cadasta Platform Documentation.

• Review 1
• Review 2

Are changes to the API documented in the API docs? If this PR introduces new API functionality or changes existing ones, the changes must be documented in the API docs.

• Review 1
• Review 2

Are reusable components documented? If this PR introduces components that are relevant to other developers (for instance a mixin for a view or a generic form) they should be documented in the Wiki.

• Review 1
• Review 2

[alukach]

@oliverroick Any reason why we wouldn't want the .relevant fields to have unlimited length by using a TextField?

Additionally, regarding risks, can you describe the effects on a Postgres table for resizing the varchar tables? I believe that this will cause the DBs to be rewritten, which would lock the DB and cause downtime during deployment.

[oliverroick]

Any reason why we wouldn't want the .relevant fields to have unlimited length by using a TextField?

I always thought that using text instead of varchar has performance implications but apparently, that's not the case in Postgres (see the box a bit down on the page).

It makes sense to change relevant to a TextField. I'll add the changes.

---

Additionally, regarding risks, can you describe the effects on a Postgres table for resizing the varchar tables? I believe that this will cause the DBs to be rewritten, which would lock the DB and cause downtime during deployment.

Quoting the Postgres docs:

Adding a column with a DEFAULT clause or changing the type of an existing column will require the entire table and its indexes to be rewritten. As an exception when changing the type of an existing column, if the USING clause does not change the column contents and the old type is either binary coercible to the new type or an unconstrained domain over the new type, a table rewrite is not needed; but any indexes on the affected columns must still be rebuilt.

To me, it reads that converting from varchar to text will not cause a rewrite of the table. There's also no index on the relevant column.

Changing the max_length for TenureRelationship.tenure_type and SpatialUnit.typewill result in a rewrite of the table and some downtime on the server. We should think about how to mitigate that. We most straightforward way would probably be to take the site down for a few minutes, but that would also require that we inform users beforehand.

[seav]

Just three unchanged migration filenames.

[seav]

This migration file needs to be renamed.

[seav]

Same here.

[alukach]

We should think about how to mitigate that. We most straightforward way would probably be to take the site down for a few minutes, but that would also require that we inform users beforehand.

Yeah, or at the very least have a fallback static page saying "Sorry, we are currently performing maintenance, check back shortly..."

So, is that in scope for this PR or not? I'm unsure.

[oliverroick]

@seav @alukach Addressed your comments.

So, is that in scope for this PR or not? I'm unsure.

I'd say it's out of scope for this PR but it needs to be addressed before the changes are shipped. I'll have a look if we can include this into the Ansible scripts.

[seav]

Sorry. This third migration file also needs renaming. 😬

[oliverroick]

👍