Fix #935: Conditional attributes not exported during download process

[bjohare]

Proposed changes in this pull request

Fix #935: Conditional attributes not exported during download process

This PR adds support for the export of conditional attributes. It makes use of the SchemaSelectorMixin added in PR #1076.

• Upgraded openpyxl to version 2.4.1
• Refactored the write_items methods of ShapeExporter and XLSExporter to export all attributes
• Refactored the make_download method of XLSExporter to use a streaming WriteOnlyWorksheet which streams rows to the spreadsheet rather than holding all data in memory before writing. This should help with improvements proposed in the Asynchronous Import/Export DR
• Refactored the get_values method of the base Exporter to return a dict of values instead of a list
• Added a method get_conditional_selector to the SchemaSelectorMixin to return the conditional selector for a particular ContentType if one exists

When should this PR be merged

This has been rebased from #1076 so should be merged after this.

Risks

Low risk.

Follow up actions

Reprovision VM's to pull in updates to dependencies.

Checklist (for reviewing)

General

• Is this PR explained thoroughly? All code changes must be accounted for in the PR description.
• Is the PR labeled correctly? It should have the migration label if a new migration is added.
  Is the risk level assessment sufficient? The risks section should contain all risks that might be introduced with the PR and which actions we need to take to mitigate these risks. Possible risks are database migrations, new libraries that need to be installed or changes to deployment scripts.

Functionality

• Are all requirements met? Compare implemented functionality with the requirements specification.
• Does the UI work as expected? There should be no Javascript errors in the console; all resources should load. There should be no unexpected errors. Deliberately try to break the feature to find out if there are corner cases that are not handled.

Code

• Do you fully understand the introduced changes to the code? If not ask for clarification, it might uncover ways to solve a problem in a more elegant and efficient way.
• Does the PR introduce any inefficient database requests? Use the debug server to check for duplicate requests.
• Are all necessary strings marked for translation? All strings that are exposed to users via the UI must be marked for translation.

Tests

• Are there sufficient test cases? Ensure that all components are tested individually; models, forms, and serializers should be tested in isolation even if a test for a view covers these components.
• If this is a bug fix, are tests for the issue in place There must be a test case for the bug to ensure the issue won’t regress. Make sure that the tests break without the new code to fix the issue.

Documentation

• Are changes to the UI documented in the platform docs? If this PR introduces new platform site functionality or changes existing ones, the changes must be documented in the Cadasta Platform Documentation.
• Are changes to the API documented in the API docs? If this PR introduces new API functionality or changes existing ones, the changes must be documented in the API docs.
• Are reusable components documented? If this PR introduces components that are relevant to other developers (for instance a mixin for a view or a generic form) they should be documented in the Wiki.

[oliverroick]

@seav We will put the review for this on hold until #1076 is merged. This branch is currently based on bugfix/#761, which will make it difficult to review the changes that were actually introduced with this branch.

[seav]

@oliverroick: I haven't actually started reviewing this PR since I'm doing my own unsolicited review of PR #1076. But I hope reviewing that PR helps me understand this PR.

[oliverroick]

The export to XLS doesn't seem to work. I had a project with a standard questionnaire and some data. Exporting to shape works, all information is in the SHP and the CSV files. Exporting to XLS however returns empty spreadsheets; both for data and resources. (This could also be a problem with Apple Numbers; I don't have Excel on my machine).

[oliverroick]

@linzjax Eugene is at a conference these days. Would you be able to jump in for the second review so we can get this sorted timely?

[seav]

I did already start reviewing this PR but nothing weird jumped out for me so far. My biggest observation is that I will need to base my search export feature code on this branch. Otherwise I'd have to wrangle with rebasing my work if I based it on the current master.

[seav]

I finished looking at the code changes and aside from a single comment, I didn't find anything wrong. Note also that I didn't test the download files this update is producing in my VM.

[seav]

This method is an example of how I feel the code seems to currently cater to the party entity being the only one right now to have a conditional selector. Maybe it is a good database design to have the conditional selector be a field of the model instead of a foreign field, but I kinda feel iffy with basing the decision on whether we have a conditional selector hinge on the presence of a period on the last schema selector.

Note that this is just an observation and should not be a blocker for the PR's approval.