Config Manager: Add Endpoint EPIC

[Ben-Hodgkiss]

Overview

Please see the Config Manager roadmap and the user stories Murals.

Ticket 1

Spike: Find framework/package to use forms within Python app

Before starting on the Config Manager, we need to do a spike on what form wizard/framework we can use to allow us to use forms with a Python application, rather than node.js

Acceptance Criteria/Tests

• Investigation into what framework can be used
• This framework has been implemented.

Ticket 2

Define ways of working for sources & endpoints.

Before starting on the Config Manager, we need to complete the review of sources and endpoints and agree what (if any) changes need to be made to what is stored in the source file and the endpoint file.

Acceptance Criteria/Tests

• Agreement has been reached on any changes to the source file and endpoint file.
• These changes have been implemented to the pipeline.

Ticket 3

Migrate config manager to AWS

Before starting on the Config Manager, we need to migrate the config manager to AWS so that we can access the Async API. This should start in the dev AWS - we will then deploy outwards through the environments once we are happy with this.

Acceptance Criteria/Tests

• Config Manager has been migrated to AWS for the development environment.
• We can use the Async API for Config Manager applications.

Ticket 4

Async API - look-up files

We need to be able to use the Async API for a new request to generate values from data to be added to look-up files. When given a new endpoint file or URL, the API should be able to put this through the pipeline and assign look-up values for this. These should then be able to go back to the application to be added to the pipeline.

Acceptance Criteria/Tests

• API can take in a file/URL and generate a set of look-ups to be added to the look-up config.
• If no look-ups are to be added, a response is returned that they are all assigned already.
• The API returns the organisation, dataset and reference.

Resourcing

Can’t be completed until Config Manager is migrated to AWS.

Ticket 5

Async API - run duplicate check

We need to be able to use the Async API for a new request to run a duplicate check. The API should accept a file/URL, process this and generate a set of entities to be output. These should be compared to entities already on the relevant sqlite file on the server (potentially using EFS storage). A check should happen between the new data and existing to see if any duplication. If so, these should be output to the application.

Data Management may already have reports for this?

Acceptance Criteria/Tests

• API can take in a file/URL and generate a set of entities from this.
• These entities should be compared to those already on the server
• If no duplicates exist, this should be output back to the application.
• If duplicates exist, details of these should be output back to the application for the user to view and potentially edit.

Resourcing

Can’t be completed until Config Manager is migrated to AWS.

Ticket 6

Async API - find columns for dataset

We need to be able to use the Async API for a new request to find the columns for a specific dataset. These will then be returned to the user to allow them to map columns on their file/URL.

Acceptance Criteria/Tests

• For any given dataset, the API can generate the columns required for that data.
• These columns should be passed back to the application for column mapping to occur.

Resourcing

Can’t be completed until Config Manager is migrated to AWS. We are holding this out of MVP - this functionality should be added later.

Ticket 7

Front-end - 'Create Endpoint' page

An initial page should be created in the Config Manager to accept details of the endpoint being submitted. This should look similar to the wireframe shown in the user stories Mural linked above.

Acceptance Criteria/Tests

• A user can add all relevant endpoint details to the form and submit this.

Resourcing

Can’t be completed until forms framework is decided.

Ticket 8

Front-end - Use API to generate look-ups and display to user

Once the initial page (#3388) is submitted, the API created in #3385 should take the endpoint file/URL and process it to create new look-up values for the user to review.

Acceptance Criteria/Tests

• When a file/URL is submitted, the user is shown the look-up values created for that data.

Ticket 9

Front-end - Use API to check and alter duplicates

Once the look-up value page is submitted, the API runs to check for duplicates. If these exist, they should be returned to the user who can alter the look-ups to correct for these duplicates or edit the current set of look-ups to correct so there is no later need for de-duping. Screens should show to the user to confirm there are duplicates.

Acceptance Criteria/Tests

• If duplicates exist, a screen is shown to allow the user to either exit the process to remove duplicates - or accept the shown duplicates and add regardless.
• If duplicates do not exist, this screen is skipped and the user is taken to the page in Ticket 10.

Ticket 10

Front-end - 'Check source details' page

Once the duplication detection has occurred, the user is taken to a confirmation screen where they are shown final details of their submission. They should be asked to approve these. Once these have been approved, they are shown a confirmation page and the pipeline is updated with the additional endpoint.

Acceptance Criteria/Tests

• When the check details page in confirmed, the endpoint is added to the pipeline.
• When the check details page in confirmed, a confirmation page is shown.
• If the check details page is not confirmed, the endpoint is not added to the pipeline and no confirmation page is shown.

Ticket 11

Front-end - 'Map columns' run API and display on page

Once the API has generated the columns for a specific dataset (#3387), these are shown to the user to allow them to map these columns to the data in their file/URL. These mappings are then returned to the pipeline prior to the look-up values being defined.

Ticket 12

Fill config manager db

Once the config manager database and tables are in place in AWS, we will need to fill the database with current data from S3.

Ticket 13

Nightly task to back-up config into S3

Each night, a task shouold run to back-up the config manager into S3 to ensure we have an up-to-date record in case of system failure.