Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated submission POST based on story, updated pytests #106

Merged
merged 8 commits into from
Mar 18, 2024

Conversation

jcadam14
Copy link
Contributor

Closes #51

  • Updated submissions POST to be in /v1/filing/institutions/{lei}/filings/{period_name}/submissions format
  • Updated POST to get filing obj and check for existence, throwing a 422_UNPROCESSABLE_ENTITY if it doesn't
  • Updated POST to call add_submission with filing.id, submitter id from request.user.id, and filename
  • Updated POST to set the submission.state = SUBMISSION_UPLOADED after saving to S3 and calling update_submission
  • Updated POST to return the SubmissionDTO after call to background thread for validation
  • Updated submission_processor to appropriately use the SubmissionDAO passed in for setting state
  • Updated the submission_processor to call json.loads() for the response from validation so the actual json is stored, not a json string (which is what pandas returns).
  • Updated pytests to match the above functionality.

@jcadam14 jcadam14 self-assigned this Mar 13, 2024
@jcadam14 jcadam14 linked an issue Mar 13, 2024 that may be closed by this pull request
Copy link

github-actions bot commented Mar 13, 2024

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  src/entities/repos
  submission_repo.py 108-111
  src/routers
  filing.py
  src/services
  submission_processor.py
Project Total  

This report was generated by python-coverage-comment-action

Comment on lines 48 to 55
async def validate_submission(lei: str, submission: SubmissionDAO, content: bytes, background_tasks: BackgroundTasks):
df = pd.read_csv(BytesIO(content), dtype=str, na_filter=False)
validator_version = imeta.version("regtech-data-validator")

submission.state = SubmissionState.VALIDATION_IN_PROGRESS
submission.validation_ruleset_version = validator_version
# Set VALIDATION_IN_PROGRESS
await update_submission(
SubmissionDAO(
submitter=submission_id,
state=SubmissionState.VALIDATION_IN_PROGRESS,
validation_ruleset_version=validator_version,
)
)
background_tasks.add_task(validate_and_update_submission, df, lei, submission_id, validator_version)
submission = await update_submission(submission)
background_tasks.add_task(validate_and_update_submission, df, lei, submission)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this needs to be a task to just update the status anymore, once the file's uploaded to storage, we can say validation has started, then this method would just merge with the validate_and_update; don't think we need to have 2 bg tasks. (submission_uploaded will be valid once we have a separate microservice that listens on s3 events to actually do the validation much further down the line, but for now, I think we can just skip it)

@jcadam14 jcadam14 requested a review from lchen-2101 March 14, 2024 16:03
Copy link
Collaborator

@lchen-2101 lchen-2101 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lchen-2101 lchen-2101 merged commit 95cb9c6 into main Mar 18, 2024
3 checks passed
@lchen-2101 lchen-2101 deleted the 51-update-post-endpoint-for-submissions branch March 18, 2024 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update POST endpoint for submissions
2 participants