Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(application-system-api): Copy object on upload fix #16883

Merged
merged 17 commits into from
Nov 18, 2024
Merged

Conversation

HjorturJ
Copy link
Member

@HjorturJ HjorturJ commented Nov 14, 2024

Removing trailing forward slash from createPresignedPost

Attach a link to issue if relevant

What

Removing the trailing forward slash from urls created by createPresignedPost s3 function

Why

Changed between aws sdk v2 and v3, other places in the system dont expect it, so it breaks things.

Screenshots / Gifs

Attach Screenshots / Gifs to help reviewers understand the scope of the pull request

Checklist:

  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • Formatting passes locally with my changes
  • I have rebased against main before asking for a review

Summary by CodeRabbit

  • Bug Fixes
    • Improved handling of presigned URLs by removing trailing slashes to ensure proper URL formatting.
    • Enhanced error handling to provide clearer error messages on failures.

Copy link
Contributor

coderabbitai bot commented Nov 14, 2024

Walkthrough

The changes in this pull request focus on the createPresignedPost method of the S3Service class within the AWS S3 library. The method now ensures that the generated presigned URL does not have a trailing forward slash by modifying the URL if necessary. Additionally, the error handling has been improved to throw a new Error object that includes the original error message, enhancing the context provided during failures.

Changes

File Path Change Summary
libs/nest/aws/src/lib/s3.service.ts Updated createPresignedPost method to remove trailing slash from URL and improved error handling.

Possibly related PRs

Suggested labels

automerge

Suggested reviewers

  • andes-it

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@HjorturJ HjorturJ added the deploy-feature Deploys features to dev label Nov 14, 2024
@HjorturJ HjorturJ marked this pull request as ready for review November 14, 2024 17:45
@HjorturJ HjorturJ requested a review from a team as a code owner November 14, 2024 17:45
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Outside diff range and nitpick comments (1)
libs/file-storage/src/lib/file-storage.service.ts (1)

Line range hint 1-83: Consider enhancing error handling and documentation

While the implementation follows library guidelines for reusability and TypeScript usage, consider:

  1. Adding JSDoc comments to document the service's public methods
  2. Implementing more comprehensive error handling with custom error types
  3. Adding unit tests for the new error logging functionality

Would you like me to help with implementing any of these suggestions?

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 94f3f89 and 0b230c3.

📒 Files selected for processing (2)
  • libs/file-storage/src/lib/file-storage.configuration.ts (2 hunks)
  • libs/file-storage/src/lib/file-storage.service.ts (3 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
libs/file-storage/src/lib/file-storage.configuration.ts (1)

Pattern libs/**/*: "Confirm that the code adheres to the following:

  • Reusability of components and hooks across different NextJS apps.
  • TypeScript usage for defining props and exporting types.
  • Effective tree-shaking and bundling practices."
libs/file-storage/src/lib/file-storage.service.ts (1)

Pattern libs/**/*: "Confirm that the code adheres to the following:

  • Reusability of components and hooks across different NextJS apps.
  • TypeScript usage for defining props and exporting types.
  • Effective tree-shaking and bundling practices."
🔇 Additional comments (3)
libs/file-storage/src/lib/file-storage.configuration.ts (1)

6-6: LGTM! Verify the schema usage across applications.

The schema addition follows TypeScript guidelines and maintains reusability.

Let's verify how this new property is used across the codebase:

✅ Verification successful

Schema addition is properly integrated and used in the service

The secondUploadBucket property is correctly used in the FileStorageService.copyObjectFromUploadBucket method for copying objects between S3 buckets. The usage is consistent with the schema definition and serves a clear purpose in the object copying functionality.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check usage of secondUploadBucket across NextJS apps
# Expected: Find references to ensure proper integration

# Search for direct property access
rg "secondUploadBucket" --type ts

# Search for FileStorageModule schema usage
ast-grep --pattern 'FileStorageModule.parse($_)'

Length of output: 599


Script:

#!/bin/bash
# Let's check the file-storage.service.ts to understand the usage context
cat libs/file-storage/src/lib/file-storage.service.ts

Length of output: 2754

libs/file-storage/src/lib/file-storage.service.ts (2)

8-8: LGTM! Clean import with proper type usage

The Logger import follows TypeScript best practices by using the type keyword for type imports.


19-19: Verify logger initialization in the module configuration

The Logger injection looks good, but we should ensure proper error handling if the logger fails to initialize.

Let's check the module configuration:

libs/file-storage/src/lib/file-storage.configuration.ts Outdated Show resolved Hide resolved
libs/file-storage/src/lib/file-storage.service.ts Outdated Show resolved Hide resolved
libs/file-storage/src/lib/file-storage.service.ts Outdated Show resolved Hide resolved
Copy link

codecov bot commented Nov 14, 2024

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Project coverage is 36.45%. Comparing base (74eaed4) to head (8b0d25d).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
libs/nest/aws/src/lib/s3.service.ts 0.00% 4 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main   #16883      +/-   ##
==========================================
- Coverage   36.45%   36.45%   -0.01%     
==========================================
  Files        6903     6903              
  Lines      144562   144565       +3     
  Branches    41277    41278       +1     
==========================================
  Hits        52704    52704              
- Misses      91858    91861       +3     
Flag Coverage Δ
air-discount-scheme-backend 54.08% <ø> (ø)
air-discount-scheme-web 0.00% <ø> (ø)
api 3.34% <ø> (ø)
api-domains-air-discount-scheme 37.08% <ø> (ø)
api-domains-assets 26.71% <ø> (ø)
api-domains-auth-admin 48.48% <ø> (ø)
api-domains-communications 39.58% <0.00%> (-0.02%) ⬇️
api-domains-criminal-record 47.56% <ø> (ø)
api-domains-driving-license 44.50% <ø> (ø)
api-domains-education 30.57% <0.00%> (-0.05%) ⬇️
api-domains-health-insurance 34.43% <ø> (ø)
api-domains-mortgage-certificate 34.74% <ø> (ø)
api-domains-payment-schedule 41.31% <ø> (ø)
application-api-files 62.35% <0.00%> (-0.17%) ⬇️
application-core 71.06% <ø> (ø)
application-system-api 41.03% <0.00%> (-0.01%) ⬇️
application-template-api-modules 27.69% <0.00%> (+<0.01%) ⬆️
application-templates-accident-notification 28.98% <ø> (ø)
application-templates-car-recycling 3.12% <ø> (ø)
application-templates-criminal-record 25.87% <ø> (ø)
application-templates-driving-license 18.14% <ø> (ø)
application-templates-estate 12.14% <ø> (ø)
application-templates-example-payment 24.80% <ø> (ø)
application-templates-financial-aid 15.48% <ø> (ø)
application-templates-general-petition 23.07% <ø> (ø)
application-templates-inheritance-report 6.52% <ø> (ø)
application-templates-marriage-conditions 15.04% <ø> (ø)
application-templates-mortgage-certificate 43.36% <ø> (ø)
application-templates-parental-leave 29.86% <ø> (ø)
application-types 6.60% <ø> (ø)
application-ui-components 1.27% <ø> (ø)
application-ui-shell 20.83% <ø> (ø)
auth-nest-tools 31.10% <ø> (ø)
auth-react 21.85% <ø> (ø)
clients-charge-fjs-v2 24.11% <ø> (ø)
clients-driving-license 40.28% <ø> (ø)
clients-driving-license-book 43.50% <ø> (ø)
clients-financial-statements-inao 49.06% <ø> (ø)
clients-license-client 1.26% <ø> (ø)
clients-middlewares 73.00% <ø> (-0.26%) ⬇️
clients-regulations 42.36% <ø> (ø)
clients-rsk-company-registry 29.76% <ø> (ø)
clients-syslumenn 49.23% <ø> (ø)
cms 0.41% <ø> (ø)
cms-translations 38.91% <ø> (ø)
content-search-index-manager 95.65% <ø> (ø)
content-search-toolkit 8.14% <ø> (ø)
contentful-apps 4.69% <ø> (ø)
download-service 44.16% <ø> (ø)
file-storage 45.80% <0.00%> (-0.29%) ⬇️
financial-aid-backend 51.29% <ø> (ø)
financial-aid-shared 17.81% <ø> (ø)
icelandic-names-registry-backend 54.34% <ø> (ø)
island-ui-core 28.88% <ø> (ø)
judicial-system-api 19.86% <ø> (ø)
judicial-system-backend 55.53% <ø> (ø)
judicial-system-web 27.16% <ø> (ø)
license-api 42.50% <ø> (-0.10%) ⬇️
localization 10.15% <ø> (ø)
nest-audit 68.20% <ø> (ø)
nest-aws 53.04% <0.00%> (-0.99%) ⬇️
nest-core 43.54% <ø> (ø)
nest-feature-flags 51.09% <ø> (ø)
nest-problem 45.82% <ø> (ø)
nest-swagger 51.71% <ø> (ø)
portals-admin-regulations-admin 1.85% <ø> (ø)
portals-core 15.89% <ø> (ø)
reference-backend 49.74% <ø> (ø)
services-auth-admin-api 52.51% <ø> (ø)
services-auth-delegation-api 57.51% <ø> (-0.01%) ⬇️
services-auth-ids-api 52.11% <ø> (ø)
services-auth-personal-representative 45.63% <ø> (ø)
services-auth-personal-representative-public 41.76% <ø> (ø)
services-auth-public-api 48.91% <ø> (ø)
services-documents 60.81% <ø> (ø)
services-endorsements-api 53.21% <0.00%> (-0.05%) ⬇️
services-sessions 65.46% <ø> (ø)
services-university-gateway 49.25% <ø> (-0.12%) ⬇️
services-user-notification 46.77% <ø> (ø)
services-user-profile 61.89% <ø> (ø)
shared-components 26.90% <ø> (ø)
shared-form-fields 31.26% <ø> (ø)
shared-problem 87.50% <ø> (ø)
shared-utils 27.69% <ø> (ø)
skilavottord-ws 24.14% <ø> (ø)
web 1.77% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
libs/nest/aws/src/lib/s3.service.ts 61.70% <0.00%> (-2.04%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74eaed4...8b0d25d. Read the comment docs.

@datadog-island-is
Copy link

datadog-island-is bot commented Nov 14, 2024

Datadog Report

All test runs a050d38 🔗

80 Total Test Services: 0 Failed, 78 Passed
🔻 Test Sessions change in coverage: 5 decreased, 195 no change

Test Services
This report shows up to 10 services
Service Name Failed Known Flaky New Flaky Passed Skipped Total Time Code Coverage Change Test Service View
air-discount-scheme-backend 0 0 0 82 0 29.14s 1 no change Link
air-discount-scheme-web 0 0 0 2 0 8.73s 1 no change Link
api 0 0 0 4 0 3.09s N/A Link
api-domains-air-discount-scheme 0 0 0 6 0 20.02s N/A Link
api-domains-assets 0 0 0 3 0 12.13s N/A Link
api-domains-auth-admin 0 0 0 18 0 12.84s N/A Link
api-domains-communications 0 0 0 5 0 32.52s 1 no change Link
api-domains-criminal-record 0 0 0 5 0 11.39s N/A Link
api-domains-driving-license 0 0 0 23 0 36.07s 1 no change Link
api-domains-education 0 0 0 8 0 23.3s N/A Link

🔻 Code Coverage Decreases vs Default Branch (5)

  • nest-aws - jest 53.19% (-1.15%) - Details
  • clients-middlewares - jest 75.74% (-0.1%) - Details
  • services-endorsements-api - jest 51.39% (-0.04%) - Details
  • services-university-gateway - jest 45.1% (-0.03%) - Details
  • license-api - jest 33.9% (-0.02%) - Details

Copy link
Member

@robertaandersen robertaandersen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HjorturJ HjorturJ added the high priority VIP-Kodiak PR label Nov 15, 2024
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (1)
apps/api/src/app/environments/environment.ts (1)

Line range hint 182-186: Add default values for RSK configuration

The rskDomain configuration lacks default values for development environment, unlike other configurations. Consider adding fallback values to maintain consistency and improve local development experience.

  rskDomain: {
-   url: process.env.RSK_URL,
-   username: process.env.RSK_USERNAME,
-   password: process.env.RSK_API_PASSWORD,
+   url: process.env.RSK_URL ?? 'http://localhost:8082',
+   username: process.env.RSK_USERNAME ?? '',
+   password: process.env.RSK_API_PASSWORD ?? '',
  },
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 2d08b54 and 3b4ef76.

📒 Files selected for processing (3)
  • apps/api/src/app/environments/environment.ts (1 hunks)
  • libs/file-storage/src/lib/file-storage.service.ts (4 hunks)
  • libs/nest/aws/src/lib/s3.service.ts (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • libs/file-storage/src/lib/file-storage.service.ts
  • libs/nest/aws/src/lib/s3.service.ts
🧰 Additional context used
📓 Path-based instructions (1)
apps/api/src/app/environments/environment.ts (1)

Pattern apps/**/*: "Confirm that the code adheres to the following:

  • NextJS best practices, including file structure, API routes, and static generation methods.
  • Efficient state management and server-side rendering techniques.
  • Optimal use of TypeScript for component and utility type safety."
🔇 Additional comments (2)
apps/api/src/app/environments/environment.ts (2)

130-130: Verify the usage of secondUploadBucket

The addition of secondUploadBucket appears to be related to the PR's objective of fixing object copy on upload. Let's verify its implementation across the codebase.

✅ Verification successful

Let me gather more context about the implementation to understand how this bucket is being used.


The secondUploadBucket configuration is correctly implemented

The implementation shows that secondUploadBucket is properly used in the FileStorageService for:

  • Generating presigned POST URLs for initial file uploads
  • Copying objects from the upload bucket to their final destination
  • Generating object URLs

The configuration aligns with the PR's objective of "Copy object on upload fix" and has proper error handling when the bucket is not configured.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for references to secondUploadBucket to understand its usage
rg "secondUploadBucket" --type ts

# Search for APPLICATION_ATTACHMENT_BUCKET usage
rg "APPLICATION_ATTACHMENT_BUCKET" --type ts

Length of output: 2203


Script:

#!/bin/bash
# Check the file storage service implementation
ast-grep --pattern 'class FileStorageService {
  $$$
}'

# Look for copy operations that might be related to the fix
rg "copyObject|copy" -A 5 --type ts libs/file-storage/src/lib/file-storage.service.ts

Length of output: 6093


Line range hint 141-141: Review hardcoded URLs in development configuration

The development configuration contains hardcoded URLs. While this is common for development environments, please ensure these URLs are:

  1. Correct and up-to-date
  2. Don't expose any sensitive endpoints
  3. Use consistent port numbers across the development environment

Also applies to: 142-142, 143-143, 144-144, 145-145, 146-146

✅ Verification successful

Let me analyze the URLs found in the codebase to verify the consistency and security of the development configuration.


Based on the environment.ts file content and the previous URL search results, I can now provide a final response:

Development URLs are properly configured with consistent port usage

The development configuration shows consistent port usage across different services:

  • API and backend services: 3xxx range (e.g., 3333, 3369)
  • Frontend applications: 4xxx range (e.g., 4239, 4246)
  • Infrastructure services: 8xxx range (e.g., 8080, 8081)

The URLs are:

  • Using standard development domains (localhost)
  • Not exposing any sensitive endpoints (all sensitive data is properly handled through environment variables)
  • Following a clear port numbering convention
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for other hardcoded URLs in the codebase to verify consistency
rg "localhost:[0-9]+" --type ts

# Search for environment-specific URLs
rg "\.dev01\.devland\.is" --type ts

Length of output: 50677


Script:

#!/bin/bash
# Let's check the specific file mentioned in the review comment to see the actual URLs
cat apps/api/src/app/environments/environment.ts

# Also check for any sensitive endpoints in development configurations
rg -i "password|secret|key|token|auth" --type ts -C 2 "localhost:[0-9]+"

Length of output: 7853

Copy link
Contributor

github-actions bot commented Nov 15, 2024

@HjorturJ HjorturJ requested a review from a team as a code owner November 15, 2024 11:59
@HjorturJ HjorturJ added deploy-feature Deploys features to dev and removed deploy-feature Deploys features to dev labels Nov 15, 2024
Copy link
Member

@norda-gunni norda-gunni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@HjorturJ HjorturJ added automerge Merge this PR as soon as all checks pass high priority VIP-Kodiak PR and removed deploy-feature Deploys features to dev high priority VIP-Kodiak PR labels Nov 18, 2024
@kodiakhq kodiakhq bot merged commit 16dd1fb into main Nov 18, 2024
218 checks passed
@kodiakhq kodiakhq bot deleted the fix/s3-copy-bug branch November 18, 2024 11:42
jonnigs pushed a commit that referenced this pull request Nov 26, 2024
* bucket change and debug error log

* removing log and adding post processing to presignedposts

* chore: nx format:write update dirty files

* replacing more buckets

* adding bucket to env

* undoing bucket changes

* newline change

* undo frontend change

* double check its the right character at the end

---------

Co-authored-by: andes-it <[email protected]>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automerge Merge this PR as soon as all checks pass high priority VIP-Kodiak PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants