Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(application-system): using aws-sdkv3 initial implementation #16281

Merged
merged 38 commits into from
Oct 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
6da2f09
porting changes from a larger PR to this smaller one
HjorturJ Oct 2, 2024
b707be9
adding a fix for 1 test suite
HjorturJ Oct 2, 2024
4b043ae
chore: nx format:write update dirty files
andes-it Oct 4, 2024
3411105
Merge branch 'main' into chore/aws-v3-s3-implementation
HjorturJ Oct 7, 2024
efd47cb
adding some insane logger logs to manually test some stuff
HjorturJ Oct 7, 2024
3013d6c
Merge branch 'main' into chore/aws-v3-s3-implementation
HjorturJ Oct 7, 2024
444d5a2
Just testing things manually through an code path I know will get called
HjorturJ Oct 8, 2024
fdf8dd7
Merge branch 'chore/aws-v3-s3-implementation' of github.com:island-is…
HjorturJ Oct 8, 2024
ed256f4
Merge branch 'main' into chore/aws-v3-s3-implementation
HjorturJ Oct 8, 2024
661b570
chore: nx format:write update dirty files
andes-it Oct 8, 2024
619d0c7
editing custom manual test
HjorturJ Oct 8, 2024
e4c0d2c
removing log lines for manual testing purposes
HjorturJ Oct 8, 2024
adf52a2
Merge branch 'main' into chore/aws-v3-s3-implementation
HjorturJ Oct 8, 2024
0fdd473
chore: nx format:write update dirty files
andes-it Oct 8, 2024
04d9d5e
Merge branch 'main' into chore/aws-v3-s3-implementation
HjorturJ Oct 8, 2024
fb4d42a
oops, left 1 test log line in
HjorturJ Oct 8, 2024
f0f26bf
:wqMerge branch 'chore/aws-v3-s3-implementation' of github.com:island…
HjorturJ Oct 8, 2024
dbc153a
chore: nx format:write update dirty files
andes-it Oct 8, 2024
2f11a13
coderabbitai review comments addressed
HjorturJ Oct 9, 2024
b4466fd
Merge branch 'chore/aws-v3-s3-implementation' of github.com:island-is…
HjorturJ Oct 9, 2024
afe39f2
chore: nx format:write update dirty files
andes-it Oct 9, 2024
6ce615f
adding a forgetten ? into a test
HjorturJ Oct 9, 2024
e6f7689
Merge branch 'chore/aws-v3-s3-implementation' of github.com:island-is…
HjorturJ Oct 9, 2024
90dec13
adding attachments to the parental leave e2e test
HjorturJ Oct 10, 2024
151fa96
Merge branch 'main' into chore/aws-v3-s3-implementation
HjorturJ Oct 10, 2024
0d40c82
missed something in shared service I suppose
HjorturJ Oct 10, 2024
467d41a
chore: nx format:write update dirty files
andes-it Oct 10, 2024
e1293f3
merge conflict
HjorturJ Oct 11, 2024
e0f2ff1
fixing PR conflicts
HjorturJ Oct 11, 2024
ceaaa0d
more fixes
HjorturJ Oct 11, 2024
c21747f
finally done fixing yarn lock after main merge
HjorturJ Oct 11, 2024
6659310
Throwing from delete function instead of failing silently
HjorturJ Oct 14, 2024
fd82254
Fxing tests after throwing when deleting s3 object
HjorturJ Oct 14, 2024
95c1941
chore: nx format:write update dirty files
andes-it Oct 14, 2024
bd44c33
Merge branch 'main' into chore/aws-v3-s3-implementation
HjorturJ Oct 14, 2024
fe6c3d8
Removing export of s3 client from aws module
HjorturJ Oct 16, 2024
05e86d2
Merge branch 'chore/aws-v3-s3-implementation' of github.com:island-is…
HjorturJ Oct 16, 2024
7aa62af
Merge branch 'main' into chore/aws-v3-s3-implementation
kodiakhq[bot] Oct 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,7 @@ export class TemplateApiApplicationService extends BaseTemplateApiApplicationSer
const s3key = `${application.id}/${attachmentKey}`
const url = await this.awsService.uploadFile(
buffer,
uploadBucket,
s3key,
{ bucket: uploadBucket, key: s3key },
uploadParameters,
)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -810,10 +810,10 @@ export class EndorsementListService {
await this.uploadFileToS3(fileBuffer, filename, fileType)

// Generate presigned URL with 60 minutes expiration
const url = await this.awsService.getPresignedUrl(
environment.exportsBucketName,
filename,
)
const url = await this.awsService.getPresignedUrl({
bucket: environment.exportsBucketName,
key: filename,
})
return { url }
} catch (error) {
this.logger.error(`Failed to export list ${listId}`, { error })
Expand Down Expand Up @@ -878,8 +878,7 @@ export class EndorsementListService {
try {
await this.awsService.uploadFile(
fileBuffer,
environment.exportsBucketName,
filename,
{ bucket: environment.exportsBucketName, key: filename },
{
ContentType: fileType === 'pdf' ? 'application/pdf' : 'text/csv',
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import {
import { helpers } from '../../../../support/locator-helpers'
import { session } from '../../../../support/session'
import { setupXroadMocks } from './setup-xroad.mocks'
import { createMockPdf, deleteMockPdf } from '../../../../support/utils'

test.use({ baseURL: urls.islandisBaseUrl })

Expand Down Expand Up @@ -323,6 +324,19 @@ test.describe('Parental leave', () => {
name: label(parentalLeaveFormMessages.attachmentScreen.title),
}),
).toBeVisible()

// Upload additional document
createMockPdf()
const fileChooserPromise = page.waitForEvent('filechooser')
await page
.getByRole('button', {
name: label(parentalLeaveFormMessages.fileUpload.attachmentButton),
})
.click()
const filechooser = await fileChooserPromise
await filechooser.setFiles('./mockPdf.pdf')
await page.waitForTimeout(1000)
deleteMockPdf()
await proceed()

// These are your rights
Expand Down
38 changes: 22 additions & 16 deletions libs/application/api/files/src/lib/file.service.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import {
signingModuleConfig,
SigningService,
} from '@island.is/dokobit-signing'
import { AwsService } from '@island.is/nest/aws'
import { AwsModule, AwsService } from '@island.is/nest/aws'
import * as pdf from './pdfGenerators'
import { Application } from '@island.is/application/api/core'
import { ApplicationTypes, PdfTypes } from '@island.is/application/types'
Expand Down Expand Up @@ -115,6 +115,7 @@ describe('FileService', () => {
imports: [
LoggingModule,
SigningModule,
AwsModule,
ConfigModule.forRoot({
isGlobal: true,
load: [
Expand All @@ -124,14 +125,14 @@ describe('FileService', () => {
],
}),
],
providers: [FileService, AwsService],
providers: [FileService],
}).compile()

awsService = module.get(AwsService)

jest
.spyOn(awsService, 'getFile')
.mockImplementation(() => Promise.resolve({ Body: 'body' }))
.spyOn(awsService, 'getFileContent')
.mockImplementation(() => Promise.resolve('body'))

jest.spyOn(awsService, 'fileExists').mockResolvedValue(false)

Expand Down Expand Up @@ -174,16 +175,18 @@ describe('FileService', () => {

expect(awsService.uploadFile).toHaveBeenCalledWith(
Buffer.from('buffer'),
bucket,
fileName,
{ bucket, key: fileName },
{
ContentEncoding: 'base64',
ContentDisposition: 'inline',
ContentType: 'application/pdf',
},
)

expect(awsService.getPresignedUrl).toHaveBeenCalledWith(bucket, fileName)
expect(awsService.getPresignedUrl).toHaveBeenCalledWith({
bucket: bucket,
key: fileName,
})

expect(response).toEqual('url')
})
Expand All @@ -195,9 +198,9 @@ describe('FileService', () => {
PdfTypes.CHILDREN_RESIDENCE_CHANGE,
)

expect(awsService.getFile).toHaveBeenCalledWith(
bucket,
`children-residence-change/${application.id}.pdf`,
expect(awsService.getFileContent).toHaveBeenCalledWith(
{ bucket, key: `children-residence-change/${application.id}.pdf` },
'binary',
)

expect(signingService.requestSignature).toHaveBeenCalledWith(
Expand All @@ -221,8 +224,8 @@ describe('FileService', () => {
const application = createApplication()

jest
.spyOn(awsService, 'getFile')
.mockImplementation(() => Promise.resolve({ Body: '' }))
.spyOn(awsService, 'getFileContent')
.mockImplementation(() => Promise.resolve(''))

const act = async () =>
await service.requestFileSignature(
Expand All @@ -232,9 +235,9 @@ describe('FileService', () => {

await expect(act).rejects.toThrowError(NotFoundException)

expect(awsService.getFile).toHaveBeenCalledWith(
bucket,
`children-residence-change/${applicationId}.pdf`,
expect(awsService.getFileContent).toHaveBeenCalledWith(
{ bucket: bucket, key: `children-residence-change/${applicationId}.pdf` },
'binary',
)

expect(signingService.requestSignature).not.toHaveBeenCalled()
Expand Down Expand Up @@ -268,7 +271,10 @@ describe('FileService', () => {
PdfTypes.CHILDREN_RESIDENCE_CHANGE,
)

expect(awsService.getPresignedUrl).toHaveBeenCalledWith(bucket, fileName)
expect(awsService.getPresignedUrl).toHaveBeenCalledWith({
bucket,
key: fileName,
})
expect(result).toEqual('url')
})

Expand Down
39 changes: 23 additions & 16 deletions libs/application/api/files/src/lib/file.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,16 +54,22 @@ export class FileService {
const fileName = `${BucketTypePrefix[pdfType]}/${application.id}.pdf`
const bucket = this.getBucketName()

if ((await this.awsService.fileExists(bucket, fileName)) === false) {
if (
(await this.awsService.fileExists({ bucket, key: fileName })) === false
) {
const content = await this.createFile(application, pdfType)
await this.awsService.uploadFile(content, bucket, fileName, {
ContentEncoding: 'base64',
ContentDisposition: 'inline',
ContentType: 'application/pdf',
})
await this.awsService.uploadFile(
content,
{ bucket, key: fileName },
{
ContentEncoding: 'base64',
ContentDisposition: 'inline',
ContentType: 'application/pdf',
},
)
}

return await this.awsService.getPresignedUrl(bucket, fileName)
return await this.awsService.getPresignedUrl({ bucket, key: fileName })
}

async uploadSignedFile(
Expand All @@ -79,11 +85,10 @@ export class FileService {
.waitForSignature(DokobitFileName[pdfType], documentToken)
.then(async (file) => {
const s3FileName = `${BucketTypePrefix[pdfType]}/${application.id}.pdf`
await this.awsService.uploadFile(
Buffer.from(file, 'binary'),
await this.awsService.uploadFile(Buffer.from(file, 'binary'), {
bucket,
s3FileName,
)
key: s3FileName,
})
})
.catch((error) => {
if (error.code === DokobitErrorCodes.NoMobileSignature) {
Expand Down Expand Up @@ -130,7 +135,7 @@ export class FileService {

const fileName = `${BucketTypePrefix[pdfType]}/${application.id}.pdf`

return await this.awsService.getPresignedUrl(bucket, fileName)
return await this.awsService.getPresignedUrl({ bucket, key: fileName })
}

private async createFile(application: Application, pdfType: PdfTypes) {
Expand All @@ -148,8 +153,10 @@ export class FileService {
) {
const bucket = this.getBucketName()
const s3FileName = `${BucketTypePrefix[pdfType]}/${application.id}.pdf`
const s3File = await this.awsService.getFile(bucket, s3FileName)
const fileContent = s3File.Body?.toString('binary')
const fileContent = await this.awsService.getFileContent(
{ bucket, key: s3FileName },
'binary',
)

const { phoneNumber, name, title } = this.getSigningOptionsFromApplication(
application,
Expand Down Expand Up @@ -224,7 +231,7 @@ export class FileService {

async getAttachmentPresignedURL(fileName: string) {
const { bucket, key } = AmazonS3URI(fileName)
const url = await this.awsService.getPresignedUrl(bucket, key)
const url = await this.awsService.getPresignedUrl({ bucket, key })
return { url }
}

Expand All @@ -250,7 +257,7 @@ export class FileService {

try {
this.logger.info(`Deleting attachment ${s3key} from bucket ${bucket}`)
await this.awsService.deleteObject(bucket, s3key)
await this.awsService.deleteObject({ bucket, key: s3key })
result = {
...result,
deleted: result.deleted++,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -269,8 +269,7 @@ export class SharedTemplateApiService {
const s3key = `${application.id}/${attachmentKey}`
const url = await this.awsService.uploadFile(
buffer,
uploadBucket,
s3key,
{ bucket: uploadBucket, key: s3key },
uploadParameters,
)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import {
NationalRegistryClientModule,
NationalRegistryClientService,
} from '@island.is/clients/national-registry-v2'
import { AwsModule } from '@island.is/nest/aws'

@Module({
imports: [
Expand All @@ -17,6 +18,7 @@ import {
SmsModule,
ApplicationApiCoreModule,
NationalRegistryClientModule,
AwsModule,
],
providers: [
ChildrenService,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import { Inject, Injectable } from '@nestjs/common'
import { S3 } from 'aws-sdk'

import { AwsService } from '@island.is/nest/aws'
import { getValueViaPath } from '@island.is/application/core'
import {
ADOPTION,
Expand Down Expand Up @@ -102,8 +101,6 @@ interface VMSTError {

@Injectable()
export class ParentalLeaveService extends BaseTemplateApiService {
s3 = new S3()

constructor(
@Inject(LOGGER_PROVIDER) private logger: Logger,
private parentalLeaveApi: ParentalLeaveApi,
Expand All @@ -114,6 +111,7 @@ export class ParentalLeaveService extends BaseTemplateApiService {
private readonly configService: ConfigService<SharedModuleConfig>,
private readonly childrenService: ChildrenService,
private readonly nationalRegistryApi: NationalRegistryClientService,
private readonly awsService: AwsService,
) {
super(ApplicationTypes.PARENTAL_LEAVE)
}
Expand Down Expand Up @@ -384,16 +382,19 @@ export class ParentalLeaveService extends BaseTemplateApiService {
)

const Key = `${application.id}/${filename}`
const file = await this.s3
.getObject({ Bucket: this.config.templateApi.attachmentBucket, Key })
.promise()
const fileContent = file.Body as Buffer
const fileContent = await this.awsService.getFileContent(
{
bucket: this.config.templateApi.attachmentBucket,
key: Key,
},
'base64',
)

if (!fileContent) {
throw new Error('File content was undefined')
}

return fileContent.toString('base64')
return fileContent
HjorturJ marked this conversation as resolved.
Show resolved Hide resolved
} catch (e) {
this.logger.error('Cannot get ' + fileUpload + ' attachment', { e })
throw new Error('Failed to get the ' + fileUpload + ' attachment')
Expand Down
9 changes: 8 additions & 1 deletion libs/nest/aws/src/lib/aws.module.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,16 @@
import { Module } from '@nestjs/common'
import { AwsService } from './aws.service'
import { S3Client } from '@aws-sdk/client-s3'

@Module({
controllers: [],
providers: [AwsService],
providers: [
AwsService,
{
provide: S3Client,
useValue: new S3Client(),
},
],
exports: [AwsService],
})
export class AwsModule {}
Loading