Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: convert CsvMergedHeadersGenerator to typescript #2080

Merged
merged 35 commits into from
Jun 22, 2021
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
0c082d5
refactor: convert response classes to typescript and change response …
chowyiyin Jun 4, 2021
3d8c1d2
refactor: rearrange types
chowyiyin Jun 4, 2021
bf5cb22
refactor: convert CsvMergedHeadersGenerator to typescript
chowyiyin Jun 4, 2021
5483a87
refactor: edit export and import statements for CsvMergedHeadersGener…
chowyiyin Jun 7, 2021
2c0168c
fix: edit import statement of generator in submissions.client.factory
chowyiyin Jun 7, 2021
2573fea
refactor: edit response types
chowyiyin Jun 8, 2021
cda7b2e
refactor: remove typing in jsdocs and edit types
chowyiyin Jun 8, 2021
d11f18f
refactor: export response subclasses input types
chowyiyin Jun 8, 2021
fdc9549
refactor: add ErrorResponse as default for response types
chowyiyin Jun 8, 2021
9455788
fix: resolve merge conflicts
chowyiyin Jun 8, 2021
1f06592
refactor: shift hasProps to shared
chowyiyin Jun 9, 2021
107e41f
refactor: athrow error instead of returning ErrorResponse and edited …
chowyiyin Jun 9, 2021
3dbe803
refactor: change interface to type
chowyiyin Jun 9, 2021
2e54dd6
fix: fix incorrect import statement
chowyiyin Jun 9, 2021
c99934f
test: remove single header row test
chowyiyin Jun 9, 2021
fdb303a
fix: fix bug caused by fieldtype guard and if else condition
chowyiyin Jun 9, 2021
bd3f94d
refactor: remove console.log
chowyiyin Jun 10, 2021
ba1cd6a
refactor: convert extractAnswer to private function
chowyiyin Jun 10, 2021
38eaf6b
docs: add jsdocs for response factory
chowyiyin Jun 10, 2021
1659012
docs: edit jsdocs for addRecord
chowyiyin Jun 10, 2021
2b8501f
refactor: extract acheck for answer array
chowyiyin Jun 10, 2021
12afa33
refactor: revert extractAnswer to non-private function
chowyiyin Jun 10, 2021
33fe947
refactor: remove repeated check for type Array
chowyiyin Jun 14, 2021
cba1116
refactor: create SubmissionRecord type
chowyiyin Jun 17, 2021
c6df4a5
refactor: rename ccsv generator in submissions client factory
chowyiyin Jun 17, 2021
85ae863
refactor: make _data field private
chowyiyin Jun 17, 2021
c0dafeb
test: rrestore deleted test and edit expected unprocessed record
chowyiyin Jun 17, 2021
ba1179e
test: remove repeated test
chowyiyin Jun 17, 2021
b8b1d02
refactor: use private modifier in typescript classes
chowyiyin Jun 17, 2021
9b0b79e
test: add tests for error cases
chowyiyin Jun 17, 2021
f6c001e
refactor: add private modifier to date comparator
chowyiyin Jun 17, 2021
8140ffa
refactor: remove underscore from private field
chowyiyin Jun 17, 2021
67db42f
refactor: extract type in csvmhgenerator
chowyiyin Jun 22, 2021
b90f99c
refactor: replace strings with basic field enums
chowyiyin Jun 22, 2021
61a49a4
fix: fix import statement
chowyiyin Jun 22, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/app/modules/bounce/bounce.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
ResultAsync,
} from 'neverthrow'

import { hasProp } from '../../../shared/util/has-prop'
import {
BounceType,
IBounceSchema,
Expand All @@ -25,7 +26,6 @@ import { EMAIL_HEADERS, EmailType } from '../../services/mail/mail.constants'
import MailService from '../../services/mail/mail.service'
import { SmsFactory } from '../../services/sms/sms.factory'
import { transformMongoError } from '../../utils/handle-mongo-error'
import { hasProp } from '../../utils/has-prop'
import { PossibleDatabaseError } from '../core/core.errors'
import { getCollabEmailsWithPermission } from '../form/form.utils'
import * as UserService from '../user/user.service'
Expand Down
2 changes: 1 addition & 1 deletion src/app/modules/myinfo/myinfo.util.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { err, ok, Result } from 'neverthrow'
import { v4 as uuidv4, validate as validateUUID } from 'uuid'

import { types as myInfoTypes } from '../../../shared/resources/myinfo'
import { hasProp } from '../../../shared/util/has-prop'
import {
AuthType,
BasicField,
Expand All @@ -16,7 +17,6 @@ import {
MapRouteError,
} from '../../../types'
import { createLoggerWithLabel } from '../../config/logger'
import { hasProp } from '../../utils/has-prop'
import { DatabaseError, MissingFeatureError } from '../core/core.errors'
import {
AuthTypeMismatchError,
Expand Down
2 changes: 1 addition & 1 deletion src/app/modules/spcp/spcp.util.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import crypto from 'crypto'
import { StatusCodes } from 'http-status-codes'
import { err, ok, Result } from 'neverthrow'

import { hasProp } from '../../../shared/util/has-prop'
import {
AuthType,
BasicField,
Expand All @@ -11,7 +12,6 @@ import {
SPCPFieldTitle,
} from '../../../types'
import { createLoggerWithLabel } from '../../config/logger'
import { hasProp } from '../../utils/has-prop'
import { MissingFeatureError } from '../core/core.errors'
import {
AuthTypeMismatchError,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,21 +1,38 @@
const moment = require('moment-timezone')
const keyBy = require('lodash/keyBy')
const { CsvGenerator } = require('./CsvGenerator')
const { getResponseInstance } = require('./response-factory')

/**
* @typedef {{
* _id: string,
* question: string,
* answer?: string,
* answerArray?: string[],
* fieldType: string,
* isHeader?: boolean,
* }} DisplayedResponse
*/

class CsvMergedHeadersGenerator extends CsvGenerator {
constructor(expectedNumberOfRecords, numOfMetaDataRows) {
import keyBy from 'lodash/keyBy'
import moment from 'moment-timezone'

import { DisplayedResponseWithoutAnswer } from '../../../../types/response'

import { Response } from './csv-response-classes'
import { CsvGenerator } from './CsvGenerator'
import { getResponseInstance } from './response-factory'

type UnprocessedRecord = {
created: string
submissionId: string
record: { [fieldId: string]: Response }
chowyiyin marked this conversation as resolved.
Show resolved Hide resolved
}

type SubmissionRecord = {
record: DisplayedResponseWithoutAnswer[]
created: string
submissionId: string
}

export class CsvMergedHeadersGenerator extends CsvGenerator {
hasBeenProcessed: boolean
hasBeenSorted: boolean
fieldIdToQuestion: Map<
string,
{
created: string
question: string
}
>
fieldIdToNumCols: Record<string, number>
unprocessed: UnprocessedRecord[]

constructor(expectedNumberOfRecords: number, numOfMetaDataRows: number) {
super(expectedNumberOfRecords, numOfMetaDataRows)

this.hasBeenProcessed = false
Expand All @@ -28,25 +45,25 @@ class CsvMergedHeadersGenerator extends CsvGenerator {
/**
* Returns current length of CSV file excluding header and meta-data
*/
length() {
length(): number {
return this.unprocessed.length
}

/**
*
* @param {Object} decryptedContent
* @param {DisplayedResponse[]} decryptedContent.record
* @param {string} decryptedContent.created
* @param {string} decryptedContent.submissionId
* Extracts information from input record, rearranges record and then adds an UnprocessedRecord to this.unprocessed
* @param decryptedContent
* @param decryptedContent.record
* @param decryptedContent.created
* @param decryptedContent.submissionId
* @throws Error when trying to convert record into a response instance. Should be caught in submissions client factory.
*/
addRecord({ record, created, submissionId }) {
addRecord({ record, created, submissionId }: SubmissionRecord): void {
// First pass, create object with { [fieldId]: question } from
// decryptedContent to get all the questions.
const fieldRecords = record.map((content) => {
const fieldRecord = getResponseInstance(content)
chowyiyin marked this conversation as resolved.
Show resolved Hide resolved
if (!fieldRecord.isHeader) {
const currentMapping = this.fieldIdToQuestion.get(fieldRecord.id)

// Only set new mapping if it does not exist or this record is a later
// submission.
// Might need to differentiate the question headers if we allow
Expand Down Expand Up @@ -77,28 +94,16 @@ class CsvMergedHeadersGenerator extends CsvGenerator {
})
}

/**
* Extracts the string representation from an unprocessed record.
* @param {Object} unprocessedRecord
* @param {string} fieldId
* @returns {string}
*/
_extractAnswer(unprocessedRecord, fieldId, colIndex) {
const fieldRecord = unprocessedRecord[fieldId]
if (!fieldRecord) return ''
return fieldRecord.getAnswer(colIndex)
}

/**
* Process the unprocessed records by creating the correct headers and
* assigning each answer to their respective locations in each response row in
* the csv data.
*/
process() {
process(): void {
if (this.hasBeenProcessed) return

// Create a header row in CSV using the fieldIdToQuestion map.
let headers = ['Reference number', 'Timestamp']
const headers = ['Reference number', 'Timestamp']
this.fieldIdToQuestion.forEach((value, fieldId) => {
for (let i = 0; i < this.fieldIdToNumCols[fieldId]; i++) {
headers.push(value.question)
Expand All @@ -109,26 +114,44 @@ class CsvMergedHeadersGenerator extends CsvGenerator {
// Craft a new csv row for each unprocessed record
// O(qn), where q = number of unique questions, n = number of submissions.
this.unprocessed.forEach((up) => {
let createdAt = moment(up.created).tz('Asia/Singapore')
createdAt = createdAt.isValid()
const createdAt = moment(up.created).tz('Asia/Singapore')
const formattedDate = createdAt.isValid()
? createdAt.format('DD MMM YYYY hh:mm:ss A')
: createdAt
let row = [up.submissionId, createdAt]
for (let [fieldId] of this.fieldIdToQuestion) {
: createdAt.toString() // just convert to string if given date is not valid
const row = [up.submissionId, formattedDate]

this.fieldIdToQuestion.forEach((_question, fieldId) => {
const numCols = this.fieldIdToNumCols[fieldId]
for (let colIndex = 0; colIndex < numCols; colIndex++) {
row.push(this._extractAnswer(up.record, fieldId, colIndex))
}
}
})
this.addLine(row)
})
this.hasBeenProcessed = true
}

/**
* Extracts the string representation from a field response
* @param unprocessedRecord
* @param fieldId
* @param colIndex
* @returns string representation of unprocessed record
*/
private _extractAnswer(
unprocessedRecord: { [fieldId: string]: Response },
fieldId: string,
colIndex: number,
): string {
const fieldRecord = unprocessedRecord[fieldId]
if (!fieldRecord) return ''
return fieldRecord.getAnswer(colIndex)
}

/**
* Sorts unprocessed records from oldest to newest
*/
sort() {
sort(): void {
if (this.hasBeenSorted) return
this.unprocessed.sort((a, b) => this._dateComparator(a.created, b.created))
this.hasBeenSorted = true
Expand All @@ -138,8 +161,8 @@ class CsvMergedHeadersGenerator extends CsvGenerator {
* Add meta-data as first three rows of the CSV. If there is already meta-data
* added, it will be replaced by the latest counts.
*/
addMetaDataFromSubmission(errorCount, unverifiedCount) {
let metaDataRows = [
addMetaDataFromSubmission(errorCount: number, unverifiedCount: number): void {
const metaDataRows = [
['Expected total responses', this.expectedNumberOfRecords],
['Success count', this.length()],
['Error count', errorCount],
Expand All @@ -151,20 +174,20 @@ class CsvMergedHeadersGenerator extends CsvGenerator {

/**
* Main method to call to retrieve a downloadable csv.
* @param {string} filename
* @param filename name of csv file
*/
downloadCsv(filename) {
downloadCsv(filename: string): void {
this.sort()
this.process()
this.triggerFileDownload(filename)
}

/**
* Comparator for dates
* @param string firstDate
* @param string secondDate
* @param firstDate
* @param secondDate
*/
_dateComparator(firstDate, secondDate) {
private _dateComparator(firstDate: string, secondDate: string): number {
// cast to Asia/Singapore to ensure both dates are of the same timezone
const first = moment(firstDate).tz('Asia/Singapore')
const second = moment(secondDate).tz('Asia/Singapore')
Expand All @@ -178,5 +201,3 @@ class CsvMergedHeadersGenerator extends CsvGenerator {
}
}
}

module.exports = CsvMergedHeadersGenerator

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
import { ArrayResponse } from '../../../../../types/response'

import { Response } from './Response.class'

export class ArrayAnswerResponse extends Response {
private response: ArrayResponse

constructor(responseData: ArrayResponse) {
super(responseData)
this.response = responseData
}

getAnswer(): string {
return this.response.answerArray.join(';')
}

get numCols(): number {
return 1
}
}

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import { DisplayedResponseWithoutAnswer } from '../../../../../types/response'

export abstract class Response {
private data: DisplayedResponseWithoutAnswer

constructor(responseData: DisplayedResponseWithoutAnswer) {
this.data = responseData
}

get id(): string {
return this.data._id
}

/**
* Gets the CSV header.
* @returns {string}
*/
get question(): string {
return this.data.question
}

get isHeader(): boolean {
return this.data.isHeader ?? false
}

abstract get numCols(): number

abstract getAnswer(colIndex?: number): string
}

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
import { SingleResponse } from '../../../../../types/response'

import { Response } from './Response.class'

export class SingleAnswerResponse extends Response {
private response: SingleResponse

constructor(responseData: SingleResponse) {
super(responseData)
this.response = responseData
}

getAnswer(): string {
return this.response.answer
}

get numCols(): number {
return 1
}
}

This file was deleted.

Loading