Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BACK-2540] Print reports #43

Merged
merged 64 commits into from
Sep 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
1992f6c
wip - initial creation of report handler
jh-bate Jun 21, 2023
588b606
move common code and create lib
jh-bate Jun 21, 2023
b305591
lint cleanup
jh-bate Jun 21, 2023
c4f88dc
move log to utils
jh-bate Jun 21, 2023
2e8e811
notes
jh-bate Jun 22, 2023
d0f3405
extreme WIP for SSR PDF gen
krystophv Jul 4, 2023
316f0ec
move to report utils
jh-bate Jul 5, 2023
4dc6d2b
update node version
jh-bate Jul 5, 2023
b0604d8
general utility functions
jh-bate Jul 5, 2023
03101c6
remove console log statements
jh-bate Jul 5, 2023
cbfeaea
addition of get endpoint and userReportsHandler
jh-bate Jul 5, 2023
712f501
update yarn lock
jh-bate Jul 5, 2023
823132d
cleanup
jh-bate Jul 5, 2023
e74995f
cleanup
jh-bate Jul 5, 2023
bf8c1aa
small fixes
krystophv Jul 5, 2023
bb0c73a
updates to process all report types
jh-bate Jul 6, 2023
7174a0a
include profile dob and fullname
jh-bate Jul 6, 2023
e253cea
minor logging and error handling cleanup
krystophv Jul 6, 2023
24844b6
ignore pre-built file for linting
krystophv Jul 6, 2023
a88f2bb
common report code for get or post handler to call
jh-bate Jul 11, 2023
29bafc0
basic tests
jh-bate Jul 11, 2023
2c5dbea
unique counter names, post body params
jh-bate Jul 11, 2023
1ff371e
test for individual data types
jh-bate Jul 11, 2023
4c368a2
Merge pull request #46 from tidepool-org/generation_post_tests
krystophv Jul 11, 2023
cb449f2
Merge pull request #47 from tidepool-org/pdf_generation_jhb
krystophv Jul 11, 2023
36d398b
POST will use standard session header
krystophv Jul 11, 2023
97fb771
remove static pdfkit, update viz, tweak require/export
krystophv Jul 11, 2023
9ce8d0a
add json body parser for POST
krystophv Jul 11, 2023
61fdb2f
default bgLog 30 days, wrap fs output in debug env var
krystophv Jul 11, 2023
d4cbbb3
test before build
krystophv Jul 11, 2023
db4259a
more testing of internal functions
jh-bate Jul 12, 2023
5922c3b
remove un-needed data
jh-bate Jul 12, 2023
b480f57
pass dates, if given, into queries
jh-bate Jul 12, 2023
df42ef0
ensure bgLogs get 30 days of data at a minimum
jh-bate Jul 13, 2023
8366bfb
Merge pull request #48 from tidepool-org/report_dates
krystophv Jul 17, 2023
ebae399
pass through errors
krystophv Jul 17, 2023
4247ba7
bump viz
krystophv Jul 17, 2023
640a585
initial setup for report creation in class
jh-bate Jul 19, 2023
f60f1d3
dep updates
jh-bate Jul 19, 2023
8677899
update deps that were outdated
jh-bate Jul 19, 2023
3c23fe0
update lock
jh-bate Jul 19, 2023
9919bb5
undo formatting
jh-bate Jul 19, 2023
2eda4cd
dep updates
jh-bate Jul 19, 2023
fbcb168
remove qs and switch to AbortController
jh-bate Jul 19, 2023
e2920f4
include abort controller for data fetch
jh-bate Jul 19, 2023
386c1ff
use --frozen-lockfile
jh-bate Jul 19, 2023
904b3d0
test out JS module conversion (#50)
krystophv Jul 20, 2023
aaa2030
convert from CJS to ESM
krystophv Jul 20, 2023
ee3ad46
Merge pull request #51 from tidepool-org/cjs_to_esm
krystophv Jul 20, 2023
6e67644
Merge pull request #49 from tidepool-org/report_class
krystophv Jul 20, 2023
4452152
update data-tools package
krystophv Jul 26, 2023
ef1e654
drop `esm` translation/loading layer from dockerfile and bash script
krystophv Jul 26, 2023
c14affa
update for prom-client breaking changes
krystophv Jul 26, 2023
ef06d47
put headers in request config structure
krystophv Jul 26, 2023
1d57a10
tweaks to classic export
krystophv Jul 26, 2023
c872fa0
bump data-tools dependency
krystophv Jul 26, 2023
e963fc6
match `formatDateEndpoints` func that adds a day to endpoint in Web
krystophv Aug 4, 2023
6bb6e88
match 'getLastNDays` func that subtracts one less than provided range
krystophv Aug 11, 2023
051cba0
adjust unit tests for date range extent change
krystophv Aug 14, 2023
f07bd18
bump viz version
krystophv Aug 23, 2023
f3bde25
Apply suggestions from code review
krystophv Aug 23, 2023
d11d3c4
linting
krystophv Aug 23, 2023
2b8609e
[BACK-2540] update packages
krystophv Sep 19, 2023
667a220
v1.7.0
krystophv Sep 19, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 0 additions & 16 deletions .eslintrc

This file was deleted.

21 changes: 21 additions & 0 deletions .eslintrc.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
module.exports = {
extends: 'airbnb',
parser: '@babel/eslint-parser',
plugins: ['lodash'],
parserOptions: {
ecmaVersion: 6,
requireConfigFile: false,
},
rules: {
'no-plusplus': [
'error',
{
allowForLoopAfterthoughts: true,
},
],
'import/extensions': [0, { '<js>': 'always' }],
},
settings: {
lodash: 3,
},
};
1 change: 1 addition & 0 deletions .nvmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
16.20.1
3 changes: 2 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ sudo: false
language: node_js

node_js:
- 12.14.0
- 16.20.1
- node

cache: yarn
Expand All @@ -28,6 +28,7 @@ services:

script:
- yarn run lint
- yarn test
- ./artifact.sh

matrix:
Expand Down
10 changes: 5 additions & 5 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
### Stage 0 - Base image
FROM node:12.14.0-alpine as base
FROM node:16.20.1-alpine as base
WORKDIR /app
RUN apk --no-cache update && \
apk --no-cache upgrade && \
apk add --no-cache --virtual .build-dependencies python make g++ && \
apk add --no-cache --virtual .build-dependencies python3 make g++ && \
mkdir -p node_modules && chown -R node:node .


Expand All @@ -14,7 +14,7 @@ COPY package.json .
COPY yarn.lock .
RUN \
# Build and separate all dependancies required for production
yarn install --production && cp -R node_modules production_node_modules \
yarn install --production --frozen-lockfile && cp -R node_modules production_node_modules \
# Build all modules, including `devDependencies`
&& yarn install \
&& yarn cache clean
Expand All @@ -29,7 +29,7 @@ COPY --chown=node:node --from=dependencies /app/node_modules ./node_modules
COPY --chown=node:node . .
USER node
EXPOSE 9300
CMD node -r esm ./app.js
CMD node ./app.js


### Stage 3 - Test
Expand All @@ -46,4 +46,4 @@ COPY --from=dependencies /app/production_node_modules ./node_modules
COPY --chown=node:node . .
USER node
EXPOSE 9300
CMD node -r esm ./app.js
CMD node ./app.js
166 changes: 23 additions & 143 deletions app.js
Original file line number Diff line number Diff line change
@@ -1,43 +1,28 @@
/* eslint no-restricted-syntax: [0, "ForInStatement"] */

import _ from 'lodash';
import fs from 'fs';
import { existsSync, readFileSync } from 'fs';
import http from 'http';
import https from 'https';
import axios from 'axios';
import express from 'express';
import bodyParser from 'body-parser';
import queryString from 'query-string';
import dataTools from '@tidepool/data-tools';
import { Registry, Counter } from 'prom-client';
import logMaker from './log';
import { createTerminus } from '@godaddy/terminus';
import { exportTimeout, register, logMaker } from './lib/utils.js';
import { getUserData, getUserReport, postUserReport } from './lib/handlers/index.js';

const log = logMaker('app.js', { level: process.env.DEBUG_LEVEL || 'info' });

const { createTerminus } = require('@godaddy/terminus');

const client = require('prom-client');

const { collectDefaultMetrics } = client;
const register = new Registry();

collectDefaultMetrics({ register });

const createCounter = (name, help, labelNames) => new Counter({
name, help, labelNames, registers: [register],
export const log = logMaker('app.js', {
level: process.env.DEBUG_LEVEL || 'info',
});

const statusCount = createCounter('tidepool_export_status_count', 'The number of errors for each status code.', ['status_code', 'export_format']);

function maybeReplaceWithContentsOfFile(obj, field) {
const potentialFile = obj[field];
if (potentialFile != null && fs.existsSync(potentialFile)) {
if (potentialFile != null && existsSync(potentialFile)) {
// eslint-disable-next-line no-param-reassign
obj[field] = fs.readFileSync(potentialFile).toString();
obj[field] = readFileSync(potentialFile).toString();
}
}

const config = {};
export const config = {};
config.httpPort = process.env.HTTP_PORT;
config.httpsPort = process.env.HTTPS_PORT;
if (process.env.HTTPS_CONFIG) {
Expand All @@ -50,134 +35,27 @@ if (process.env.HTTPS_CONFIG) {
if (!config.httpPort) {
config.httpPort = 9300;
}
config.exportTimeout = _.defaultTo(parseInt(process.env.EXPORT_TIMEOUT, 10), 120000);
config.exportTimeout = exportTimeout;
log.info(`Export download timeout set to ${config.exportTimeout} ms`);

const app = express();

app.get('/metrics', async (req, res) => {
res.set('Content-Type', register.contentType);
res.end(register.metrics());
res.end(await register.metrics());
});

function buildHeaders(request) {
if (request.headers['x-tidepool-session-token']) {
return {
headers: {
'x-tidepool-session-token': request.headers['x-tidepool-session-token'],
},
};
}
return {};
}

app.use(bodyParser.urlencoded({
extended: false,
}));

app.get('/export/:userid', async (req, res) => {
// Set the timeout for the request. Make it 10 seconds longer than
// our configured timeout to give the service time to cancel the API data
// request, and close the outgoing data stream cleanly.
req.setTimeout(config.exportTimeout + 10000);

const queryData = [];

let logString = `Requesting download for User ${req.params.userid}`;
if (req.query.bgUnits) {
logString += ` in ${req.query.bgUnits}`;
}
if (req.query.startDate) {
queryData.startDate = req.query.startDate;
logString += ` from ${req.query.startDate}`;
}
if (req.query.endDate) {
queryData.endDate = req.query.endDate;
logString += ` until ${req.query.endDate}`;
}
if (req.query.restricted_token) {
queryData.restricted_token = req.query.restricted_token;
logString += ' with restricted_token';
}
log.info(logString);

const exportFormat = req.query.format;

try {
const cancelRequest = axios.CancelToken.source();

const requestConfig = buildHeaders(req);
requestConfig.responseType = 'stream';
requestConfig.cancelToken = cancelRequest.token;
const dataResponse = await axios.get(`${process.env.API_HOST}/data/${req.params.userid}?${queryString.stringify(queryData)}`, requestConfig);
log.debug(`Downloading data for User ${req.params.userid}...`);

const processorConfig = { bgUnits: req.query.bgUnits || 'mmol/L' };

let writeStream = null;
app.use(
bodyParser.urlencoded({
extended: false,
}),
);

if (exportFormat === 'json') {
res.attachment('TidepoolExport.json');
writeStream = dataTools.jsonStreamWriter();

dataResponse.data
.pipe(dataTools.jsonParser())
.pipe(dataTools.splitPumpSettingsData())
.pipe(dataTools.tidepoolProcessor(processorConfig))
.pipe(writeStream)
.pipe(res);
} else {
res.attachment('TidepoolExport.xlsx');
writeStream = dataTools.xlsxStreamWriter(res, processorConfig);

dataResponse.data
.pipe(dataTools.jsonParser())
.pipe(dataTools.splitPumpSettingsData())
.pipe(dataTools.tidepoolProcessor(processorConfig))
.pipe(writeStream);
}

// Create a timeout timer that will let us cancel the incoming request gracefully if
// it's taking too long to fulfil.
const timer = setTimeout(() => {
res.emit('timeout', config.exportTimeout);
}, config.exportTimeout);

// Wait for the stream to complete, by wrapping the stream completion events in a Promise.
try {
await new Promise((resolve, reject) => {
dataResponse.data.on('end', resolve);
dataResponse.data.on('error', (err) => reject(err));
res.on('error', (err) => reject(err));
res.on('timeout', async () => {
statusCount.inc({ status_code: 408, export_format: exportFormat });
reject(new Error('Data export request took too long to complete. Cancelling the request.'));
});
});
statusCount.inc({ status_code: 200, export_format: exportFormat });
log.debug(`Finished downloading data for User ${req.params.userid}`);
} catch (e) {
log.error(`Error while downloading: ${e}`);
// Cancel the writeStream, rather than let it close normally.
// We do this to show error messages in the downloaded files.
writeStream.cancel();
cancelRequest.cancel('Data export timed out.');
}

clearTimeout(timer);
} catch (error) {
if (error.response && error.response.status === 403) {
statusCount.inc({ status_code: 403, export_format: exportFormat });
res.status(error.response.status).send('Not authorized to export data for this user.');
log.error(`${error.response.status}: ${error}`);
} else {
statusCount.inc({ status_code: 500, export_format: exportFormat });
res.status(500).send('Server error while processing data. Please contact Tidepool Support.');
log.error(`500: ${error}`);
}
}
});
app.use(bodyParser.json());

app.get('/export/:userid', getUserData());
app.get('/export/report/:userid', getUserReport());
app.post('/export/report/:userid', postUserReport());

function beforeShutdown() {
return new Promise((resolve) => {
Expand Down Expand Up @@ -208,7 +86,9 @@ if (config.httpPort) {

if (config.httpsPort) {
if (_.isEmpty(config.httpsConfig)) {
log.error('SSL endpoint is enabled, but no valid config was found. Exiting.');
log.error(
'SSL endpoint is enabled, but no valid config was found. Exiting.',
);
process.exit(1);
} else {
const server = https.createServer(config.httpsConfig, app);
Expand Down
Loading