Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(glossary): from export to publish #1424

Merged
merged 45 commits into from
Jun 18, 2024
Merged
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
83ecfb4
fix: glossary
maxgfr May 30, 2024
0be4d5e
fix: glossary
maxgfr May 30, 2024
1df3298
fix: glossary
maxgfr May 30, 2024
b052b35
fix: glossary
maxgfr May 30, 2024
8da5511
fix: glossary
maxgfr May 31, 2024
1e9ac30
fix: glossary
maxgfr May 31, 2024
03d6824
fix: glossary
maxgfr Jun 3, 2024
3bc5d38
fix: deployment
maxgfr Jun 3, 2024
cda1c3d
fix: ui
maxgfr Jun 4, 2024
abd5023
fix: variable
maxgfr Jun 4, 2024
ffa2bd3
fix: variable
maxgfr Jun 4, 2024
54336da
fix: variable
maxgfr Jun 4, 2024
4d67071
fix: variable
maxgfr Jun 4, 2024
4b0ba18
fix: merge
maxgfr Jun 4, 2024
2178a5e
Merge branch 'master' into maxgfr/glossary
maxgfr Jun 4, 2024
b204a24
Merge branch 'master' into maxgfr/glossary
maxgfr Jun 4, 2024
ba90d32
fix: replace uuid
maxgfr Jun 5, 2024
942637b
fix: formattage
maxgfr Jun 5, 2024
c7ce777
fix: alert-cli
maxgfr Jun 5, 2024
c88a929
empty
maxgfr Jun 5, 2024
b45b138
fix: glossary
maxgfr Jun 5, 2024
61c32f6
fix: glossary
maxgfr Jun 5, 2024
49cd64e
fix: glossary
maxgfr Jun 6, 2024
53c3771
fix: glossary
maxgfr Jun 6, 2024
60d50b2
fix: glossary
maxgfr Jun 6, 2024
d917a8b
empty
maxgfr Jun 6, 2024
089058d
fix: glossary
maxgfr Jun 6, 2024
b4a4c3c
fix: glossary
maxgfr Jun 6, 2024
2d8e475
fix: glossary
maxgfr Jun 6, 2024
d662156
fix: log
maxgfr Jun 7, 2024
7c87a33
fix: log
maxgfr Jun 7, 2024
decc534
fix: donnnnnne
maxgfr Jun 7, 2024
e2ab2c6
fix: done
maxgfr Jun 7, 2024
4e082a7
fix: done
maxgfr Jun 7, 2024
1e780d9
empty
maxgfr Jun 10, 2024
7afed52
fix: done
maxgfr Jun 10, 2024
94745d3
fix: performance
maxgfr Jun 10, 2024
18ed206
fix: performance
maxgfr Jun 10, 2024
261f6ea
fix: merge
maxgfr Jun 11, 2024
687d80c
fix: returning
maxgfr Jun 17, 2024
34dc741
Merge branch 'master' into linked-opti-glossary
maxgfr Jun 18, 2024
248674f
fix: performance
maxgfr Jun 18, 2024
b859eb4
fix: fetch
maxgfr Jun 18, 2024
a603b44
fix: build encore
maxgfr Jun 18, 2024
03b007b
fix: build encore
maxgfr Jun 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: glossary
maxgfr committed Jun 3, 2024

Verified

This commit was signed with the committer’s verified signature.
BryanttV Bryann Valderrama
commit 03d6824a1fef8c71de5419ffd717bd7554ce3a52
1 change: 1 addition & 0 deletions shared/types/src/hasura/editorial-content.ts
Original file line number Diff line number Diff line change
@@ -77,6 +77,7 @@ export enum EditorialSectionDisplayMode {
export type EditorialContentDoc = {
date: string;
intro: string;
introWithGlossary: string;
section_display_mode?: EditorialSectionDisplayMode;
dismissalProcess?: boolean;
contents: EditorialContentBaseContentPart[];
Original file line number Diff line number Diff line change
@@ -22,6 +22,7 @@ const mockFiches: FicheTravail[] = [
anchor: "sous-titre",
description: "desc",
html: "html",
htmlWithGlossary: "html",
references: [
{
cid: "cid123",
@@ -47,6 +48,7 @@ const mockFiches: FicheTravail[] = [
anchor: "sous-titre-2",
description: "desc 2",
html: "html",
htmlWithGlossary: "html",
references: [
{
cid: "cid123",
Original file line number Diff line number Diff line change
@@ -27,6 +27,7 @@ export type ContributionElasticDocumentLightRelatedContent = Omit<
"linkedContent"
> & {
linkedContent: ContributionLinkedContent[];
contentWithGlossary?: string;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Il est optional pour le cas des fiches SP ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Je crois j'avais fait ça par simplicité pour éviter de modifier le type en le destructurant. Mais oui, ce champ n'existe pas en théorie. Mais ça demandait pas mal de refacto au niveau des types....

};

export async function generateContributions(
@@ -78,13 +79,8 @@ export async function generateContributions(
};
}

const contribWithoutGlossary = {
...contrib.document,
content,
};

generatedContributions.push({
...contribWithoutGlossary,
const contribution: ContributionElasticDocumentLightRelatedContent = {
...contrib,
...generateMetadata(contrib),
...getContributionContent(content),
...doc,
@@ -95,7 +91,11 @@ export async function generateContributions(
highlight,
messageBlock,
references,
});
};

delete contribution.contentWithGlossary;

generatedContributions.push(contribution);
}

// Some related content link to another customized contribution
44 changes: 23 additions & 21 deletions targets/export-elasticsearch/src/ingester/informations/generate.ts
Original file line number Diff line number Diff line change
@@ -12,28 +12,30 @@ interface Return {
export const generateEditorialContents = (
documents: DocumentElasticWithSource<EditorialContentDoc>[]
): Return => {
// intro: data.intro,
// introWithGlossary: addGlossaryContentToMarkdown(
// glossary,
// data.intro ?? ""
// ),
// description: data.description,
// sectionDisplayMode: data.sectionDisplayMode,
// dismissalProcess: data.dismissalProcess,
// ? {
// title: block.content,
// }
// : {
// markdown: block.content,
// htmlWithGlossary: addGlossaryContentToMarkdown(
// glossary,
// block.content
// ),
// }),
// remove markdown et intro
const relatedIdsDocuments = getRelatedIdsDocuments(documents);
const documentsOptimized = documents.map((document: any) => {
const introWithGlossary = document.introWithGlossary;
delete document.intro;
delete document.introWithGlossary;
Comment on lines +17 to +18
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

je ne suis pas fan des delete, on ne peux pas passer par une destructuration ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ouais j'avais essayé, mais c'était moins lisible je trouve

return {
...document,
intro: introWithGlossary,
contents: document.contents.map((content: any) => {
content.blocks = content.blocks.map((block: any) => {
const htmlWithGlossary = block.htmlWithGlossary;
delete block.markdown;
delete block.htmlWithGlossary;
return {
...block,
html: htmlWithGlossary,
};
});
return content;
}),
};
});
const relatedIdsDocuments = getRelatedIdsDocuments(documentsOptimized);
return {
documents,
documents: documentsOptimized,
relatedIdsDocuments,
};
};

Unchanged files with check annotations Beta

FROM elasticsearch:7.13.4

Check failure on line 1 in docker/elasticsearch/Dockerfile

GitHub Actions / Lint Dockerfile

DL3026 error: Use only an allowed registry in the FROM image
WORKDIR /usr/share/elasticsearch
ARG NODE_VERSION=20.3.1-alpine
FROM node:$NODE_VERSION AS deps

Check failure on line 3 in targets/alert-cli/Dockerfile

GitHub Actions / Lint Dockerfile

DL3026 error: Use only an allowed registry in the FROM image
WORKDIR /app
RUN yarn workspaces focus --production alert-cli && yarn cache clean
FROM node:$NODE_VERSION

Check failure on line 35 in targets/alert-cli/Dockerfile

GitHub Actions / Lint Dockerfile

DL3026 error: Use only an allowed registry in the FROM image
# Add git to docker image
RUN apk add --no-cache git openssh-client

Check failure on line 38 in targets/alert-cli/Dockerfile

GitHub Actions / Lint Dockerfile

DL3018 warning: Pin versions in apk add. Instead of `apk add <package>` use `apk add <package>=<version>`
USER 1000
ARG NODE_VERSION=20.3.1-alpine
FROM node:$NODE_VERSION AS deps

Check failure on line 3 in targets/export-elasticsearch/Dockerfile

GitHub Actions / Lint Dockerfile

DL3026 error: Use only an allowed registry in the FROM image
RUN apk add --update python3 make g++ && rm -rf /var/cache/apk/*

Check failure on line 5 in targets/export-elasticsearch/Dockerfile

GitHub Actions / Lint Dockerfile

DL3018 warning: Pin versions in apk add. Instead of `apk add <package>` use `apk add <package>=<version>`

Check failure on line 5 in targets/export-elasticsearch/Dockerfile

GitHub Actions / Lint Dockerfile

DL3019 info: Use the `--no-cache` switch to avoid the need to use `--update` and remove `/var/cache/apk/*` when done installing packages
WORKDIR /app
RUN yarn workspaces focus --production export-elasticsearch && yarn cache clean
RUN mkdir -p /app/targets/export-elasticsearch/node_modules
FROM node:$NODE_VERSION

Check failure on line 35 in targets/export-elasticsearch/Dockerfile

GitHub Actions / Lint Dockerfile

DL3026 error: Use only an allowed registry in the FROM image
WORKDIR /app
ARG NODE_VERSION=20.3.1-alpine
FROM node:$NODE_VERSION AS deps

Check failure on line 3 in targets/frontend/Dockerfile

GitHub Actions / Lint Dockerfile

DL3026 error: Use only an allowed registry in the FROM image
WORKDIR /app
RUN yarn workspaces focus --production frontend && yarn cache clean
RUN mkdir -p targets/frontend/node_modules
FROM node:$NODE_VERSION

Check failure on line 39 in targets/frontend/Dockerfile

GitHub Actions / Lint Dockerfile

DL3026 error: Use only an allowed registry in the FROM image
WORKDIR /app
});
it("should throw error if no xlsx file found", async () => {
const url = "https://example.com/files";

Check warning on line 40 in targets/alert-cli/src/dares/__tests__/scrapping.test.ts

GitHub Actions / Lint (alert-cli)

'url' is assigned a value but never used
const html = `
<html>
<body>
const dataJson = JSON.parse(fs.readFileSync(pathIndex, "utf8"));
const supportedCcIndexJson: Agreement[] = dataJson.map((cc: any) => {

Check warning on line 29 in targets/alert-cli/src/dares/difference.ts

GitHub Actions / Lint (alert-cli)

Unexpected any. Specify a different type
return {
name: cc.title,
num: cc.num,
);
const html = response.data;
const regex = /href="([^"]*\.xlsx)"/g;
const match = regex.exec(html);

Check warning on line 10 in targets/alert-cli/src/dares/scrapping.ts

GitHub Actions / Lint (alert-cli)

Unsafe argument of type `any` assigned to a parameter of type `string`
if (!match) {
throw new Error("No xlsx file found");
}
return match[1];
}
const baseRegex = /<base href="(.*)" \/>/g;
const baseMatch = baseRegex.exec(html);

Check warning on line 18 in targets/alert-cli/src/dares/scrapping.ts

GitHub Actions / Lint (alert-cli)

Unsafe argument of type `any` assigned to a parameter of type `string`
if (!baseMatch) {
throw new Error(`xlsx file url not valid : ${match[1]}`);
}
implements RelevantDocumentsExtractor
{
extractReferences({
modified,

Check warning on line 8 in targets/alert-cli/src/diff/dila/__tests__/RelevantDocumentsExtractorStub.ts

GitHub Actions / Lint (alert-cli)

'modified' is defined but never used
removed,

Check warning on line 9 in targets/alert-cli/src/diff/dila/__tests__/RelevantDocumentsExtractorStub.ts

GitHub Actions / Lint (alert-cli)

'removed' is defined but never used
}: Pick<DilaChanges, "modified" | "removed">): Promise<DocumentReferences[]> {
return Promise.resolve([]);
}
import payload from "./mocks/editorialContent.payload.json";
jest.mock("@shared/utils", () => {
return {

Check warning on line 6 in targets/alert-cli/src/diff/dila/extractReferences/__tests__/editorialContents.test.ts

GitHub Actions / Lint (alert-cli)

Unsafe return of an `any` typed value
...jest.requireActual("@shared/utils"),
gqlClient: jest.fn(),
createGetArticleReference: () =>
});
test("extractMailTemplateRef", async () => {
const references = await extractEditorialContentTemplateRef(payload as any);

Check warning on line 23 in targets/alert-cli/src/diff/dila/extractReferences/__tests__/editorialContents.test.ts

GitHub Actions / Lint (alert-cli)

Unsafe argument of type `any` assigned to a parameter of type `EditorialContentSubset[]`

Check warning on line 23 in targets/alert-cli/src/diff/dila/extractReferences/__tests__/editorialContents.test.ts

GitHub Actions / Lint (alert-cli)

Unexpected any. Specify a different type
expect(references).toMatchInlineSnapshot(`
[
{
import payload from "./mocks/mailTemplate.payload.json";
jest.mock("@shared/utils", () => {
return {

Check warning on line 8 in targets/alert-cli/src/diff/dila/extractReferences/__tests__/mailTemplates.test.ts

GitHub Actions / Lint (alert-cli)

Unsafe return of an `any` typed value
...jest.requireActual("@shared/utils"),
gqlClient: jest.fn(),
createGetArticleReference: () =>
server.setErrorConfig((app) => {
// eslint-disable-next-line @typescript-eslint/no-unused-vars
app.use(
(err: Error, _req: Request, res: Response, _next: NextFunction) => {

Check warning on line 41 in targets/export-elasticsearch/src/controllers/__test__/export.test.ts

GitHub Actions / Lint (export-elasticsearch)

'_next' is defined but never used
res.status(500).json({ errors: err.message });
}
);
],
];
const duplicateSlugs = await getDuplicateSlugs(documents);
expect(Object.entries(duplicateSlugs).length).toBe(0);

Check warning on line 22 in targets/export-elasticsearch/src/ingester/__tests__/cdtnDocuments.test.ts

GitHub Actions / Lint (export-elasticsearch)

Unsafe argument of type `any` assigned to a parameter of type `{ [s: string]: unknown; } | ArrayLike<unknown>`
});
test("should return an array of duplicated slug", async () => {
],
];
const duplicateSlugs = await getDuplicateSlugs(documents);
expect(Object.entries(duplicateSlugs).length).toBe(1);

Check warning on line 42 in targets/export-elasticsearch/src/ingester/__tests__/cdtnDocuments.test.ts

GitHub Actions / Lint (export-elasticsearch)

Unsafe argument of type `any` assigned to a parameter of type `{ [s: string]: unknown; } | ArrayLike<unknown>`
});
});
describe("updateContributionsAndGetIDCCs", () => {
test("should return a list of iddc", async () => {
const contributions: any[] = [

Check warning on line 48 in targets/export-elasticsearch/src/ingester/__tests__/cdtnDocuments.test.ts

GitHub Actions / Lint (export-elasticsearch)

Unexpected any. Specify a different type
{ idcc: "0292" },
{ idcc: "0829" },
{ idcc: "1557" },
{ idcc: "1909" },
];
const idccs = await getIDCCs(contributions);

Check warning on line 54 in targets/export-elasticsearch/src/ingester/__tests__/cdtnDocuments.test.ts

GitHub Actions / Lint (export-elasticsearch)

Unsafe argument of type `any[]` assigned to a parameter of type `ContributionElasticDocument[]`
expect(Array.from(idccs)).toEqual([292, 829, 1557, 1909]);
});
});
describe("getIDCCs", () => {
it("returns unique IDCC values from old and new contributions", () => {
const contribs: any = [{ idcc: "5678" }];

Check warning on line 5 in targets/export-elasticsearch/src/ingester/agreements/__tests__/getIdcc.test.ts

GitHub Actions / Lint (export-elasticsearch)

Unexpected any. Specify a different type
const idccs = getIDCCs(contribs);

Check warning on line 7 in targets/export-elasticsearch/src/ingester/agreements/__tests__/getIdcc.test.ts

GitHub Actions / Lint (export-elasticsearch)

Unsafe argument of type `any` assigned to a parameter of type `ContributionElasticDocument[]`
expect(idccs).toEqual(new Set([5678]));
});
describe("getInfoMessage", () => {
test("getInfoMessage returns expected message for ANSWER", () => {
const data: any = {

Check warning on line 5 in targets/export-elasticsearch/src/ingester/agreements/__tests__/getInfoMessage.test.ts

GitHub Actions / Lint (export-elasticsearch)

Unexpected any. Specify a different type
contentType: "ANSWER",
};
const expected =
"Les informations ci-dessous sont issues de l’analyse des règles prévues par votre convention collective de branche étendue et par le Code du travail.";
expect(getInfoMessage(data)).toBe(expected);

Check warning on line 12 in targets/export-elasticsearch/src/ingester/agreements/__tests__/getInfoMessage.test.ts

GitHub Actions / Lint (export-elasticsearch)

Unsafe argument of type `any` assigned to a parameter of type `ContributionElasticDocument`
});
test("getInfoMessage returns expected message for CDT", () => {
const data: any = {

Check warning on line 16 in targets/export-elasticsearch/src/ingester/agreements/__tests__/getInfoMessage.test.ts

GitHub Actions / Lint (export-elasticsearch)

Unexpected any. Specify a different type
contentType: "CDT",
};
});
describe(`should return undefined`, () => {
it(`with a nonexistent article ID or CID`, async () => {

Check warning on line 25 in targets/ingester/src/articles/kali/__tests__/indexed-article.test.ts

GitHub Actions / Lint (ingester)

Async arrow function has no 'await' expression
const articles = Articles;
expect(indexedArticle(articles, "KALIARTI123456789012")).toBeUndefined();
): Promise<AgreementArticleWithPath[] | undefined[]> {
const agreement = await loadAgreement(agreementIdOrIdcc);
const rootedArticles = flatFilter(agreement as any, { type: "article" });

Check warning on line 22 in targets/ingester/src/articles/kali/agreement-articles-with-path.ts

GitHub Actions / Lint (ingester)

Unsafe argument of type `any` assigned to a parameter of type `Node`

Check warning on line 22 in targets/ingester/src/articles/kali/agreement-articles-with-path.ts

GitHub Actions / Lint (ingester)

Unexpected any. Specify a different type
if (!rootedArticles) {
return [];
}
const articles = await loadArticles();
return rootedArticles.children.map((article: AgreementArticle) => {

Check warning on line 29 in targets/ingester/src/articles/kali/agreement-articles-with-path.ts

GitHub Actions / Lint (ingester)

Unsafe return of an `any` typed value
const a = indexedArticle(articles, article.data.cid);
if (!a) return;
return {
import unistUtilFlatFilter from "unist-util-flat-filter";
import parents from "unist-util-parents";
export type NormalizedArticle = {

Check warning on line 5 in targets/ingester/src/articles/legi/load-labour-code-articles.ts

GitHub Actions / Lint (ingester)

Use an `interface` instead of a `type`
id: string;
cid: string;
label: string;
const tree = await loadCodeDuTravail();
const code = parents(tree);
const codeArticles = unistUtilFlatFilter(code as any, {

Check warning on line 32 in targets/ingester/src/articles/legi/load-labour-code-articles.ts

GitHub Actions / Lint (ingester)

Unsafe argument of type `any` assigned to a parameter of type `Node`

Check warning on line 32 in targets/ingester/src/articles/legi/load-labour-code-articles.ts

GitHub Actions / Lint (ingester)

Unexpected any. Specify a different type
type: "article",
});
return codeArticles.children.map((codeArticle: CodeArticle) =>

Check warning on line 36 in targets/ingester/src/articles/legi/load-labour-code-articles.ts

GitHub Actions / Lint (ingester)

Unsafe return of an `any` typed value
convertCodeArticleToArticle(codeArticle)
);
};
console.timeEnd(` getDocuments ${pkgName}`);
console.log(` ${pkgName}: ${documents.length} documents`);
if (!args.dryRun && documents.length > 0) {
await initDocAvailabity(documents[0].source);

Check warning on line 144 in targets/ingester/src/cli.ts

GitHub Actions / Lint (ingester)

Unsafe argument of type `any` assigned to a parameter of type `string`
console.log(
` ready to ingest ${documents.length} documents from ${pkgName}`
);
Answer,
GenericContributionDoc,
CustomizedContributionDoc,
LegalRef,

Check warning on line 7 in targets/ingester/src/index.d.ts

GitHub Actions / Lint (ingester)

'LegalRef' is defined but never used
FicheServicePublicDoc,
GenericAnswer,
} from "@socialgouv/cdtn-types";