Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node Attributes #141

Open
wants to merge 37 commits into
base: main
Choose a base branch
from
Open

Node Attributes #141

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
b2504bd
3 nodes complete
JesusGuerrero Feb 15, 2023
f863a4b
regex node attributes
JesusGuerrero Feb 17, 2023
ac8dc53
cleanup for prod
JesusGuerrero Feb 17, 2023
9c87f85
Fix server crash
Mar 9, 2023
3f2bf33
Fix attribute bug
Mar 9, 2023
07f22e9
Fix sequence node
Mar 13, 2023
b51fdc0
Update to use attributes list instead of hash
Mar 14, 2023
1c716b3
Add mapped terms support
Mar 14, 2023
f6ec88a
Update regex to use arrays
Mar 14, 2023
87ba823
Fix check box
Mar 14, 2023
a6a1129
Fix regex
Mar 15, 2023
ccd9eb7
Clean up code
Mar 15, 2023
346bd05
Clean up sequence node
Mar 15, 2023
df2e782
Update literal to use arrays for attributes
Mar 15, 2023
06fb8db
Update union to use attributes [wip]
Mar 16, 2023
86289e1
Add initial UI for attributes in filter/ consolidate nodes
Mar 16, 2023
a92db8c
Prevent consolidate nodes from having multiple inputs
Mar 16, 2023
bcd3b8b
Fix consolidate node
Mar 17, 2023
fee1f81
Fix union panel
Mar 23, 2023
fb041c2
Fix sequence node
Mar 24, 2023
7f8c0d9
Fix consolidate and regex
Mar 24, 2023
a52f5c3
Fix union node
Mar 24, 2023
0097d72
Prevent filter crash when no attributes are defined
Mar 27, 2023
eb8139c
Improve error handling and fix regex
Mar 27, 2023
f27051f
Fix regex label
Mar 28, 2023
d7a96a4
Update tutorial flow to include attributes and add attributes to unio…
Mar 28, 2023
f8c301f
Remove info icon
Mar 28, 2023
174e8f9
Fix checkbox
Mar 28, 2023
c792929
Automatically save attribute without pressing enter
Mar 29, 2023
a8bf673
Prevent crash if attributes aren't defined
Mar 29, 2023
2df0aea
Add validation to attributes list
Mar 30, 2023
90c878e
Fix tokens sequence bug
Mar 31, 2023
2c5206e
Update wording of rename attributes
Apr 3, 2023
8efe74e
Automatically update regex when attributes change
Apr 4, 2023
36b5adf
Update filter to select attributes instead of nodes
Apr 4, 2023
5903aee
Update consolidate to select attributes instead of nodes
Apr 4, 2023
71b12f5
Fix > symbol
Apr 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
/.pnp
.pnp.js

/Seer-Core
/Seer-Core/aql-processor/run-aql-result/.gitignore
/Seer-Core/aql-processor/user-data-in/.gitignore
# testing
/coverage

Expand Down
4 changes: 0 additions & 4 deletions Seer-Core/aql-processor/run-aql-result/.gitignore
View file
Open in desktop

This file was deleted.

4 changes: 0 additions & 4 deletions Seer-Core/aql-processor/user-data-in/.gitignore
View file
Open in desktop

This file was deleted.

230 changes: 172 additions & 58 deletions sample-flows/tutorial-flow.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,6 +444,41 @@
"schemas": []
},
"nodes": [
{
"label": "RevenueByDivision",
"nodeId": "295c7f55-9008-46d2-9f00-e5fcfa8cdbf6",
"type": "sequence",
"description": "Connect your extractors for execution",
"isValid": true,
"renamed": "RevenueByDivision",
"pattern": "(<RevenueOfDivision.RevenueOfDivision>)<Token>{0,35}(<Amount.Amount>)",
"upstreamNodes": [
{
"label": "RevenueOfDivision",
"nodeId": "3f122664-92b1-4d36-ac49-4341d3281294",
"type": "union",
"visible": false,
"renamed": "RevenueOfDivision"
},
{
"label": "Amount",
"nodeId": "7ac4050a-15a3-4c8b-8eb3-ece334de9bd2",
"type": "regex",
"visible": false,
"renamed": "Amount"
}
],
"tokens": [{ "min": "0", "max": "35" }]
},
{
"label": "RevenueConsolidated",
"nodeId": "9de33a19-783f-4a93-83dd-3467710dd430",
"type": "consolidate",
"description": "Remove overlapping annotations",
"isValid": true,
"consolidateTarget": "RevenueByDivision",
"consolidatePolicy": "NotContainedWithin"
},
{
"label": "Division",
"nodeId": "bba756a6-5a45-426c-a8d9-4a1cd6a19881",
Expand All @@ -459,31 +494,83 @@
"caseSensitivity": "match",
"lemmaMatch": false,
"externalResourceChecked": false,
"mapTerms": false
"mapTerms": false,
"attributes": [
{
"nodeId": "bba756a6-5a45-426c-a8d9-4a1cd6a19881",
"label": "Division",
"value": "Division",
"visible": true,
"disabled": true
}
]
},
{
"label": "Preposition",
"nodeId": "df2832a2-957f-43aa-88e4-a09fdf4c3b46",
"label": "Metric",
"nodeId": "8494ae16-4f5e-4e48-87d0-988b98c2cefd",
"type": "dictionary",
"description": "Phrases to be matched.",
"isValid": true,
"items": ["from", "for"],
"items": ["revenue"],
"caseSensitivity": "ignore",
"lemmaMatch": false,
"lemmaMatch": true,
"externalResourceChecked": false,
"mapTerms": false
"mapTerms": false,
"attributes": [
{
"nodeId": "8494ae16-4f5e-4e48-87d0-988b98c2cefd",
"label": "Metric",
"value": "Metric",
"visible": true,
"disabled": true
}
]
},
{
"label": "Metric",
"nodeId": "8494ae16-4f5e-4e48-87d0-988b98c2cefd",
"label": "Preposition",
"nodeId": "df2832a2-957f-43aa-88e4-a09fdf4c3b46",
"type": "dictionary",
"description": "Phrases to be matched.",
"isValid": true,
"items": ["revenue"],
"items": ["from", "for"],
"caseSensitivity": "ignore",
"lemmaMatch": true,
"lemmaMatch": false,
"externalResourceChecked": false,
"mapTerms": false
"mapTerms": false,
"attributes": [
{
"nodeId": "df2832a2-957f-43aa-88e4-a09fdf4c3b46",
"label": "Preposition",
"value": "Preposition",
"visible": true,
"disabled": true
}
]
},
{
"label": "Amount",
"nodeId": "7ac4050a-15a3-4c8b-8eb3-ece334de9bd2",
"type": "regex",
"description": "Build a regular expression extractor.",
"isValid": true,
"regexInput": "\\$\\d+(\\.\\d+)?\\s+billion",
"expressionType": "regular",
"caseSensitivity": "match",
"tokenRange": { "checked": false, "range": [0, 0] },
"canonEq": false,
"dotAll": true,
"multiline": false,
"unixLines": false,
"attributes": [
{
"nodeId": "7ac4050a-15a3-4c8b-8eb3-ece334de9bd2",
"label": "Amount",
"visible": true,
"disabled": true
}
],
"editLabel": "",
"editId": null
},
{
"label": "RevenueOfDivision1",
Expand All @@ -509,7 +596,30 @@
"renamed": "Metric"
}
],
"tokens": [{ "min": "1", "max": "2" }]
"tokens": [{ "min": "1", "max": "2" }],
"attributes": [
{
"nodeId": "863f5c4c-6c1e-427a-9482-5592732aa81e",
"label": "RevenueOfDivision1",
"visible": true,
"value": "RevenueOfDivision",
"disabled": true
},
{
"label": "Division",
"nodeId": "bba756a6-5a45-426c-a8d9-4a1cd6a19881",
"value": "Division",
"disabled": false,
"visible": true
},
{
"label": "Metric",
"nodeId": "8494ae16-4f5e-4e48-87d0-988b98c2cefd",
"value": "Metric",
"disabled": false,
"visible": true
}
]
},
{
"label": "RevenueOfDivision2",
Expand Down Expand Up @@ -545,23 +655,38 @@
"tokens": [
{ "min": "0", "max": "1" },
{ "min": "0", "max": "2" }
],
"attributes": [
{
"nodeId": "27fc9430-cb52-4dff-965b-f5f2261c43eb",
"label": "RevenueOfDivision2",
"visible": true,
"value": "RevenueOfDivision",
"disabled": true
},
{
"label": "Metric",
"nodeId": "8494ae16-4f5e-4e48-87d0-988b98c2cefd",
"value": "Metric",
"disabled": false,
"visible": true
},
{
"label": "Preposition",
"nodeId": "df2832a2-957f-43aa-88e4-a09fdf4c3b46",
"value": "Preposition",
"disabled": false,
"visible": false
},
{
"label": "Division",
"nodeId": "bba756a6-5a45-426c-a8d9-4a1cd6a19881",
"value": "Division",
"disabled": false,
"visible": true
}
]
},
{
"label": "Amount",
"nodeId": "7ac4050a-15a3-4c8b-8eb3-ece334de9bd2",
"type": "regex",
"description": "Build a regular expression extractor.",
"isValid": true,
"regexInput": "\\$\\d+(\\.\\d+)?\\s+billion",
"expressionType": "regular",
"caseSensitivity": "match",
"tokenRange": { "checked": false, "range": [0, 0] },
"canonEq": false,
"dotAll": true,
"multiline": false,
"unixLines": false
},
{
"label": "RevenueOfDivision",
"nodeId": "3f122664-92b1-4d36-ac49-4341d3281294",
Expand All @@ -577,42 +702,31 @@
"label": "RevenueOfDivision2",
"nodeId": "27fc9430-cb52-4dff-965b-f5f2261c43eb"
}
]
},
{
"label": "RevenueByDivision",
"nodeId": "295c7f55-9008-46d2-9f00-e5fcfa8cdbf6",
"type": "sequence",
"description": "Connect your extractors for execution",
"isValid": true,
"renamed": "RevenueByDivision",
"pattern": "(<RevenueOfDivision.RevenueOfDivision>)<Token>{0,35}(<Amount.Amount>)",
"upstreamNodes": [
],
"attributes": [
{
"label": "RevenueOfDivision",
"nodeId": "3f122664-92b1-4d36-ac49-4341d3281294",
"type": "union",
"visible": false,
"renamed": "RevenueOfDivision"
"nodeId": "863f5c4c-6c1e-427a-9482-5592732aa81e",
"label": "RevenueOfDivision1",
"visible": true,
"value": "RevenueOfDivision",
"disabled": true
},
{
"label": "Amount",
"nodeId": "7ac4050a-15a3-4c8b-8eb3-ece334de9bd2",
"type": "regex",
"visible": false,
"renamed": "Amount"
"label": "Division",
"nodeId": "bba756a6-5a45-426c-a8d9-4a1cd6a19881",
"value": "Division",
"disabled": false,
"visible": true
},
{
"label": "Metric",
"nodeId": "8494ae16-4f5e-4e48-87d0-988b98c2cefd",
"value": "Metric",
"disabled": false,
"visible": true
}
],
"tokens": [{ "min": "0", "max": "35" }]
},
{
"label": "RevenueConsolidated",
"nodeId": "9de33a19-783f-4a93-83dd-3467710dd430",
"type": "consolidate",
"description": "Remove overlapping annotations",
"isValid": true,
"consolidateTarget": "RevenueByDivision",
"consolidatePolicy": "NotContainedWithin"
"mismatchedAttributes": false
},
{
"label": "Input Documents_1",
Expand Down
10 changes: 6 additions & 4 deletions server.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,10 +87,9 @@ const createZipArchive = async (tmpFolder, fileName) => {
const moveZipFile = (tmpFolder, fileName) => {
const currentPath = `${tmpFolder}/${fileName}`;
const destinationPath = `${systemTdataFolder}/user-data-in/${fileName}`;
fs.rename(currentPath, destinationPath, function (err) {
fs.copyFile(currentPath, destinationPath, function (err) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you copying the file instead of renaming?

Check failure

Code scanning / CodeQL

Uncontrolled data used in path expression

This path depends on a [user-provided value](1).

Check failure

Code scanning / CodeQL

Uncontrolled data used in path expression

This path depends on a [user-provided value](1).
if (err) {
console.log(`error moving zipfile ${fileName}`);
throw err;
} else {
console.log(`moved zipfile ${fileName} to be processed.`);
}
Expand Down Expand Up @@ -201,7 +200,7 @@ app.post(
//read document to render in UI
const docPath = `${workingFolder}/payload.txt`;
const document = fs.readFileSync(docPath, 'utf8');
fs.rmSync(workingFolder, { recursive: true, force: true });
// fs.rmSync(workingFolder, { recursive: true, force: true });

res.status(200).send({
message: 'Execution submitted successfully.',
Expand All @@ -227,6 +226,9 @@ const formatResults = ({ annotations, instrumentationInfo }) => {
annotation.forEach((elem) => {
const attributes = {};
Object.keys(elem).forEach((key) => {
if (elem[key] === null) {
return;
}
const { location, text } = elem[key];
attributes[key] = {
start: location?.begin,
Expand Down Expand Up @@ -305,7 +307,7 @@ app.get(
if (errorMessage) {
console.log('found error message', errorMessage);
deleteFile(file, resultFileName);
return res.status(200).send({ status: 'error', message: errorMessage });
return res.status(409).send({ status: 'error', message: errorMessage });
}

const results =
Expand Down
Loading