basics

basics

Ce plugin propose une série d'instructions transformer plusieurs formats text (xml, json, cvs, etc.) en flux d'objets Javascript

installation

npm install @ezs/basics

usage

Table of Contents

• BIBParse
• BUFObject
• CSVObject
• CSVParse
• CSVString
• FILELoad
• FILEMerge
• FILESave
• INIString
• JSONParse
• JSONString
• OBJCount
• OBJFlatten
• OBJNamespaces
• OBJStandardize
• TARDump
• TARExtract
• TXTConcat
• TXTObject
• TXTParse
• TXTZip
• URLConnect
• URLFetch
• URLPagination
• URLParse
• URLRequest
• URLStream
• URLString
• XMLConvert
• XMLParse
• XMLString
• ZIPExtract

BIBParse

Take a String and split it at bibtext entry.

Input:

["@article{my_article,\ntitle = {Hello world},\n", "journal = \"Some Journal\"\n"]

Output:

["a", "b", "c", "d"]

Returns Object

BUFObject

Take Mixed and produce Buffer. For example, it's useful to send string to browser.

Parameters

• none undefined

Returns Buffer

CSVObject

Take an Array of arrays and transform rows into objects.

Each row (Array) is tranformed into an object where keys are the values of the first row.

See CSVParse.

Input:

[
  ["a", "b", "c"],
  [1, 2, 3],
  [4, 5, 6]
]

Output:

[{
 "a": 1,
 "b": 2,
 "c": 3
}, {
 "a": 4,
 "b": 5,
 "c": 6
}]

Tip: this is useful after a CSVParse, to convert raw rows into n array of Javascript objects

When several values of the first row are the same, produced keys are suffixed with a number.

Input:

[
  ["a", "a", "b", "b", "b"],
  [1, 2, 3, 4, 5]
]

Output:

[{
   "a1": 1,
   "a2": 2,
   "b1": 3,
   "b2": 4,
   "b3": 5
}]

Parameters

• none undefined

Returns (Object | Array<Object>)

CSVParse

Take String and parse it as CSV to generate arrays.

See:

• CSVObject
• https://github.com/Inist-CNRS/node-csv-string

Input:

"a,b,c\nd,e,d\n"

Output:

[
  ["a", "b", "c"],
  ["d", "e", "d"]
]

Tip: see CSVObject, to convert arrays of values to array of objects.

Parameters

• separator String to indicate the CSV separator (optional, default auto)
• quote String to indicate the CSV quote. (optional, default auto)

Returns Array<Array<String>>

CSVString

Take an array of objects and transform row into a string where each field is separated with a character.

The resulting string is CSV-compliant.

See CSVObject

Input:

[{
  "a": 1,
  "b": 2,
  "c": 3
}, {
  "a": 4,
  "b": 5,
  "c": 6
}]

Output:

a;b;c
1;2;3
4;5;6

Parameters

• format String if set to "strict" the fields will be wrapped with double quote (optional, default standard)
• separator String to indicate the CSV separator (optional, default ";")
• header Boolean first line contains key name (optional, default true)

Returns String

FILELoad

Take Object containing filename et throw content by chunk

[ fi1e1.csv, file2.csv ]

Script:

[use]
plugin = analytics
plugin = basics

[FILELoad]
location = /tmp
[CSVParse]

Output:

[
(...)
]

Parameters

• location String Directory location (optional, default TMPDIR)
• compress Boolean Enable gzip compression (optional, default false)

Returns Object

FILEMerge

Take Object or Buffer and throw only one document

[ fi1e1.csv, file2.csv ]

Script:

[use]
plugin = basics

[FILELoad]
[FILEMerge]
[replace]
path = contentOfFile1AndFile2
value = self()

Output:

[
(...)
]

Returns Object

FILESave

Take data, convert it to buffer and append it to file

Example

Input:

[
  {"a": "a"},
  {"a": "b"},
  {"a": "c" }
]

Script:

[FILESave]
location = /tmp
identifier = toto

Output:

[{ filename: "/tmp/toto", size: XXX, ... }]

Parameters

• location String Directory location (optional, default TMPDIR)
• identifier String? File name
• content String? Content to save instead of using input object
• jsonl Boolean Save as json line (optional, default false)
• compress Boolean Enable gzip compression (optional, default false)
• append Boolean Enable append mode (add content to an existing file) (optional, default false)

Returns Object

INIString

Take Object and generate INI

Take an array of ezs's statements in JSON, and yield an ezs script in a string.

Input:

[
    { "param": 1, "section": { "arg1": "a", "arg2": "b" } },
    { "param": 1, "section": { "arg1": "a", "arg2": "b" } },
    { "section": { "arg1": "a", "arg2": true } },
    { "sec1": { "arg1": "a", "arg2": [3, 4, 5] }, "sec2": { "arg1": "a", "arg2": { "x": 1, "y": 2 } } },
    { "secvide1": {}, "secvide2": {} },
]

Output:

param = 1
[section]
arg1 = a
arg2 = b
param = 1

[section]
arg1 = a
arg2 = b

[section]
arg1 = a
arg2 = true

[sec1]
arg1 = a
arg2 = [3,4,5]

[sec2]
arg1 = a
arg2 = {"x":1,"y":2}

[secvide1]

[secvide2]

Returns String

JSONParse

Parse a String to JSON and generate objects.

See https://github.com/dominictarr/JSONStream

Example 1: with separator

Input:

["{ \"a\": 1, \"b\": 3 }", "{ \"a\": 2, \"b\": 4 }"]

Script:

[JSONParse]
separator = b

Output:

[3, 4]

Example 2: without separator

Input:

["{ \"a\": 1 }", "{ \"a\": 2 }"]

Output:

[1, 2]

Parameters

• separator String to split at every JSONPath found (optional, default "*")

Returns Object

JSONString

Take an Object and generate a JSON string.

Input:

[{ "a": 1 }, { "b": 2 }]

Output:

"[{\"a\":1},{\"b\":2}]"

Parameters

• wrap String every document is wrapped into an array (optional, default true)
• indent String indent JSON (optional, default false)

Returns String

OBJCount

Count how many objects are received, and yield the total.

Input:

["a", "b", "c", "d"]

Output:

[4]

Parameters

• none undefined

Returns Number

OBJFlatten

Flatten an Object with a path delimiting character.

See https://www.npmjs.com/package/flat

Input:

[
  { "a": { "b": 1, "c": 2}},
  { "a": { "b": 3, "c": 4}}
]

Output:

[
  { "a/b": 1, "a/c": 2 },
  { "a/b": 3, "a/c": 4 }
]

Parameters

• separator String choose a character to flatten keys (optional, default "/")
• reverse Boolean unflatten instead of flatten keys (optional, default false)
• safe Boolean preserve arrays and their contents, (optional, default false)

Returns Object

OBJNamespaces

Take Object and throw the same object, all keys parsed to replace namespaces with their prefixes

Note: You can also parse values for specific keys (keys containing references to other keys)

[
  {
   "http://purl.org/dc/terms/title": "Life is good",
   "http://purl.org/ontology/places#Countryl": "France",
 },
 {
   "http://purl.org/dc/terms/title": "The rising sun",
   "http://purl.org/ontology/places#Country": "Japan",
 },
 {
   "http://purl.org/dc/terms/title": "Dolce Vista",
   "http://purl.org/ontology/places#Country": "Italy",
 }
]

Script:

[use]
plugin = basics

[OBJNamespaces]
prefix = dc:
namespace = http://purl.org/dc/terms/

prefix = place:
namespace = http://purl.org/ontology/places#

Output:

[
 {
   "dc:title": "Life is good",
   "place:Country": "France",
 },
 {
   "dc:title": "The rising sun",
   "place:Country": "Japan",
 },
 {
   "dc:title": "Dolce Vista",
   "place:Country": "Italy",
 }
]

Parameters

• prefix String? the alias for a namespace
• namespace String? the namespace to substitute by a prefix
• reference String a regex to find key that contains a namespace to substitute (optional, default null)

Returns Object

OBJStandardize

Standardize Objects so that each object have the same keys.

Input:

[{ "a": 1, "b": 2},
 { "b": 2, "c": 3},
 { "a": 1, "c": 3}]

Output:

[{ "a": 1, "b": 2, "c": ""},
 { "b": 2, "b": "", "c": 3},
 { "a": 1, "b": "", "c": 3}]

Parameters

• none undefined

Returns Object

TARDump

Take all recevied objects and build a tar file

{
}

Parameters

• manifest String? Location path to store files in the tarball
• location String Location path to store files in the tarball (optional, default data)
• json String Convert to JSON the content of each chunk (optional, default true)
• extension String Choose extension fo each file (optional, default json)
• additionalFile String? Path to an additional file that will be add to tarball
• compress Boolean Enable gzip compression (optional, default false)

TARExtract

Take the content of a tar file, extract some files. The JSON object is sent to the output stream for each file. It returns to the output stream

{
   "id": "file name",
   "value": "file contents"
}

Parameters

• path String Regex to select the files to extract (optional, default "**\/*.json")
• json String Parse as JSON the content of each file (optional, default true)
• text Boolean The content of each file is converted to a string (otherwise it remains a buffer) (optional, default true)
• compress Boolean Enable gzip compression (optional, default false)

Returns Array<{id: String, value: String}>

TXTConcat

Concatenate all String items into one string

Input:

["a", "b"]

Output:

["ab"]

Parameters

• none undefined

Returns String

TXTObject

Take an array of values and generate an array containing objects with the given key and matching value from the input array.

Input:

[1, "b"]

Output:

[{ "value": 1 }, { "value": "b" }]

Parameters

• key String choose a the key name (optional, default "value")

Returns Object

TXTParse

Take a String and split it at each separator found.

Input:

["a\nb\n", "c\nd\n"]

Output:

["a", "b", "c", "d"]

Parameters

• separator String choose character which trigger the split (optional, default "\n")

Returns String

TXTZip

Take a String and zip it.

Uses gzip algorithm to compress strings.

Parameters

• unzip Boolean to Unzip input (optional, default false)

Returns Buffer

URLConnect

Take an Object and send it to an URL.

The output will be the returned content of URL.

Useful to send JSON data to an API and get results.

Parameters

• url String? URL to fetch
• streaming String Direct connection to the Object Stream server (disables the retries setting) (optional, default false)
• json String Parse as JSON the content of URL (optional, default false)
• timeout Number Timeout in milliseconds (optional, default 1000)
• noerror Boolean Ignore all errors (optional, default false)
• retries Number The maximum amount of times to retry the connection (optional, default 5)
• encoder String The statement to encode each chunk to a string (optional, default dump)

Returns Object

URLFetch

Add a new field to an Object, with the returned content of URL.

Or if no target is specified, the output will be the returned content of URL.

Parameters

• url String? URL to fetch
• path String? if present select value to send (by POST)
• target String? choose the key to set
• json String parse as JSON the content of URL (optional, default false)
• timeout Number timeout in milliseconds (optional, default 1000)
• mimetype String mimetype for value of path (if presents) (optional, default "application/json")
• noerror Boolean ignore all errors, the target field will remain undefined (optional, default false)
• retries Number The maximum amount of times to retry the connection (optional, default 5)

Returns Object

URLPagination

Take Object and multiple it to make it one object per page

Input:

[{"q": "a"}]

Script:

[URLRequest]
url = https://api.search.net

[URLPagination]
total = get('total')

Output:

[
     {
         "q": "a",
         "total": 22
         "offset": 0,
         "pageNumber": 1,
         "totalPages", 3,
         "maxPages": 1000,
         "limit": 10
     },
     {
         "q": "a",
         "total": 22
         "offset": 10,
         "pageNumber": 2,
         "totalPages", 3,
         "maxPages": 1000,
         "limit": 10
     },
     {
         "q": "a",
         "total": 22
         "offset": 20,
         "pageNumber": 3,
         "totalPages", 3,
         "maxPages": 1000,
         "limit": 10
     }
 ]

Parameters

• total Number total to use for the pagination (optional, default 0)
• limit Number limit to use to pagination (optional, default 10)
• maxPages Number maxPages to use to pagination (optional, default 1000)

Returns Object

URLParse

Take an URL String, parse it and return Object.

Fields of the returned object:

• href
• origin
• protocol
• username
• password
• host
• hostname
• port
• pathname
• search
• hash

URLString statement convert such an object to a string.

See:

• URLString
• https://nodejs.org/api/url.html

Returns Object

URLRequest

Take Object as parameters of URL, throw each chunk from the result

Input:

[{"q": "a"}]

Script:

[URLRequest]
url = https://api.search.net

Output:

[
     {
         "result": "a"
     }
 ]

Parameters

• url String? URL to fetch
• json Boolean parse result as json (optional, default true)
• target String? choose the key to set
• timeout Number Timeout in milliseconds (optional, default 1000)
• noerror Boolean Ignore all errors, the target field will remain undefined (optional, default false)
• retries Number The maximum amount of times to retry the connection (optional, default 5)
• insert String? a header response value in the result

Returns Object

URLStream

Take String as URL, throw each chunk from the result or Take Object as parameters of URL, throw each chunk from the result

Next examples use an API https://httpbin.org/get?a=n returning

{ args: { "a": "n" }}

Example with objects

Input:

[{"a": "a"}, {"a": "b"}, {"a": "c" }]

Script:

[URLStream]
url = https://httpbin.org/get
path = .args

Output:

[{"a": "a"}, {"a": "b"}, {"a": "c" }]

Example with URLs

Input:

[
  "https://httpbin.org/get?a=a",
  "https://httpbin.org/get?a=b",
  "https://httpbin.org/get?a=c"
]

Script:

[URLStream]
path = .args

Output:

[{"a": "a"}, {"a": "b"}, {"a": "c" }]

Parameters

• url String? URL to fetch (by default input string is taken)
• path String choose the path to split JSON result (optional, default "*")
• timeout Number Timeout in milliseconds (optional, default 1000)
• noerror Boolean Ignore all errors, the target field will remain undefined (optional, default false)
• retries Number The maximum amount of times to retry the connection (optional, default 5)

Returns Object

URLString

Take an Object representing an URL and stringify it.

See URLParse

Returns String

XMLConvert

Convert each chunk as XML String to JSON Object

Example 1: XML to JSON (default parameters)

Input:

[
  "<xml>A</xml>",
  "<xml>B</xml>"
]

Output:

[
  { "xml": { "$t": "A" } },
  { "xml": { "$t": "B" } }
]

Example 2: JSON to XML (invert parameter true)

Input:

[
  { "x": { "a": 1 } },
  { "x": { "a": 2 } }
]

Output:

[
  "<x a=\"1\"/>",
  "<x a=\"2\"/>",
]

Example 3: JSON to XML (prologue and invert true)

Input:

[
  { "x": { "a": 1 } },
  { "x": { "a": 2 } }
]

Output:

[
  "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<x a=\"1\"/>",
  "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<x a=\"2\"/>",
]

See https://www.npmjs.com/package/xml-mapping

Parameters

• invert String change conversion (JSON to XML) (optional, default false)
• prologue String add XML prologue (optional, default false)

Returns Object

XMLParse

Take String as XML input, parse it and split it in multi document at each path found

Input:

<* ["<a><b>x</b><b>y</b></a>"]

Script:

[XMLParse]
separator: /a/b

Output:

["x", "y"]

See https://www.npmjs.com/package/xml-splitter

Parameters

• separator String choose a character for flatten keys (optional, default "/")

Returns Object

XMLString

Transform an Object into an XML string.

Input:

[{ "$t": "a" }]

Output:

[
  "<items><item>a</item></items>"
]

See XMLParse

Parameters

• rootElement String Root element name for the tag which starts and close the feed (optional, default "items")
• contentElement String Content element name for the tag which starts and closes each item (optional, default "item")
• rootNamespace String? Namespace for the root tag (xmlns=)
• prologue Boolean Add XML prologue <?xml (optional, default false)

Returns String

ZIPExtract

Take the content of a zip file, extract some files. The JSON object is sent to the output stream for each file. It returns to the output stream

{
   "id": "file name",
   "value": "file contents"
}

Parameters

• path String Regex to select the files to extract (optional, default "**\/*.json")

Returns Array<{id: String, value: String}>