From 689f41ee8770699d74e5356548b63064d14d469b Mon Sep 17 00:00:00 2001 From: TobiasNx <61879957+TobiasNx@users.noreply.github.com> Date: Tue, 9 Apr 2024 21:29:09 +0200 Subject: [PATCH 1/3] Update Fix-function-and-Cookbook.md This brings the documentation up to the status of metafacture fix release 1.0.0 https://github.com/metafacture/metafacture-fix/releases/tag/1.0.0 --- Fix-function-and-Cookbook.md | 76 +++++++++++++++++++++++++++++++++++- 1 file changed, 75 insertions(+), 1 deletion(-) diff --git a/Fix-function-and-Cookbook.md b/Fix-function-and-Cookbook.md index 8c882e5..490246c 100644 --- a/Fix-function-and-Cookbook.md +++ b/Fix-function-and-Cookbook.md @@ -1,4 +1,4 @@ -This page is a replication of the passage of the Fix Readme.md. +This page is a replication of the passage of the Fix Readme.md. Status: [Fix Release 1.0.0](https://github.com/metafacture/metafacture-fix/releases/tag/1.0.0) ## Functions and cookbook @@ -66,6 +66,8 @@ Does nothing. It is used for benchmarking in Catmandu. nothing() ``` +[Example in Playground](https://metafacture.org/playground/?example=nothing) + ##### `put_filemap` Defines an external map for [lookup](#lookup) from a file or a URL. Maps with more than 2 columns are supported but are reduced to a defined key and a value column. @@ -74,6 +76,8 @@ Defines an external map for [lookup](#lookup) from a file or a URL. Maps with mo put_filemap("", "", sep_char: "\t") ``` +[Example in Playground](https://metafacture.org/playground/?example=put_filemap) + The separator (`sep_char`) will vary depending on the source file, e.g.: | Type | Separator | @@ -103,6 +107,8 @@ put_map("", ) ``` +[Example in Playground](https://metafacture.org/playground/?example=put_map) + ##### `put_rdfmap` Defines an external RDF map for lookup from a file or an HTTP(S) resource. @@ -114,6 +120,8 @@ put_rdfmap("", "", target: "") put_rdfmap("", "", target: "", select_language: "") ``` +[Example in Playground](https://metafacture.org/playground/?example=put_rdfmap) + ##### `put_var` Defines a single global variable that can be referenced with `$[]`. @@ -122,6 +130,8 @@ Defines a single global variable that can be referenced with `$[]` put_var("", "") ``` +[Example in Playground](https://metafacture.org/playground/?example=put_var) + ##### `put_vars` Defines multiple global variables that can be referenced with `$[]`. @@ -133,6 +143,8 @@ put_vars( ) ``` +[Example in Playground](https://metafacture.org/playground/?example=put_vars) + #### Record-level functions ##### `add_field` @@ -143,6 +155,8 @@ Creates a field with a defined value. add_field("", "") ``` +[Example in Playground](https://metafacture.org/playground/?example=add_field) + ##### `array` Converts a hash/object into an array. @@ -177,6 +191,8 @@ end call_macro(""[, ...]) ``` +[Example in Playground](https://metafacture.org/playground/?example=call_macro) + ##### `copy_field` Copies a field from an existing field. @@ -185,6 +201,9 @@ Copies a field from an existing field. copy_field("", "") ``` +[Example in Playground](https://metafacture.org/playground/?example=copy_field) + + ##### `format` Replaces the value with a formatted (`sprintf`-like) version. @@ -217,6 +236,9 @@ Moves a field from an existing field. Can be used to rename a field. move_field("", "") ``` +[Example in Playground](https://metafacture.org/playground/?example=move_field) + + ##### `parse_text` Parses a text into an array or hash of values. @@ -248,6 +270,8 @@ paste("my.string", "~Hi", "a", "~how are you?") # "my.string": "Hi eeny how are you?" ``` +[Example in Playground](https://metafacture.org/playground/?example=paste) + ##### `print_record` Prints the current record as JSON either to standard output or to a file. @@ -297,6 +321,8 @@ Removes a field. remove_field("") ``` +[Example in Playground](https://metafacture.org/playground/?example=remove_field) + ##### `rename` Replaces a regular expression pattern in subfield names of a field. Does not change the name of the source field itself. @@ -305,6 +331,8 @@ Replaces a regular expression pattern in subfield names of a field. Does not cha rename("", "", "") ``` +[Example in Playground](https://metafacture.org/playground/?example=rename) + ##### `retain` Deletes all fields except the ones listed (incl. subfields). @@ -313,6 +341,8 @@ Deletes all fields except the ones listed (incl. subfields). retain(""[, ...]) ``` +[Example in Playground](https://metafacture.org/playground/?example=retain) + ##### `set_array` Creates a new array (with optional values). @@ -322,6 +352,8 @@ set_array("") set_array("", ""[, ...]) ``` +[Example in Playground](https://metafacture.org/playground/?example=set_array) + ##### `set_field` Creates (or replaces) a field with a defined value. @@ -361,6 +393,8 @@ Deletes empty fields, arrays and objects. vacuum() ``` +[Example in Playground](https://metafacture.org/playground/?example=vacuum) + #### Field-level functions ##### `append` @@ -371,6 +405,8 @@ Adds a string at the end of a field value. append("", "") ``` +[Example in Playground](https://metafacture.org/playground/?example=append) + ##### `capitalize` Upcases the first character in a field value. @@ -379,6 +415,8 @@ Upcases the first character in a field value. capitalize("") ``` +[Example in Playground](https://metafacture.org/playground/?example=capitalize) + ##### `count` Counts the number of elements in an array or a hash and replaces the field value with this number. @@ -395,6 +433,8 @@ Downcases all characters in a field value. downcase("") ``` +[Example in Playground](https://metafacture.org/playground/?example=downcase) + ##### `filter` Only keeps field values that match the regular expression pattern. Works only with array of strings/repeated fields. @@ -411,6 +451,8 @@ Flattens a nested array field. flatten("") ``` +[Example in Playground](https://metafacture.org/playground/?example=flatten) + ##### `from_json` Replaces the string with its JSON deserialization. @@ -445,6 +487,8 @@ Options: isbn(""[, to: ""][, verify_check_digit: ""][, error_string: ""]) ``` +[Example in Playground](https://metafacture.org/playground/?example=isbn) + ##### `join_field` Joins an array of strings into a single string. @@ -453,6 +497,8 @@ Joins an array of strings into a single string. join_field("", "") ``` +[Example in Playground](https://metafacture.org/playground/?example=join_field) + ##### `lookup` Looks up matching values in a map and replaces the field value with this match. [External files](#put_filemap), [internal maps](#put_map) as well as [RDF resources](#put_rdfmap) can be used. @@ -520,6 +566,8 @@ Adds a string at the beginning of a field value. prepend("", "") ``` +[Example in Playground](https://metafacture.org/playground/?example=prepend) + ##### `replace_all` Replaces a regular expression pattern in field values with a replacement string. Regexp capturing is possible; refer to capturing groups by number (`$`) or name (`${}`). @@ -528,6 +576,8 @@ Replaces a regular expression pattern in field values with a replacement string. replace_all("", "", "") ``` +[Example in Playground](https://metafacture.org/playground/?example=replace_all) + ##### `reverse` Reverses the character order of a string or the element order of an array. @@ -546,6 +596,8 @@ sort_field("", reverse: "true") sort_field("", numeric: "true") ``` +[Example in Playground](https://metafacture.org/playground/?example=sort_field) + ##### `split_field` Splits a string into an array and replaces the field value with this array. @@ -554,6 +606,8 @@ Splits a string into an array and replaces the field value with this array. split_field("", "") ``` +[Example in Playground](https://metafacture.org/playground/?example=split_field) + ##### `substring` Replaces a string with its substring as defined by the start position (offset) and length. @@ -591,6 +645,8 @@ Deletes whitespace at the beginning and the end of a field value. trim("") ``` +[Example in Playground](https://metafacture.org/playground/?example=trim) + ##### `uniq` Deletes duplicate values in an array. @@ -599,6 +655,9 @@ Deletes duplicate values in an array. uniq("") ``` +[Example in Playground](https://metafacture.org/playground/?example=uniq) + + ##### `upcase` Upcases all characters in a field value. @@ -607,6 +666,8 @@ Upcases all characters in a field value. upcase("") ``` +[Example in Playground](https://metafacture.org/playground/?example=upcase) + ##### `uri_encode` Encodes a field value as URI. Aka percent-encoding. @@ -638,6 +699,8 @@ if end ``` +[Example in Playground](https://metafacture.org/playground/?example=reject) + ### Binds #### `do list` @@ -650,6 +713,8 @@ do list(path: "") end ``` +[Example in Playground](https://metafacture.org/playground/?example=do_list) + Only the current element is accessible in this case (as the root element). When specifying a variable name for the current element, the record remains accessible as the root element and the current element is accessible through the variable name: @@ -660,6 +725,8 @@ do list(path: "", "var": "") end ``` +[Example in Playground](https://metafacture.org/playground/?example=do_list_with_var) + #### `do list_as` Iterates over each _named_ element of an array (like [`do list`](#do-list) with a variable name). If multiple arrays are given, iterates over the _corresponding_ elements from each array (i.e., all elements with the same array index, skipping elements whose arrays have already been exhausted). @@ -693,6 +760,8 @@ do once() end ``` +[Example in Playground](https://metafacture.org/playground/?example=do_once) + In order to execute multiple blocks only once, tag them with unique identifiers: ```perl @@ -729,6 +798,8 @@ end call_macro(""[, ...]) ``` +[Example in Playground](https://metafacture.org/playground/?example=do_put_macro) + ### Conditionals Conditionals start with `if` in case of affirming the condition or `unless` rejecting the condition. @@ -865,3 +936,6 @@ Executes the functions if/unless the field value does not match the regular expr Executes the functions if/unless the string matches the regular expression pattern. +## Xtext + +This repo has been originally set up with [Xtext](https://www.eclipse.org/Xtext/) 2.17.0 and Eclipse for Java 2019-03, following [https://www.eclipse.org/Xtext/documentation/104_jvmdomainmodel.html](https://www.eclipse.org/Xtext/documentation/104_jvmdomainmodel.html). From b6293eeb3b3ad620286dd44ac8ae4813db2787ec Mon Sep 17 00:00:00 2001 From: TobiasNx <61879957+TobiasNx@users.noreply.github.com> Date: Thu, 18 Apr 2024 14:18:30 +0200 Subject: [PATCH 2/3] Update linksAndExamples.tsv --- linksAndExamples.tsv | 2 ++ 1 file changed, 2 insertions(+) diff --git a/linksAndExamples.tsv b/linksAndExamples.tsv index 580d6b4..f04a242 100644 --- a/linksAndExamples.tsv +++ b/linksAndExamples.tsv @@ -54,6 +54,8 @@ handle-picaxml https://github.com/metafacture/metafacture-core/blob/master/metaf jscript https://github.com/metafacture/metafacture-core/blob/master/metafacture-scripting/src/main/java/org/metafacture/scripting/JScriptObjectPipe.java json-to-elasticsearch-bulk https://github.com/metafacture/metafacture-core/blob/master/metafacture-elasticsearch/src/main/java/org/metafacture/elasticsearch/JsonToElasticsearchBulk.java lines-to-records https://github.com/metafacture/metafacture-core/blob/master/metafacture-strings/src/main/java/org/metafacture/strings/LineRecorder.java https://metafacture.org/playground/?example=lines-to-records +list-fix-path https://github.com/metafacture/metafacture-fix/blob/master/metafix/src/main/java/org/metafacture/metafix/ListFixPaths.java https://metafacture.org/playground/?example=list-fix-paths +list-fix-values https://github.com/metafacture/metafacture-fix/blob/master/metafix/src/main/java/org/metafacture/metafix/ListFixValues.java https://metafacture.org/playground/?example=list-fix-values literal-to-object https://github.com/metafacture/metafacture-core/blob/master/metafacture-mangling/src/main/java/org/metafacture/mangling/LiteralToObject.java https://metafacture.org/playground/?example=literal-to-object log-object https://github.com/metafacture/metafacture-core/blob/master/metafacture-monitoring/src/main/java/org/metafacture/monitoring/ObjectLogger.java log-stream https://github.com/metafacture/metafacture-core/blob/master/metafacture-monitoring/src/main/java/org/metafacture/monitoring/StreamLogger.java From b1a399685fa76e91b36c0179aed93033c4fff91c Mon Sep 17 00:00:00 2001 From: TobiasNx <61879957+TobiasNx@users.noreply.github.com> Date: Fri, 19 Apr 2024 13:56:01 +0200 Subject: [PATCH 3/3] Use marc example without encoding problems #29 (#39) --- MF-in-5-min.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/MF-in-5-min.md b/MF-in-5-min.md index b4a2b5d..843171c 100644 --- a/MF-in-5-min.md +++ b/MF-in-5-min.md @@ -161,12 +161,12 @@ retain("title", "id") | as-records | decode-marc21 | fix("titles.fix") -| encode-csv +| encode-csv(includeHeader="true") | print ; ``` -[Example in Playground](https://metafacture.org/playground/?flux=%22https%3A//raw.githubusercontent.com/metafacture/metafacture-core/master/metafacture-runner/src/main/dist/examples/read/marc21/10.marc21%22%0A%7C+open-http%0A%7C+as-lines%0A%7C+decode-marc21%0A%7C+fix%28transformationFile%29%0A%7C+encode-csv%0A%7C+print%0A%3B&transformation=set_array%28%22title%22%29%0Acopy_field%28%22245%3F%3F.%3F%22%2C%22title.%24append%22%29%0Ajoin_field%28%22title%22%29%0Acopy_field%28%22001%22%2C%22id%22%29%0Aretain%28%22title%22%2C+%22id%22%29) +[Example in Playground](https://metafacture.org/playground/?flux=%22https%3A//raw.githubusercontent.com/metafacture/metafacture-tutorial/main/data/sample.marc21%22%0A%7C+open-http%0A%7C+as-lines%0A%7C+decode-marc21%0A%7C+fix%28transformationFile%29%0A%7C+encode-csv%28includeheader%3D%22true%22%29%0A%7C+print%0A%3B&transformation=set_array%28%22title%22%29%0Acopy_field%28%22245%3F%3F.%3F%22%2C%22title.%24append%22%29%0Ajoin_field%28%22title%22%29%0Acopy_field%28%22001%22%2C%22id%22%29%0Aretain%28%22title%22%2C+%22id%22%29) In the example above marc data is converted to a csv file. The 245 field with its subfields of each MARC record is mapped to the title field.