diff --git a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/loops-in-apache-hop.adoc b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/loops-in-apache-hop.adoc index 9a37ef5908c..50eb4a8b51d 100644 --- a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/loops-in-apache-hop.adoc +++ b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/loops-in-apache-hop.adoc @@ -98,7 +98,7 @@ All of this combined makes it hard to maintain and debug this type of loops. == Pipeline and Workflow executor -The xref:pipeline/transforms/workflowexecutor.adoc[Workflow executor^] and xref:pipeline/transforms/pipelineexecutor.adoc[Pipeline executor^] offer flexible and elegant ways to run workflows and pipelines from within an existing pipeline. +The xref:pipeline/transforms/workflow-executor.adoc[Workflow executor^] and xref:pipeline/transforms/pipeline-executor.adoc[Pipeline executor^] offer flexible and elegant ways to run workflows and pipelines from within an existing pipeline. === Pipeline Executor diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getdatafromxml.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getdatafromxml.adoc index 3085bbc8848..435b2c29060 100644 --- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getdatafromxml.adoc +++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/getdatafromxml.adoc @@ -29,6 +29,14 @@ The Get Data From XML transform provides the ability to read data from any type Get Data From XML can read data from 3 kind of sources (files, stream and url) in 2 modes (user can define files and urls at static mode or in a dynamic way). +See also: + +* xref:pipeline/transforms/xmlinputstream.adoc[XML Input Stream (StAX)] transform. + +Samples (Samples project): + +* transforms/get-data-from-xml.hpl + | == Supported Engines [%noheader,cols="2,1a",frame=none, role="table-supported-engines"] @@ -47,7 +55,7 @@ Get Data From XML can read data from 3 kind of sources (files, stream and url) i The files tab is where you define the location of the XML files from which you want to read. The table below contains options associated with the Files tab. -[options="header"] +[options="header", cols="1,3"] |=== |Option|Description |Transform name|Name of the transform. @@ -58,8 +66,9 @@ These are read. * Read source as URL : the previous transform is giving URLs in a certain field in the input stream. These are read. * Get XML source from a field : specify the field to read XML, filename or URL from. -|File or directory|Specifies the location and/or name of the input text file. -Note: Click Add to add the file/directory/wildcard combination to the list of selected files (grid) below. +|File or directory a|Specifies the location and/or name of the input text file. + +TIP: Click Add to add the file/directory/wildcard combination to the list of selected files (grid) below. |Regular expression|Specifies the regular expression you want to use to select the files in the directory specified in the previous option. |Selected Files|Contains a list of selected files (or wildcard selections) and a property specifying if file is required or not. If a file is required and it is not found, an error is generated;otherwise, the file name is skipped. @@ -68,7 +77,7 @@ If a file is required and it is not found, an error is generated;otherwise, the === Content Tab -[options="header"] +[options="header", cols="1,3"] |=== |Option|Description |Settings a| @@ -88,7 +97,7 @@ Please see the Example 1 to see how it works. Check this if you want to ignore those altogether. * Do not raise an error if no file: Don't raise a stink if no files are found. * Limit : Limits the number of rows to this number (zero (0) means all rows). -* Prune path to handle large files: almost the same value as the "Loop XPath" property with some exceptions, see Get Data from XML - Handling Large Files for more details. +* Prune path to handle large files: almost the same value as the "Loop XPath" property with some exceptions. Use the prune path to speed up processing of large files, or consider using the xref:pipeline/transforms/xmlinputstream.adoc[XML Input Stream (StAX)] transform. Note that you can use this parameter to avoid multiple HTTP URL requests. |Additional fields a| @@ -105,12 +114,17 @@ A unique list is being kept in memory that can be used in the next workflow acti === Fields Tab -[options="header"] +[options="header", cols="1,3"] |=== |Option|Description |Name|The name of the output field |XPath|The path to the element node or attribute to read |Element|The element type to read: Node or Attribute +|Result Type a|"Value of" or "Single node" + +* Value of: retrieve the value of your XPath expression, e.g. the contents of an element or the value of an attribute. +* Single node: retrieve the XML contents returned by an XPath expression. Contrary to "Value of", the result of "Single node" is an XML snippet. + |Type|The data type to convert to |Format|The format or conversion mask to use in the data type conversion |Length|The length of the output data type diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc index 258d70ea3e5..b4a0bf9a727 100644 --- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc +++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc @@ -56,10 +56,11 @@ See also: * The xref:workflow/actions/workflow.adoc[Workflow action] that executes a sub-workflow from a workflow. * The xref:workflow/actions/pipeline.adoc[Pipeline action] that executes a pipeline from a workflow. -* The xref:pipeline/transforms/workflowexecutor.adoc[Workflow Executor transform] that executes a workflow from a pipeline. +* The xref:pipeline/transforms/workflow-executor.adoc[Workflow Executor transform] that executes a workflow from a pipeline. * The xref:how-to-guides/loops-in-apache-hop.adoc[Loops in Apache Hop] how-to guide. Samples (samples project): + * loops/pipeline-executor.hpl == Options diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-executor.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-executor.adoc index 423a17a04fe..1c0f857d075 100644 --- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-executor.adoc +++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/workflow-executor.adoc @@ -52,10 +52,11 @@ See also: * The xref:workflow/actions/workflow.adoc[Workflow action] that executes a sub-workflow from a workflow. * The xref:workflow/actions/pipeline.adoc[Pipeline action] that executes a pipeline from a workflow. -* The xref:pipeline/transforms/pipelineexecutor.adoc[Pipeline Executor transform] that executes a sub-pipeline from a pipeline. +* The xref:pipeline/transforms/pipeline-executor.adoc[Pipeline Executor transform] that executes a sub-pipeline from a pipeline. * The xref:how-to-guides/loops-in-apache-hop.adoc[Loops in Apache Hop] how-to guide. Samples (samples project) + * loops/workflow-executor.hpl == Options diff --git a/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/loops.adoc b/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/loops.adoc index 95494ab2b41..febfeac6525 100644 --- a/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/loops.adoc +++ b/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/loops.adoc @@ -21,7 +21,7 @@ under the License. Avoid looping in workflows. The easiest way to loop over a set of values, rows, files, ... is to use an Executor transform. -* xref:pipeline/transforms/pipelineexecutor.adoc[Pipeline Executor] : run a pipeline for each input row -* xref:pipeline/transforms/workflowexecutor.adoc[Workflow Executor] : run a workflow for each input row +* xref:pipeline/transforms/pipeline-executor.adoc[Pipeline Executor] : run a pipeline for each input row +* xref:pipeline/transforms/workflow-executor.adoc[Workflow Executor] : run a workflow for each input row You can nicely map field values to parameters of the pipeline or workflow making loops a breeze. diff --git a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/pipeline.adoc b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/pipeline.adoc index e6a9df80888..a8fd1a73b1e 100644 --- a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/pipeline.adoc +++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/pipeline.adoc @@ -35,8 +35,8 @@ For this example, the Pipeline action defines which pipeline to run to populate See also: * The xref:workflow/actions/workflow.adoc[Workflow action] that executes a sub-workflow from a workflow. -* The xref:pipeline/transforms/workflowexecutor.adoc[Workflow Executor transform] that executes a workflow from a pipeline. -* The xref:pipeline/transforms/pipelineexecutor.adoc[Pipeline Executor transform] that executes a sub-pipeline from a pipeline. +* The xref:pipeline/transforms/workflow-executor.adoc[Workflow Executor transform] that executes a workflow from a pipeline. +* The xref:pipeline/transforms/pipeline-executor.adoc[Pipeline Executor transform] that executes a sub-pipeline from a pipeline. == Options diff --git a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/workflow.adoc b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/workflow.adoc index b043efcad6e..50d747e1a91 100644 --- a/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/workflow.adoc +++ b/docs/hop-user-manual/modules/ROOT/pages/workflow/actions/workflow.adoc @@ -40,8 +40,8 @@ It is better to create smaller workflows and compose them. See also: * The xref:workflow/actions/pipeline.adoc[Pipeline action] that executes a pipeline from a workflow. -* The xref:pipeline/transforms/workflowexecutor.adoc[Workflow Executor transform] that executes a workflow from a pipeline. -* The xref:pipeline/transforms/pipelineexecutor.adoc[Pipeline Executor transform] that executes a sub-pipeline from a pipeline. +* The xref:pipeline/transforms/workflow-executor.adoc[Workflow Executor transform] that executes a workflow from a pipeline. +* The xref:pipeline/transforms/pipeline-executor.adoc[Pipeline Executor transform] that executes a sub-pipeline from a pipeline. == Options diff --git a/plugins/misc/testing/src/main/java/org/apache/hop/testing/gui/TestingGuiPlugin.java b/plugins/misc/testing/src/main/java/org/apache/hop/testing/gui/TestingGuiPlugin.java index 7c243df2e56..b2c4bbd5f39 100644 --- a/plugins/misc/testing/src/main/java/org/apache/hop/testing/gui/TestingGuiPlugin.java +++ b/plugins/misc/testing/src/main/java/org/apache/hop/testing/gui/TestingGuiPlugin.java @@ -175,7 +175,7 @@ public void setInputDataSet(HopGuiPipelineTransformContext context) { Collections.sort(setNames); EnterSelectionDialog esd = new EnterSelectionDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), setNames.toArray(new String[setNames.size()]), BaseMessages.getString(PKG, "TestingGuiPlugin.ContextAction.SetInputDataset.Header"), BaseMessages.getString( @@ -192,7 +192,7 @@ public void setInputDataSet(HopGuiPipelineTransformContext context) { } } catch (Exception e) { new ErrorDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), BaseMessages.getString( PKG, "TestingGuiPlugin.ContextAction.SetInputDataset.Error.Header"), BaseMessages.getString( @@ -230,7 +230,7 @@ private boolean setInputDataSetOnTransform( String[] setFieldNames = setFields.getFieldNames(); EnterMappingDialog mappingDialog = - new EnterMappingDialog(hopGui.getDisplay().getActiveShell(), setFieldNames, transformFieldNames); + new EnterMappingDialog(hopGui.getShell(), setFieldNames, transformFieldNames); List mappings = mappingDialog.open(); if (mappings == null) { return false; @@ -249,7 +249,7 @@ private boolean setInputDataSetOnTransform( } EditRowsDialog orderDialog = new EditRowsDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), SWT.NONE, BaseMessages.getString(PKG, "TestingGuiPlugin.SortOrder.Title"), BaseMessages.getString(PKG, "TestingGuiPlugin.SortOrder.Message"), @@ -330,7 +330,7 @@ public void clearInputDataSet(HopGuiPipelineTransformContext context) { context.getPipelineGraph().updateGui(); } catch (Exception e) { new ErrorDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), BaseMessages.getString( PKG, "TestingGuiPlugin.ContextAction.ClearInputDataset.Error.Header"), BaseMessages.getString( @@ -349,7 +349,7 @@ private boolean checkTestPresent(HopGui hopGui, PipelineMeta pipelineMeta) { // there is no test defined of selected in the pipeline. // Show a warning // - MessageBox box = new MessageBox(hopGui.getDisplay().getActiveShell(), SWT.OK | SWT.ICON_INFORMATION); + MessageBox box = new MessageBox(hopGui.getShell(), SWT.OK | SWT.ICON_INFORMATION); box.setMessage( BaseMessages.getString(PKG, "TestingGuiPlugin.ContextAction.CheckTestPresent.Message")); box.setText( @@ -395,7 +395,7 @@ public void setGoldenDataSet(HopGuiPipelineTransformContext context) { Collections.sort(setNames); EnterSelectionDialog esd = new EnterSelectionDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), setNames.toArray(new String[setNames.size()]), BaseMessages.getString(PKG, "TestingGuiPlugin.ContextAction.SetGoldenDataset.Header"), BaseMessages.getString( @@ -412,7 +412,7 @@ public void setGoldenDataSet(HopGuiPipelineTransformContext context) { } } catch (Exception e) { new ErrorDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), BaseMessages.getString( PKG, "TestingGuiPlugin.ContextAction.SetGoldenDataset.Error.Header"), BaseMessages.getString( @@ -545,7 +545,7 @@ public void clearGoldenDataSet(HopGuiPipelineTransformContext context) { saveUnitTest(variables, hopGui.getMetadataProvider(), currentUnitTest, pipelineMeta); } catch (Exception e) { new ErrorDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), BaseMessages.getString( PKG, "TestingGuiPlugin.ContextAction.ClearGoldenDataset.Error.Header"), BaseMessages.getString( @@ -655,7 +655,7 @@ public void createDataSetFromTransform(HopGuiPipelineTransformContext context) { // to use it on the transform? // MessageBox box = - new MessageBox(hopGui.getDisplay().getActiveShell(), SWT.YES | SWT.NO | SWT.CANCEL | SWT.ICON_QUESTION); + new MessageBox(hopGui.getShell(), SWT.YES | SWT.NO | SWT.CANCEL | SWT.ICON_QUESTION); box.setText( BaseMessages.getString( PKG, "TestingGuiPlugin.ContextAction.CreateDataset.DatasetType.Header")); @@ -689,7 +689,7 @@ public void createDataSetFromTransform(HopGuiPipelineTransformContext context) { } } catch (Exception e) { new ErrorDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), BaseMessages.getString(PKG, "TestingGuiPlugin.ContextAction.CreateDataset.Error.Header"), BaseMessages.getString(PKG, "TestingGuiPlugin.ContextAction.CreateDataset.Error.Message"), e); @@ -719,7 +719,7 @@ public void writeTransformDataToDataSet(HopGuiPipelineTransformContext context) PipelineMeta pipelineMeta = context.getPipelineMeta(); if (pipelineMeta.hasChanged()) { - MessageBox box = new MessageBox(hopGui.getDisplay().getActiveShell(), SWT.OK | SWT.ICON_INFORMATION); + MessageBox box = new MessageBox(hopGui.getShell(), SWT.OK | SWT.ICON_INFORMATION); box.setText( BaseMessages.getString( PKG, "TestingGuiPlugin.ContextAction.Run.SavePipelineDialog.Header")); @@ -739,7 +739,7 @@ public void writeTransformDataToDataSet(HopGuiPipelineTransformContext context) Collections.sort(setNames); EnterSelectionDialog esd = new EnterSelectionDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), setNames.toArray(new String[setNames.size()]), BaseMessages.getString(PKG, "TestingGuiPlugin.ContextAction.Run.ActionList.Item1"), BaseMessages.getString(PKG, "TestingGuiPlugin.ContextAction.Run.ActionList.Item2")); @@ -765,7 +765,7 @@ public void writeTransformDataToDataSet(HopGuiPipelineTransformContext context) // Ask for the mapping between the output row and the data set field // EnterMappingDialog mappingDialog = - new EnterMappingDialog(hopGui.getDisplay().getActiveShell(), transformFields, setFields); + new EnterMappingDialog(hopGui.getShell(), transformFields, setFields); List mapping = mappingDialog.open(); if (mapping == null) { return; @@ -790,7 +790,7 @@ public void writeTransformDataToDataSet(HopGuiPipelineTransformContext context) } catch (Exception e) { new ErrorDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), BaseMessages.getString(PKG, "TestingGuiPlugin.ContextAction.Run.Error.Header"), BaseMessages.getString(PKG, "TestingGuiPlugin.ContextAction.Run.Error.Message"), e); @@ -850,7 +850,7 @@ public void detachUnitTest() { pipelineGraph.updateGui(); } catch (Exception e) { new ErrorDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), BaseMessages.getString(PKG, "TestingGuiPlugin.ToolbarElement.Detach.Error.Header"), BaseMessages.getString(PKG, "TestingGuiPlugin.ToolbarElement.Detach.Error.Message"), e); @@ -935,7 +935,7 @@ public void deleteUnitTest() { return; // doesn't exist } - MessageBox box = new MessageBox(hopGui.getDisplay().getActiveShell(), SWT.YES | SWT.NO | SWT.ICON_QUESTION); + MessageBox box = new MessageBox(hopGui.getShell(), SWT.YES | SWT.NO | SWT.ICON_QUESTION); box.setMessage( BaseMessages.getString( PKG, @@ -960,7 +960,7 @@ public void deleteUnitTest() { refreshUnitTestsList(); } catch (Exception e) { new ErrorDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), BaseMessages.getString(PKG, "TestingGuiPlugin.ToolbarElement.Delete.Error.Header"), BaseMessages.getString(PKG, "TestingGuiPlugin.ToolbarElement.Delete.Error.Message"), e); @@ -1021,7 +1021,7 @@ public void editPipelineUnitTest() { } } catch (Exception e) { new ErrorDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), BaseMessages.getString(PKG, "TestingGuiPlugin.ToolbarElement.UnitTest.Error.Header"), BaseMessages.getString( PKG, "TestingGuiPlugin.ToolbarElement.UnitTest.Error.Message", unitTestName), @@ -1144,7 +1144,7 @@ public void selectUnitTest() { } } catch (Exception e) { new ErrorDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), BaseMessages.getString( PKG, "TestingGuiPlugin.ToolbarElement.GetUnitTestList.Error.Header"), BaseMessages.getString( @@ -1304,7 +1304,7 @@ private void tweakUnitTestTransform( hopGui.getActiveFileTypeHandler().updateGui(); } catch (Exception exception) { new ErrorDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), BaseMessages.getString(PKG, "TestingGuiPlugin.TweakUnitTestTransform.Error.Header"), BaseMessages.getString( PKG, @@ -1347,7 +1347,7 @@ public RowMetaAndData selectUnitTestFromAllTests() { // SelectRowDialog dialog = new SelectRowDialog( - hopGui.getDisplay().getActiveShell(), new Variables(), SWT.DIALOG_TRIM | SWT.MAX | SWT.RESIZE, rows); + hopGui.getShell(), new Variables(), SWT.DIALOG_TRIM | SWT.MAX | SWT.RESIZE, rows); RowMetaAndData selection = dialog.open(); if (selection != null) { return selection; @@ -1355,7 +1355,7 @@ public RowMetaAndData selectUnitTestFromAllTests() { return null; } catch (Exception e) { new ErrorDialog( - hopGui.getDisplay().getActiveShell(), + hopGui.getShell(), BaseMessages.getString(PKG, "TestingGuiPlugin.SelectUnitTestFromAllTests.Error.Header"), BaseMessages.getString(PKG, "TestingGuiPlugin.SelectUnitTestFromAllTests.Error.Message"), e); diff --git a/plugins/transforms/xml/src/main/samples/transforms/get-data-from-xml.hpl b/plugins/transforms/xml/src/main/samples/transforms/get-data-from-xml.hpl new file mode 100644 index 00000000000..f3eaa932ea5 --- /dev/null +++ b/plugins/transforms/xml/src/main/samples/transforms/get-data-from-xml.hpl @@ -0,0 +1,353 @@ + + + + + get-data-from-xml + Y + + + + Normal + + + N + 1000 + 100 + - + 2023/07/17 19:26:28.962 + - + 2023/07/17 19:26:28.962 + + + + 251 + 232 + 201 + 90 + 58 + 14 + N + 90 + 58 + 14 + N + Noto Sans + 10 + 27 + 144 + 96 + Sample XML taken from https://www.javatpoint.com/xml-example + 371 + + + + + sample xml + Get data from XML + Y + + + Get data from XML + clean + Y + + + clean + preview + Y + + + + Get data from XML + getXMLData + + Y + + 1 + + none + + + N + + N + N + N + N + N + N + N + N + Y + + UTF-8 + + ${Internal.Pipeline.Filename.Name} + + + N + N + + + + category + @category + attribut + valueof + String + + + + + -1 + -1 + none + N + + + language + title/@lang + attribut + valueof + String + + + + + -1 + -1 + none + N + + + title + title + node + valueof + String + + + + + -1 + -1 + none + N + + + author + author + node + valueof + String + + + + + -1 + -1 + none + N + + + year + year + node + valueof + Integer + + + + + -1 + -1 + none + N + + + price + price + node + valueof + Number + + + + + -1 + -1 + none + N + + + element + . + node + singlenode + String + + + + + -1 + -1 + none + N + + + 0 + /bookstore/book + Y + N + xml + + + + + + + + + + + + 304 + 144 + + + + sample xml + DataGrid + + Y + + 1 + + none + + + + + <bookstore> + <book category="COOKING"> + <title lang="en">Everyday Italian</title> + <author>Giada De Laurentiis</author> + <year>2005</year> + <price>30.00</price> + </book> + <book category="CHILDREN"> + <title lang="en">Harry Potter</title> + <author>J K. Rowling</author> + <year>2005</year> + <price>29.99</price> + </book> + <book category="WEB"> + <title lang="en">Learning XML</title> + <author>Erik T. Ray</author> + <year>2003</year> + <price>39.95</price> + </book> +</bookstore> + + + + + -1 + -1 + N + xml + String + + + + + 144 + 144 + + + + clean + SelectValues + + Y + + 1 + + none + + + + + category + + + + language + + + + title + + + + author + + + + year + + + + price + + + + element + + + N + + + + 480 + 144 + + + + preview + Dummy + + Y + + 1 + + none + + + + + 640 + 144 + + + + + +