From 8cd33460586245c977ba5dc156f067fc3af3e993 Mon Sep 17 00:00:00 2001 From: Sidita Duli Date: Tue, 3 Oct 2023 13:37:04 +0200 Subject: [PATCH 1/4] udf onnx config Signed-off-by: Sidita Duli --- docs/configuration/udfs-onnx.md | 39 +++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 docs/configuration/udfs-onnx.md diff --git a/docs/configuration/udfs-onnx.md b/docs/configuration/udfs-onnx.md new file mode 100644 index 0000000..f1b3b37 --- /dev/null +++ b/docs/configuration/udfs-onnx.md @@ -0,0 +1,39 @@ +## Configuring UDF on ONNX placeholder +ONNX is an open format built to represent machine learning models. [https://onnx.ai/]. Using ONNX, you can make transformations, and write User Defined Functions (UDFs) in the Dozer configuration file to perform SQL transformation. Let's set these cofigurations step by step. + +#### Write the ONNX model +You can write the ONNX models, by using pre-trained models, services, or by converting models from various frameworks. For more details about the ONNX models follow this tutorial [https://github.com/onnx/models]. + +To convert the model in ONNX format you can use the Pytorch library. +Install Pytorch in your system, in order to use its functionalities. +You can organize the models in a separate folder, named model. In this folder, put the Python file. + +First step, is to run the Python file, with this command: +```bash +python3 onnx_modeling.py +``` +where onnx_modeling.py is the model in Python. As result, it generates the .onnx file that contains data in ONNX format. + +#### Configure the UDFs in Dozer +In the ```dozer_config.yaml``` file, you can configure the udfs, similar to this structure: + +``` bash +udfs: + - name: torch_jit + config: !Onnx + path: ./model/sum.onnx +``` +The ```config: !Onnx``` supports the transformation from a ONNX model to a udfs schema. +The ```path``` refers to the ```.onnx``` file, and the ```torch_jit``` corresponds to the name of the transformation performed. + +#### Perform the SQL transformation +In the ```dozer_config.yaml``` file, you can perform the SQL transformation, similar to this structure: + +```SQL +sql: | + SELECT torch_jit(col1, col2, col3, col4) INTO res + FROM onnx_test; + +``` +This transformation is based on the UDFs structure. The result will be put in the endpoint named ```res```. + From 50ab9bf9a76ec939c4164540100b4adfdc3141b9 Mon Sep 17 00:00:00 2001 From: Sidita Duli Date: Tue, 3 Oct 2023 13:44:16 +0200 Subject: [PATCH 2/4] Update udfs-onnx.md Signed-off-by: Sidita Duli --- docs/configuration/udfs-onnx.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/configuration/udfs-onnx.md b/docs/configuration/udfs-onnx.md index f1b3b37..eb39965 100644 --- a/docs/configuration/udfs-onnx.md +++ b/docs/configuration/udfs-onnx.md @@ -1,18 +1,18 @@ ## Configuring UDF on ONNX placeholder -ONNX is an open format built to represent machine learning models. [https://onnx.ai/]. Using ONNX, you can make transformations, and write User Defined Functions (UDFs) in the Dozer configuration file to perform SQL transformation. Let's set these cofigurations step by step. +ONNX is an open format built to represent machine learning models. [https://onnx.ai/]. Using ONNX, you can make transformations, and write User Defined Functions (UDFs) in the Dozer configuration file to perform SQL transformation. Let's set these configurations step by step. #### Write the ONNX model -You can write the ONNX models, by using pre-trained models, services, or by converting models from various frameworks. For more details about the ONNX models follow this tutorial [https://github.com/onnx/models]. +You can write the ONNX models, by using pre-trained models, and services, or by converting models from various frameworks. For more details about the ONNX models follow this tutorial [https://github.com/onnx/models]. To convert the model in ONNX format you can use the Pytorch library. Install Pytorch in your system, in order to use its functionalities. You can organize the models in a separate folder, named model. In this folder, put the Python file. -First step, is to run the Python file, with this command: +The first step, is to run the Python file, with this command: ```bash python3 onnx_modeling.py ``` -where onnx_modeling.py is the model in Python. As result, it generates the .onnx file that contains data in ONNX format. +where onnx_modeling.py is the model in Python. As a result, it generates the .onnx file that contains data in ONNX format. #### Configure the UDFs in Dozer In the ```dozer_config.yaml``` file, you can configure the udfs, similar to this structure: @@ -23,7 +23,7 @@ udfs: config: !Onnx path: ./model/sum.onnx ``` -The ```config: !Onnx``` supports the transformation from a ONNX model to a udfs schema. +The ```config: !Onnx``` supports the transformation from an ONNX model to a UDFs schema. The ```path``` refers to the ```.onnx``` file, and the ```torch_jit``` corresponds to the name of the transformation performed. #### Perform the SQL transformation @@ -37,3 +37,4 @@ sql: | ``` This transformation is based on the UDFs structure. The result will be put in the endpoint named ```res```. +Check the use case [https://github.com/getdozer/dozer-samples/tree/main/usecases/onnx] for more implementation details. From 41b892113f994ca9e921b3664927cdeb25e1c7c7 Mon Sep 17 00:00:00 2001 From: Sidita Duli Date: Tue, 3 Oct 2023 13:50:13 +0200 Subject: [PATCH 3/4] feat: udf onnx config placeholder #1831 --- docs/configuration/udfs-onnx.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/configuration/udfs-onnx.md b/docs/configuration/udfs-onnx.md index eb39965..9af8efc 100644 --- a/docs/configuration/udfs-onnx.md +++ b/docs/configuration/udfs-onnx.md @@ -1,8 +1,8 @@ ## Configuring UDF on ONNX placeholder -ONNX is an open format built to represent machine learning models. [https://onnx.ai/]. Using ONNX, you can make transformations, and write User Defined Functions (UDFs) in the Dozer configuration file to perform SQL transformation. Let's set these configurations step by step. +Open Neural Network Exchange[(ONNX)](https://onnx.ai/) is an open format built to represent machine learning models. . Using ONNX, you can make transformations, and write User Defined Functions (UDFs) in the Dozer configuration file to perform SQL transformation. Let's set these configurations step by step. #### Write the ONNX model -You can write the ONNX models, by using pre-trained models, and services, or by converting models from various frameworks. For more details about the ONNX models follow this tutorial [https://github.com/onnx/models]. +You can write the ONNX models, by using pre-trained models, and services, or by converting models from various frameworks. For more details about the ONNX models follow this [ONNX tutorial](https://github.com/onnx/models). To convert the model in ONNX format you can use the Pytorch library. Install Pytorch in your system, in order to use its functionalities. @@ -37,4 +37,4 @@ sql: | ``` This transformation is based on the UDFs structure. The result will be put in the endpoint named ```res```. -Check the use case [https://github.com/getdozer/dozer-samples/tree/main/usecases/onnx] for more implementation details. +Check the [use case](https://github.com/getdozer/dozer-samples/tree/main/usecases/onnx) for more implementation details. From 9e24caa0cca277c65febbd52398b58f4d01ea317 Mon Sep 17 00:00:00 2001 From: Sidita Duli Date: Tue, 3 Oct 2023 13:51:34 +0200 Subject: [PATCH 4/4] Update udfs-onnx.md Signed-off-by: Sidita Duli --- docs/configuration/udfs-onnx.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/configuration/udfs-onnx.md b/docs/configuration/udfs-onnx.md index 9af8efc..964915d 100644 --- a/docs/configuration/udfs-onnx.md +++ b/docs/configuration/udfs-onnx.md @@ -1,5 +1,5 @@ ## Configuring UDF on ONNX placeholder -Open Neural Network Exchange[(ONNX)](https://onnx.ai/) is an open format built to represent machine learning models. . Using ONNX, you can make transformations, and write User Defined Functions (UDFs) in the Dozer configuration file to perform SQL transformation. Let's set these configurations step by step. +Open Neural Network Exchange [(ONNX)](https://onnx.ai/) is an open format built to represent machine learning models. . Using ONNX, you can make transformations, and write User Defined Functions (UDFs) in the Dozer configuration file to perform SQL transformation. Let's set these configurations step by step. #### Write the ONNX model You can write the ONNX models, by using pre-trained models, and services, or by converting models from various frameworks. For more details about the ONNX models follow this [ONNX tutorial](https://github.com/onnx/models). @@ -37,4 +37,4 @@ sql: | ``` This transformation is based on the UDFs structure. The result will be put in the endpoint named ```res```. -Check the [use case](https://github.com/getdozer/dozer-samples/tree/main/usecases/onnx) for more implementation details. +Check the [use case](https://github.com/getdozer/dozer-samples/tree/main/usecases/onnx) in Dozer for more implementation details of this UDFs transformation.