Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Good First Issue][TF FE]: Support complex tensors for AddN operation #22944

Closed
rkazants opened this issue Feb 20, 2024 · 8 comments
Closed

[Good First Issue][TF FE]: Support complex tensors for AddN operation #22944

rkazants opened this issue Feb 20, 2024 · 8 comments
Assignees
Labels
category: TF FE OpenVINO TensorFlow FrontEnd good first issue Good for newcomers gsoc-prerequisite-task Prerequisite task related to Google Summer of Code projects no_stale Do not mark as stale
Milestone

Comments

@rkazants
Copy link
Contributor

Context

OpenVINO component responsible for support of TensorFlow models is called as TensorFlow Frontend (TF FE). TF FE converts a model represented in TensorFlow opset to a model in OpenVINO opset.
Some audio models use tensors of complex type. Complex type tensor is a tensor that has elements of complex type. For example, 1D tensor with three elements x = [1+2j, 2, -2j].

For supporting AddN operation on complex type tensor, you need to extend the corresponding loader for AddN.

What needs to be done?

The existing loader for AddN needs to be extended by propagating ComplexTypeMark from input to output and to represent output complex type tensor as a floating-point type tensor with auxiliary dimension that concatenates real and imaginary parts of complex tensor.
To validate the extension, the corresponding layer test needs to be updated with complex tensor cases.

Here is an example of how to extend Reshape loader to support complex type tensors:

OutputVector translate_reshape_op(const NodeContext& node) {
    default_op_checks(node, 2, {"Reshape"}, true);
    auto tensor = node.get_input(0);
    auto complex_type_mark = as_type_ptr<ComplexTypeMark>(tensor.get_node_shared_ptr());
    auto shape = node.get_input(1);
    if (complex_type_mark) {
        element::Type complex_part_type = complex_type_mark->get_complex_part_type();
        tensor = complex_type_mark->input_value(0);

        OutputVector concat_inputs;
        concat_inputs.push_back(shape);
        concat_inputs.push_back(make_shared<v0::Constant>(shape.get_element_type(), Shape{1}, 2));

        auto concat = make_shared<v0::Concat>(concat_inputs, 0);
        auto reshape = make_shared<v1::Reshape>(tensor, concat, false);
        set_node_name(node.get_name(), reshape);
        auto complex_reshape = make_shared<ComplexTypeMark>(reshape, complex_part_type);
        return {complex_reshape->output(0)};
    }

    auto reshape = make_shared<v1::Reshape>(tensor, shape, false);
    set_node_name(node.get_name(), reshape);
    return {reshape};
}

Since OpenVINO does not have native support of complex tensors, we handle complex type in intermediate layers by representing them as a floating-point type with additional dimension (specially created) to store real and imaginary parts of the original complex tensor so slicing by the last dimension will give either real or imaginary parts: x[...,0] - real and x[...,1] - imaginary parts.

On the first step, we update default_op_checks with true flag to indicate that loader for Reshape operation now handles complex tensors:

default_op_checks(node, 2, {"Reshape"}, true);

Secondly, we check if complex type mark exists by anticipated inputs. This mark indicates that input tensor of complex type:

auto complex_type_mark = as_type_ptr<ComplexTypeMark>(tensor.get_node_shared_ptr());

Thirdly, we retrieve a floating-point tensor (with additional dimension to store real and imaginary parts) simulating complex tensor:

tensor = complex_type_mark->input_value(0);

After that, we implement conversion for Reshape for this particular case. Since a floating-point tensor simulating complex tensor has additional dimension equal to 2,
we update input target shape by appending 2 value and perform reshape on a floating-point tensor simulating complex tensor.

Finally, since Reshape should produce complex tensor by output we insert a new mark ComplexTypeMark into the output.

To validate support of complex tensors for Reshape, the new layer test TestComplexReshape was added.

Example how to run the layer test:

export TEST_DEVICE=CPU
cd openvino/tests/layer_tests/tensorflow_tests
pytest test_tf_Reshape.py

Example Pull Requests

Resources

Contact points

  • @openvinotoolkit/openvino-tf-frontend-maintainers
  • rkazants in Discord

Ticket

No response

@rkazants rkazants added no_stale Do not mark as stale category: TF FE OpenVINO TensorFlow FrontEnd good first issue Good for newcomers labels Feb 20, 2024
@github-project-automation github-project-automation bot moved this to Contributors Needed in Good first issues Feb 20, 2024
@dnabanita7
Copy link

dnabanita7 commented Feb 20, 2024

.take

Copy link
Contributor

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

@rkazants rkazants moved this from Contributors Needed to Assigned in Good first issues Feb 20, 2024
@rkazants rkazants added the gsoc-prerequisite-task Prerequisite task related to Google Summer of Code projects label Feb 22, 2024
@p-wysocki
Copy link
Contributor

Hello @dnabanita7, can we help you with anything?

@dnabanita7
Copy link

Hello @p-wysocki ! My bad! I forgot to create a PR. I did it now.

@rkazants rkazants linked a pull request Mar 3, 2024 that will close this issue
@mlukasze mlukasze moved this from Assigned to In Review in Good first issues Mar 4, 2024
@rkazants rkazants moved this from In Review to Contributors Needed in Good first issues Mar 19, 2024
@LucaTamSapienza
Copy link
Contributor

.take

Copy link
Contributor

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

@Jessielovecodings
Copy link

#WLB#
Can I take this issue?

@mlukasze mlukasze moved this from Contributors Needed to Assigned in Good first issues Mar 20, 2024
github-merge-queue bot pushed a commit that referenced this issue Mar 28, 2024
### Details:
- extended addN.cpp file to support complex tensor
- extended test_tf_addN.py with test for complex input

### Tickets:
 - #22944

---------

Co-authored-by: Roman Kazantsev <[email protected]>
@rkazants
Copy link
Contributor Author

@LucaTamSapienza, congrats. Keep this way. Your PR resolved this GFI.

Best regards,
Roman

@github-project-automation github-project-automation bot moved this from Assigned to Closed in Good first issues Mar 29, 2024
@mlukasze mlukasze added this to the 2024.1 milestone Mar 29, 2024
bbielawx pushed a commit to bbielawx/openvino that referenced this issue Apr 12, 2024
### Details:
- extended addN.cpp file to support complex tensor
- extended test_tf_addN.py with test for complex input

### Tickets:
 - openvinotoolkit#22944

---------

Co-authored-by: Roman Kazantsev <[email protected]>
alvoron pushed a commit to alvoron/openvino that referenced this issue Apr 29, 2024
### Details:
- extended addN.cpp file to support complex tensor
- extended test_tf_addN.py with test for complex input

### Tickets:
 - openvinotoolkit#22944

---------

Co-authored-by: Roman Kazantsev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: TF FE OpenVINO TensorFlow FrontEnd good first issue Good for newcomers gsoc-prerequisite-task Prerequisite task related to Google Summer of Code projects no_stale Do not mark as stale
Projects
Archived in project
Development

No branches or pull requests

6 participants