[NPU] Switching the I/O identification convention to indices #24248

razvanapetroaie · 2024-04-25T15:09:55Z

Details:

Please see PR#10348 (compiler repository) for a detailed description and some extra validation.

Tickets:

CVS-142751

src/plugins/intel_npu/src/compiler/src/zero_compiler_in_driver.cpp

razvanapetroaie · 2024-05-28T17:49:26Z

src/plugins/intel_npu/src/al/include/intel_npu/al/icompiler.hpp

-struct IONodeDescriptor {
-    std::string legacyName;
-    std::string currentNodeName;
+struct IODescriptor {


Most of the changes snowballed from here.

razvanapetroaie · 2024-05-28T17:56:11Z

src/plugins/intel_npu/src/backend/src/zero_infer_request.cpp

+                    ioDescriptor.nameFromCompiler,
+                    " vs. ",
+                    zeDescriptorName,
+                    ". The I/O order may have been altered, which could lead to an erroneous behavior.");


Safeguard - checks if the driver has altered the I/O order.

razvanapetroaie · 2024-05-28T18:01:08Z

src/plugins/intel_npu/src/backend/src/zero_infer_request.cpp


-    return std::nullopt;
+    return candidateBatchSize;
 }


I admit refactoring this one wasn't necessary. I ended up writting this in another context, then I've decided to move it here and replace the older function. I'm leaning towards this implementation for several minor reasons:

Previously, the logger verbosity level was not handled correctly.

The current implementation is not relying on Level Zero specific objects.

Fail fast approach - slightly more efficient I believe.

razvanapetroaie · 2024-05-28T18:04:02Z

src/plugins/intel_npu/src/backend/src/zero_infer_request.cpp

+        if (inputDescriptor.isShapeTensor) {
+            OPENVINO_ASSERT(inputDescriptor.relatedDescriptorIndex.has_value(),
+                            "The link between the dynamic tensor and its shape tensor is missing, entry name: ",
+                            inputDescriptor.nameFromCompiler);


See bindRelatedDescriptors() inside icompiler.hpp for more details on how these links are established.

razvanapetroaie · 2024-05-28T18:08:24Z

src/plugins/intel_npu/src/compiler/include/zero_compiler_in_driver.hpp

-                                    IONodeDescriptorMap& results,
-                                    IONodeDescriptorMap& states,
-                                    std::vector<std::string>& stateNames,
-                                    const T& arg) const;


We're not sending layouts anymore (other than the default one), so it's safe to assume extracting the layout is now a redundant operation.

razvanapetroaie · 2024-05-28T18:09:56Z

src/plugins/intel_npu/src/compiler/src/graph_transformations.cpp

+    // Flag used for indicating an NPU plugin version which switched the I/O identification convention from names to
+    // indices. The flag is required in order to inform the driver-compiler adapter to expect indices when attempting to
+    // deserialize the I/O metadata.
+    const auto useIndicesForIOMetadata = "use_indices_for_io_metadata";


No plugin versioning unfortunately, we have to rely on flags for signaling custom behavior inside the driver-compiler adapter.

razvanapetroaie · 2024-05-28T18:13:03Z

src/plugins/intel_npu/src/plugin/src/plugin.cpp

-        }();
+        std::shared_ptr<ov::op::v0::Parameter> parameter = std::make_shared<ov::op::v0::Parameter>(
+            inputDescriptor.precision,
+            inputDescriptor.shapeFromIRModel.has_value() ? *inputDescriptor.shapeFromIRModel


shapeFromIRModel may have no value if using an old driver in the CiD scenario. See NotSupportArgumentMetadata inside zero_compiler_in_driver.hpp.

razvanapetroaie · 2024-05-28T18:14:08Z

src/plugins/intel_npu/src/plugin/src/plugin.cpp

+    for (const IODescriptor& outputDescriptor : outputDescriptors) {
+        if (outputDescriptor.isStateInput || outputDescriptor.isStateOutput || outputDescriptor.isShapeTensor) {
+            continue;
+        }


States/shape tensors are added by the compile as I/O, these are not part of the original IR model.

razvanapetroaie · 2024-05-28T18:14:54Z

src/tests/functional/plugin/shared/src/execution_graph_tests/duplicate_inputs_outputs_names.cpp

@@ -0,0 +1,118 @@
+// Copyright (C) 2018-2024 Intel Corporation


Please lemme know if there's a better place for this test.

razvanapetroaie · 2024-05-29T12:45:07Z

...s/functional/shared_tests_instances/execution_graph_tests/duplicate_inputs_outputs_names.cpp

+
+namespace {
+
+INSTANTIATE_TEST_SUITE_P(smoke_BehaviorTests,


Although not under the behavior/ directory, I've chosen the smoke_BehaviorTests in order to allow the test to fall under one of the validation jobs regexes.

razvanapetroaie · 2024-05-29T13:21:29Z

src/plugins/intel_npu/src/al/include/sync_infer_request.hpp

-    mutable std::unordered_map<std::string, std::shared_ptr<ov::ITensor>> _shapesTensors;
-    // A copy of each tensor is needed to maintain the original L0 memory allocation in case the user provides another
-    // memory area for the tensor.
-    std::unordered_map<std::string, std::shared_ptr<ov::ITensor>> _copyAllTensors;


Moved to zero_infer_request.hpp as _levelZeroInputTensors and _levelZeroOutputTensors since only the L0 backend was using this attribute.

…es-as-ids-poc

github-actions bot added the category: NPU OpenVINO NPU plugin label Apr 25, 2024

razvanapetroaie force-pushed the EISW-121295-indices-as-ids-poc branch 2 times, most recently from 329edac to c904368 Compare May 1, 2024 11:38

razvanapetroaie force-pushed the EISW-121295-indices-as-ids-poc branch 4 times, most recently from 828b305 to 9d65acc Compare May 13, 2024 16:33

pereanub reviewed May 15, 2024

View reviewed changes

src/plugins/intel_npu/src/compiler/src/zero_compiler_in_driver.cpp Show resolved Hide resolved

razvanapetroaie force-pushed the EISW-121295-indices-as-ids-poc branch 2 times, most recently from 17045cf to 5017159 Compare May 27, 2024 16:52

github-actions bot added category: IE Tests OpenVINO Test: plugins and common category: GPU OpenVINO GPU plugin category: CPU OpenVINO CPU plugin category: TEMPLATE OpenVINO Template plugin labels May 27, 2024

razvanapetroaie force-pushed the EISW-121295-indices-as-ids-poc branch from 5017159 to 4432a71 Compare May 27, 2024 17:43

razvanapetroaie changed the title ~~[Draft][NPU] Attempting to switch the I/O identification to indices~~ [NPU] Switching the I/O identification convention to indices May 27, 2024

razvanapetroaie force-pushed the EISW-121295-indices-as-ids-poc branch from f53d917 to f384e88 Compare May 28, 2024 17:08

razvanapetroaie commented May 28, 2024

View reviewed changes

razvanapetroaie force-pushed the EISW-121295-indices-as-ids-poc branch 2 times, most recently from 21a3990 to c8a4132 Compare May 29, 2024 11:40

razvanapetroaie commented May 29, 2024

View reviewed changes

razvanapetroaie requested a review from a team as a code owner July 29, 2024 16:29

github-actions bot added the category: dependency_changes Pull requests that update a dependency file label Jul 29, 2024

razvanapetroaie force-pushed the EISW-121295-indices-as-ids-poc branch from 73f1fd0 to d78083d Compare July 29, 2024 16:32

github-actions bot removed the category: dependency_changes Pull requests that update a dependency file label Jul 29, 2024

razvanapetroaie force-pushed the EISW-121295-indices-as-ids-poc branch from d78083d to 76dbce1 Compare July 29, 2024 16:52

Merge remote-tracking branch 'upstream/master' into EISW-121295-indic…

3b8d69b

…es-as-ids-poc

razvanapetroaie force-pushed the EISW-121295-indices-as-ids-poc branch from 76dbce1 to 3b8d69b Compare July 29, 2024 16:59

razvanapetroaie added 2 commits July 30, 2024 16:10

Merge branch 'master' into EISW-121295-indices-as-ids-poc

92e79eb

Replacing NetworkMetadata::findByName with lambda functions

d9334df

razvanapetroaie force-pushed the EISW-121295-indices-as-ids-poc branch from ecf8093 to d9334df Compare July 31, 2024 13:57

razvanapetroaie added 7 commits July 31, 2024 16:57

Merge branch 'master' into EISW-121295-indices-as-ids-poc

a2998f4

Updating the compiler version once more

3d0a194

Merge branch 'master' into EISW-121295-indices-as-ids-poc

c110096

Merge branch 'master' into EISW-121295-indices-as-ids-poc

4b0246d

Merge branch 'master' into EISW-121295-indices-as-ids-poc

c2f47b5

Merge branch 'master' into EISW-121295-indices-as-ids-poc

8543438

Merge remote-tracking branch 'upstream/master' into EISW-121295-indic…

860552c

…es-as-ids-poc

razvanapetroaie force-pushed the EISW-121295-indices-as-ids-poc branch from a24ee00 to 860552c Compare August 7, 2024 14:20

Merge branch 'master' into EISW-121295-indices-as-ids-poc

7c106e3

razvanapetroaie enabled auto-merge August 11, 2024 07:22

razvanapetroaie disabled auto-merge August 11, 2024 07:42

Merge branch 'master' into EISW-121295-indices-as-ids-poc

4853fd7

razvanapetroaie enabled auto-merge August 11, 2024 07:44

razvanapetroaie disabled auto-merge August 11, 2024 10:21

razvanapetroaie enabled auto-merge August 11, 2024 18:19

Merge branch 'master' into EISW-121295-indices-as-ids-poc

d3e1657

razvanapetroaie added this pull request to the merge queue Aug 12, 2024

Merged via the queue into openvinotoolkit:master with commit e567c9e Aug 12, 2024
138 checks passed

razvanapetroaie deleted the EISW-121295-indices-as-ids-poc branch August 12, 2024 12:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[NPU] Switching the I/O identification convention to indices #24248

[NPU] Switching the I/O identification convention to indices #24248

razvanapetroaie commented Apr 25, 2024 •

edited

Loading

razvanapetroaie May 28, 2024 •

edited

Loading

razvanapetroaie May 28, 2024

razvanapetroaie May 28, 2024

razvanapetroaie May 28, 2024 •

edited

Loading

razvanapetroaie May 28, 2024

razvanapetroaie May 28, 2024

razvanapetroaie May 28, 2024

razvanapetroaie May 28, 2024

razvanapetroaie May 28, 2024

razvanapetroaie May 29, 2024

razvanapetroaie May 29, 2024

		@@ -0,0 +1,118 @@
		// Copyright (C) 2018-2024 Intel Corporation


		namespace {

		INSTANTIATE_TEST_SUITE_P(smoke_BehaviorTests,

[NPU] Switching the I/O identification convention to indices #24248

[NPU] Switching the I/O identification convention to indices #24248

Conversation

razvanapetroaie commented Apr 25, 2024 • edited Loading

Details:

Tickets:

razvanapetroaie May 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

razvanapetroaie May 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

razvanapetroaie commented Apr 25, 2024 •

edited

Loading

razvanapetroaie May 28, 2024 •

edited

Loading

razvanapetroaie May 28, 2024 •

edited

Loading