Luocheng/vpux/prealloc mem kmb #1

luo-cheng2021 · 2021-02-26T06:41:47Z

Details:

Add prealloc image memory in remote device feature
When passing -use_remote_mem in command line the inferrequest will allocate remote memory using HddlUnite. Then all infer will use the remote memory and there is no need to copy the image memory from IA to remote device. The creation steps:
-- load image from the specified folder
-- allocate remote memory and copy the image to it
-- set remote memory handle
-- do the infer request in the benchmark loop

riverlijunjie · 2021-02-26T07:39:59Z

inference-engine/samples/benchmark_app/CMakeLists.txt

+        add_definitions(-DUSE_PREALLOC_MEM)
+        set(HDDL2_DEP "HddlUnite::HddlUnite")
+    else()
+        message(WARNING "hddl2_params.hpp could not find. Preallocate in KMB feature is disabled.")


Please remove specified string, such as "KMB".

riverlijunjie · 2021-02-26T07:41:12Z

inference-engine/samples/benchmark_app/README.md

@@ -85,6 +85,7 @@ Options:
    -t                        Optional. Time, in seconds, to execute topology.
    -progress                 Optional. Show progress bar (can affect performance measurement). Default values is "false".
    -shape                    Optional. Set shape for input. For example, "input1[1,3,224,224],input2[1,4]" or "[1,3,224,224]" in case of one input size.
+    -use_prealloc_mem           Optional. Prealloc remote memory in xBay to execute infer request.


Is it will be better if use "-use_remote_mem"?

riverlijunjie · 2021-02-26T07:42:19Z

inference-engine/samples/benchmark_app/benchmark_app.hpp

@@ -97,6 +97,11 @@ static const char load_config_message[] = "Optional. Path to XML/YAML/JSON file
 static const char dump_config_message[] = "Optional. Path to XML/YAML/JSON file to dump IE parameters, which were set by application.";
 #endif

+#ifdef USE_PREALLOC_MEM
+// @brief message for preallocing memory option
+static const char use_prealloc_mem_message[] = "Optional. Prealloc remote memory in xBay to execute infer request.";


static const char use_prealloc_mem_message[] = "Optional. Prealloc remote memory in device to execute infer request."

riverlijunjie · 2021-02-26T07:48:24Z

inference-engine/samples/benchmark_app/inputs_filling.cpp

+    size_t width;
+    size_t height;
+    remoteIE.GetWxH(width, height);
+    const size_t nv12Size = width * height * 3 / 2 * batchSize;


We also need support pure NN without preprocess, so the input can be RGB_Plannar that can feed into NN directly.
NV12 buffer will go into PP first and then to NN.

Let's first support pure inference without PP.

Pure inference without pp done.

riverlijunjie · 2021-02-26T07:49:44Z

inference-engine/samples/benchmark_app/inputs_filling.cpp

@@ -324,3 +326,89 @@ void fillBlobs(const std::vector<std::string>& inputFiles,
        }
    }
 }
+
+#ifdef USE_PREALLOC_MEM
+void fillRemoteBlobs(RemoteHelper& remoteIE,


FillRemoteBlobsNV12

fillRemoteBlobs removed. The function merged into fillBlobs.

riverlijunjie · 2021-02-26T08:22:26Z

inference-engine/samples/benchmark_app/inputs_filling.cpp

+            auto minputHolder = minput->rmap();
+            auto inputBlobData = minputHolder.as<uint8_t*>();
+
+            BGR2NV12(inputBlobData, width, height, batchSize, data.get());


It will do CSC + Resize or only CSC?

Preprocess removed.

riverlijunjie · 2021-02-26T08:24:25Z

inference-engine/samples/benchmark_app/remote_helper.cpp

+using namespace InferenceEngine;
+
+#define REMOTE_IMAGE_WIDTH 1920
+#define REMOTE_IMAGE_HEIGHT 1080


Maybe we need assign the input resolution if need do PP, the parameter can be put into benchmark input config file?

Preprocessing removed.

riverlijunjie · 2021-02-26T08:27:00Z

inference-engine/samples/benchmark_app/remote_helper.cpp

+            THROW_IE_EXCEPTION << "Could not open file: " << graphPath;
+        }
+        std::istream graphBlob(&blobFile);
+        return ie.ImportNetwork(graphBlob, _contextPtr);


In the future we also need support online compiling IR. - call LoadNetwork()

LoadNetwork supported.

riverlijunjie · 2021-02-26T12:31:00Z

inference-engine/samples/benchmark_app/utils.cpp

@@ -205,4 +206,27 @@ void load_config(const std::string& filename,
        }
    }
 }
+
+void BGR2NV12(uint8_t* src, size_t width, size_t height, size_t imageNum, uint8_t* dst) {


Why convert RGB to NV12?

CSC removed.(no needed anymore)

luo-cheng2021 · 2021-03-01T09:09:02Z

@riverlijunjie Preprocessing removed, please help to review, thanks.

riverlijunjie · 2021-03-01T13:48:10Z

inference-engine/samples/benchmark_app/main.cpp

        ExecutableNetwork exeNetwork;

+#ifdef USE_REMOTE_MEM
+        RemoteHelper remoteHelper;


Can we get the device_name="VPUX"?

If yes, we can put the the init code block into "if(device == "VPUX")" like "CPU" or "GPU" below.

Done. Move the initialization to the branch of the 'VPUX'.

riverlijunjie · 2021-03-01T13:53:25Z

inference-engine/samples/benchmark_app/remote_helper.cpp

+
+using namespace InferenceEngine;
+
+class RemoteHelper::Impl {


Is it better if name it RemoteContextHelper?

mashoujiang · 2021-03-03T03:26:36Z

As I understand the benchmark_app is image workload case, right?
If yes, why use workload context?
if use the workload context, the process will only scheduled to single device, considering one test case like: In SRB, we run the benchmark test, it will only be scheduled to one device.
Correct me if I misunderstand.

riverlijunjie · 2021-03-03T04:52:10Z

Video workload is used for E2E pipeline, but benchmark didn't provide such test case for it, especially for KPI. So we need some official KPI data for video workload.

mangguo321 · 2021-03-08T06:03:26Z

Do we need to add CPU_THROUGHPUT_STREAMS and CPU_THREADS_NUM configuration in video case?

luo-cheng2021 · 2021-03-12T01:28:19Z

We can use the command line 'benchmark_app -load_config config.yml ...' to get support 'VPUX_THROUGHPUT_STREAMS, VPUX_INFERENCE_SHAVES and etc' and the config.yml is just like:

%YAML:1.0

VPUX: { VPUX_THROUGHPUT_STREAMS:"3", VPUX_INFERENCE_SHAVES:"16"}
`

…_definitions

Add support for multiple outputs

Bym/pdpd frontend/op add relu & softmax

* Moved cmake/templates to <root> * Removed ngraph versioning, reused IE one * Merged converage * Removed duplicatde ngraph cmake options * Moved dependencies to <root>/cmake * Removed installing of VERSION * Start #1 * cpack * Added component type * Added installation of tests targets * Added ngraph tests target install * Fixed runtime dependencies location * Disable GNA unit tests * Revert "Disable GNA unit tests" This reverts commit da53986. * Installed only core component * Replaced ENABLE_DEV_PKG_INSTALL with EXCLUDE_FROM_ALL * Removed extra cmake options

* [FrontEnd]enable pdpd ops conversion part3 * Add adaptive pool2d op conversion (#1) * param support tensor (#2) * add missing sync_batch_norm * Update pow.cpp * deal empty axis (#5) * deal empty axis * apply review comments * fix code style * fix code style * change shape to i32 * fix code in shape * fix code style * fix paddle code style * remove redandent ops * fix maxAdativePool * fix expand_v2 * remove redandent code Co-authored-by: Mang Guo <[email protected]> Co-authored-by: Luo Cheng <[email protected]>

* remove reader tests #1 * remove reader tests #2 * remove reader tests #3 * remove reader tests #4 * Add clone_with_new_inputs to visitor tests * fixes

…f POT (openvinotoolkit#17398) * Update model_optimization_guide.md * Update model_optimization_guide.md * Update model_optimization_guide.md * Update model_optimization_guide.md * Update model_optimization_guide.md * Update model_optimization_guide.md * Update model_optimization_guide.md * Update home.rst * Update ptq_introduction.md * Update Introduction.md * Update Introduction.md * Update Introduction.md * Update ptq_introduction.md * Update ptq_introduction.md * Update basic_quantization_flow.md * Update basic_quantization_flow.md * Update basic_quantization_flow.md * Update quantization_w_accuracy_control.md * Update quantization_w_accuracy_control.md * Update quantization_w_accuracy_control.md * Update quantization_w_accuracy_control.md * Update quantization_w_accuracy_control.md * Update quantization_w_accuracy_control.md * Update quantization_w_accuracy_control.md * Update quantization_w_accuracy_control.md * Update quantization_w_accuracy_control.md * Update basic_quantization_flow.md * Update basic_quantization_flow.md * Update quantization_w_accuracy_control.md * Update basic_quantization_flow.md * Update basic_quantization_flow.md * Update model_optimization_guide.md * Update ptq_introduction.md * Update quantization_w_accuracy_control.md * Update model_optimization_guide.md * Update quantization_w_accuracy_control.md * Update model_optimization_guide.md * Update quantization_w_accuracy_control.md * Update model_optimization_guide.md * Update Introduction.md * Update basic_quantization_flow.md * Update basic_quantization_flow.md * Update quantization_w_accuracy_control.md * Update ptq_introduction.md * Update Introduction.md * Update model_optimization_guide.md * Update basic_quantization_flow.md * Update quantization_w_accuracy_control.md * Update quantization_w_accuracy_control.md * Update quantization_w_accuracy_control.md * Update Introduction.md * Update FrequentlyAskedQuestions.md * Update model_optimization_guide.md * Update Introduction.md * Update model_optimization_guide.md * Update model_optimization_guide.md * Update model_optimization_guide.md * Update model_optimization_guide.md * Update model_optimization_guide.md * Update ptq_introduction.md * Update ptq_introduction.md * added code snippet (#1) * Update basic_quantization_flow.md * Update basic_quantization_flow.md * Update quantization_w_accuracy_control.md * Update basic_quantization_flow.md * Update basic_quantization_flow.md * Update ptq_introduction.md * Update model_optimization_guide.md * Update basic_quantization_flow.md * Update ptq_introduction.md * Update quantization_w_accuracy_control.md * Update basic_quantization_flow.md * Update basic_quantization_flow.md * Update basic_quantization_flow.md * Update ptq_introduction.md * Update ptq_introduction.md * Delete ptq_introduction.md * Update FrequentlyAskedQuestions.md * Update Introduction.md * Update quantization_w_accuracy_control.md * Update introduction.md * Update basic_quantization_flow.md code blocks * Update quantization_w_accuracy_control.md code snippets * Update docs/optimization_guide/nncf/ptq/code/ptq_torch.py Co-authored-by: Alexander Suslov <[email protected]> * Update model_optimization_guide.md * Optimization docs proofreading (#2) * images updated * delete reminder * review * text review * change images to original ones * Update filter_pruning.md code blocks * Update basic_quantization_flow.md * Update quantization_w_accuracy_control.md * Update images (#3) * images updated * delete reminder * review * text review * change images to original ones * Update filter_pruning.md code blocks * update images * resolve conflicts * resolve conflicts * change images to original ones * resolve conflicts * update images * fix conflicts * Update model_optimization_guide.md * Update docs/optimization_guide/nncf/ptq/code/ptq_tensorflow.py Co-authored-by: Alexander Suslov <[email protected]> * Update docs/optimization_guide/nncf/ptq/code/ptq_torch.py Co-authored-by: Alexander Suslov <[email protected]> * Update docs/optimization_guide/nncf/ptq/code/ptq_onnx.py Co-authored-by: Alexander Suslov <[email protected]> * Update docs/optimization_guide/nncf/ptq/code/ptq_aa_openvino.py Co-authored-by: Alexander Suslov <[email protected]> * Update docs/optimization_guide/nncf/ptq/code/ptq_openvino.py Co-authored-by: Alexander Suslov <[email protected]> * table format fix * Update headers * Update qat.md code blocks --------- Co-authored-by: Alexander Suslov <[email protected]> Co-authored-by: Tatiana Savina <[email protected]>

* Remove `set_preprocess.cpp` * Remove `preprocessing.hpp` * Remove `locale.hpp` - ported to `CanCompileModelWithCustomLocale` * Port `version.cpp` and remove legacy * Revert shared `version.hpp`

* Delete `ngraph/visibility.hpp` * Delete `ngraph/log.hpp` * Delete `ngraph/file_util.hpp` * Delete `ngraph/type.hpp` * Delete `ngraph/dimension.hpp` * Delete `ngraph/coordinate.hpp` * ClangFormat * Fix build * Fix pyngraph * Remove comment * Fix build

github-actions · 2024-09-06T01:17:15Z

This PR will be closed in a week because of 2 weeks of no activity.

riverlijunjie reviewed Feb 26, 2021

View reviewed changes

riverlijunjie reviewed Mar 1, 2021

View reviewed changes

luo-cheng2021 added 2 commits April 15, 2021 14:47

pre-alloc remote memory; rebase from vpux/2021/3

4cdb4bb

remove unused includes, use target_compile_definitions instead of add…

d7d1ef9

…_definitions

luo-cheng2021 force-pushed the luocheng/vpux/prealloc_mem_kmb branch from 9528f90 to d7d1ef9 Compare April 15, 2021 07:04

luo-cheng2021 changed the base branch from releases/vpux/2021/2 to releases/vpux/2021/3 April 15, 2021 07:08

remove RemoteContextHelper::Impl class

1d7579d

luo-cheng2021 pushed a commit that referenced this pull request Apr 19, 2021

Merge pull request #1 from itikhono/itikhono/pdpd/support_mul_outputs

9174c87

Add support for multiple outputs

modify help message, move Core def location

3eed091

ceciliapeng2011 added a commit that referenced this pull request Apr 22, 2021

Merge pull request #1 from BaiYM0117/bym/pdpd_frontend/op

40692a8

Bym/pdpd frontend/op add relu & softmax

luo-cheng2021 pushed a commit that referenced this pull request Jul 19, 2021

Add adaptive pool2d op conversion (#1)

a15b5dd

luo-cheng2021 pushed a commit that referenced this pull request Oct 28, 2022

Eliminate reader tests (openvinotoolkit#13409)

ad1c824

* remove reader tests #1 * remove reader tests #2 * remove reader tests #3 * remove reader tests #4 * Add clone_with_new_inputs to visitor tests * fixes

luo-cheng2021 pushed a commit that referenced this pull request Mar 14, 2023

Apply review comments #1

b87ba5d

luo-cheng2021 pushed a commit that referenced this pull request Mar 16, 2023

Apply review comments #1

845e9f0

github-actions bot added the Stale label Sep 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Luocheng/vpux/prealloc mem kmb #1

Luocheng/vpux/prealloc mem kmb #1

luo-cheng2021 commented Feb 26, 2021 •

edited

Loading

riverlijunjie Feb 26, 2021

luo-cheng2021 Mar 1, 2021

riverlijunjie Feb 26, 2021

luo-cheng2021 Mar 1, 2021

riverlijunjie Feb 26, 2021

luo-cheng2021 Mar 1, 2021

riverlijunjie Feb 26, 2021

riverlijunjie Feb 26, 2021

luo-cheng2021 Mar 1, 2021

riverlijunjie Feb 26, 2021

luo-cheng2021 Mar 1, 2021

riverlijunjie Feb 26, 2021

luo-cheng2021 Mar 1, 2021

riverlijunjie Feb 26, 2021

luo-cheng2021 Mar 1, 2021

riverlijunjie Feb 26, 2021

luo-cheng2021 Mar 1, 2021

riverlijunjie Feb 26, 2021

luo-cheng2021 Mar 1, 2021

luo-cheng2021 commented Mar 1, 2021

riverlijunjie Mar 1, 2021

luo-cheng2021 Mar 2, 2021

riverlijunjie Mar 1, 2021

luo-cheng2021 Mar 2, 2021

mashoujiang commented Mar 3, 2021

riverlijunjie commented Mar 3, 2021

mangguo321 commented Mar 8, 2021

luo-cheng2021 commented Mar 12, 2021 •

edited

Loading

github-actions bot commented Sep 6, 2024

Luocheng/vpux/prealloc mem kmb #1

Are you sure you want to change the base?

Luocheng/vpux/prealloc mem kmb #1

Conversation

luo-cheng2021 commented Feb 26, 2021 • edited Loading

Details:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luo-cheng2021 commented Mar 1, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mashoujiang commented Mar 3, 2021

riverlijunjie commented Mar 3, 2021

mangguo321 commented Mar 8, 2021

luo-cheng2021 commented Mar 12, 2021 • edited Loading

%YAML:1.0

github-actions bot commented Sep 6, 2024

luo-cheng2021 commented Feb 26, 2021 •

edited

Loading

luo-cheng2021 commented Mar 12, 2021 •

edited

Loading