forked from openvinotoolkit/openvino
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Doc Migration from Gitlab (openvinotoolkit#1289)
* doc migration * fix * Update FakeQuantize_1.md * Update performance_benchmarks.md * Updates graphs for FPGA * Update performance_benchmarks.md * Change DL Workbench structure (openvinotoolkit#1) * Changed DL Workbench structure * Fixed tags * fixes * Update ie_docs.xml * Update performance_benchmarks_faq.md * Fixes in DL Workbench layout * Fixes for CVS-31290 * [DL Workbench] Minor correction * Fix for CVS-30955 * Added nGraph deprecation notice as requested by Zoe * fix broken links in api doxy layouts * CVS-31131 fixes * Additional fixes * Fixed POT TOC * Update PAC_Configure.md PAC DCP 1.2.1 install guide. * Update inference_engine_intro.md * fix broken link * Update opset.md
- Loading branch information
Showing
531 changed files
with
44,744 additions
and
163 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,83 @@ | ||
# Regression tests howto {#openvino_docs_HOWTO_add_regression_test_vpu} | ||
|
||
## Purpose | ||
|
||
This document contains instructions for correctly modifying a set of regression tests. | ||
|
||
## Common | ||
|
||
Regression tests for Myriad and HDDL plugins are on the path: | ||
`inference-engine/tests/functional/vpu/regression_tests/` | ||
|
||
The tests are divided into 4 groups: | ||
* Classification | ||
* Detection | ||
* Raw-results | ||
* Compilation | ||
* VPU hetero | ||
|
||
Testing framework – [Google Test](https://github.com/google/googletest/). | ||
Each group contains [parameterized](https://github.com/google/googletest/blob/master/googletest/docs/advanced.md) tests. The main idea is that to add a new test, you only need to add a new parameter. Except for scenarios different from the generalized case. | ||
|
||
## Classsification and Detection tests | ||
|
||
These groups contains two cases: | ||
|
||
* For generalized scenario (` VpuNoClassificationRegression, VpuNoDetectionRegression`) | ||
* For specific scenario (` VpuNoClassificationRegressionSpecific, VpuNoDetectionRegressionSpecific`) | ||
|
||
### Generalized scenario | ||
|
||
If You want test new parameter(batch, precision, model and etc.) then You need to edit the existing initialization of parameterized tests or create a new one. | ||
Example of initialization of parameterized tests: | ||
|
||
``` c++ | ||
INSTANTIATE_TEST_CASE_P( | ||
VPURegTestWithResources_nightly, | ||
VpuNoClassificationRegression, | ||
Combine(ValuesIn(VpuTestParamsContainer::testingPlugin()), | ||
Values(Precision::FP16), | ||
Values(1), // batches | ||
Values(true), //IsHwAdaptiveMode | ||
Values(false), //DoReshape | ||
Values(3, 5, 7), //Resources | ||
Values(false), //IsIgnoreStatistic | ||
Values(ClassificationSrcParam{ModelName::GoogleNetV1, SourceImages::kCat3, 0.01, Regression::EMean::eValues})), | ||
VpuNoClassificationRegression::getTestCaseName); | ||
``` | ||
### Specific scenario | ||
If You need a test to perform some actions that are not provided in the generalized scenario, then add a specific test case. As with the generalized scenario You can change parameters for these tests. | ||
Example of specific test case: | ||
``` c++ | ||
TEST_P(VpuNoClassificationRegressionSpecific, onAlexNetWithNetworkConfig) { | ||
DISABLE_ON_WINDOWS_IF(HDDL_PLUGIN); | ||
DISABLE_IF(do_reshape_); | ||
if (!hw_adaptive_mode_) { | ||
config_[VPU_CONFIG_KEY(NETWORK_CONFIG)] = "data=data,scale=1"; | ||
} | ||
assertThat().classificationResultsForInferRequestAPI() | ||
.on(SourceImages::kDog2) | ||
.withInputPrecision(in_precision_) | ||
.times(batch_) | ||
.withBatch(batch_) | ||
.onModel(ModelName::AlexNet) | ||
.setMean(Regression::EMean::eImage) | ||
.onFP16() | ||
.withTopK(1) | ||
.withPluginConfig(config_) | ||
.equalToReferenceWithDelta(0.04); | ||
} | ||
``` | ||
|
||
## Raw-results tests | ||
|
||
There is no generalized scenario and recommendations are the same as for specific test cases for Classification/Detection groups. | ||
|
||
## Compilation tests | ||
|
||
The tests are in the `vpu_classification_regression.cpp` file and contains only one scenario ` VpuNoRegressionWithCompilation `. To add a new test just update parameters just as in generalized scenarion of Classification/Detection test groups. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,94 @@ | ||
# Fuzzing howto {#openvino_docs_HOWTO_fuzzing_HOWTO} | ||
|
||
## Intended Audience | ||
|
||
This document is for a developer who wants to contribute fuzz tests. | ||
|
||
## Purpose | ||
|
||
This document walks you through creating your first fuzzer, running it and evaluating its quality. | ||
|
||
## Prerequisites | ||
|
||
- Linux OS or Mac OS. | ||
|
||
- [American Fuzzy Loop](http://lcamtuf.coredump.cx/afl/) if building with GCC. | ||
|
||
## Steps | ||
|
||
1. Create a fuzz test in the existing project at `./tests/fuzz`. Fuzz test must | ||
follow `<test name>-fuzzer.cc` naming scheme and implement a | ||
`LLVMFuzzerTestOneInput` entry point. | ||
|
||
``` bash | ||
cat << EOF > ./tests/fuzz/test_name-fuzzer.cc | ||
#include <stdint.h> | ||
#include <cstdlib> | ||
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) { | ||
// put your fuzzing code here and use data+size as input. | ||
return 0; // always return 0 | ||
} | ||
EOF | ||
``` | ||
|
||
2. Implement test logic under `LLVMFuzzerTestOneInput`. | ||
|
||
See example fuzz test at `tests/fuzz/read_network-fuzzer.cc`. | ||
|
||
3. Build fuzz tests with `-DENABLE_FUZZING=ON` flag for cmake. | ||
|
||
``` bash | ||
mkdir -p build && \ | ||
(cd build && \ | ||
CXX=afl-g++ CC=afl-gcc cmake -DCMAKE_BUILD_TYPE=Debug -DENABLE_FUZZING=ON -DENABLE_TESTS=ON .. && \ | ||
make fuzz --jobs=$(getconf _NPROCESSORS_ONLN)) | ||
``` | ||
|
||
4. Prepare sample inputs for your fuzz test to teach fuzzer engine on input | ||
structure | ||
|
||
``` bash | ||
(cd bin/intel64/Debug && \ | ||
mkdir test_name-corpus && \ | ||
echo sample input > test_name-corpus/in1.txt) | ||
``` | ||
|
||
5. Evaluate fuzz test with `afl-fuzz` fuzzing engine | ||
|
||
Run fuzz test: | ||
|
||
``` bash | ||
(cd bin/intel64/Debug && \ | ||
afl-fuzz -i test_name-corpus -o test_name-out -- ./test_name-fuzzer @@ | ||
``` | ||
While fuzz test is running it prints out statistics. Besides just crashes `uniq | ||
crashes` and hangs `uniq hangs` you should care about fuzz test quality: | ||
- Fuzz test should be fast - speed of execution `exec speed` should be at least | ||
100 exec/s. Speed less than 20 exec/s is not acceptable. | ||
- Fuzz test should be able to explore new code paths `map coverage` and | ||
`findings in depth`. Confirm it is increasing while fuzz test is running. | ||
6. Reproduce fuzz test findings | ||
All issues found by fuzz test are stored as a file in output folder specified | ||
earlier via `-o` afl-fuzz option. To reproduce an issue run fuzz test executable | ||
with an issue file as an argument. | ||
## Summary | ||
We have created a simple fuzz test, run it and asses its results. | ||
## Extension | ||
Try run parallel fuzzing with the help of | ||
[afl-utils](https://gitlab.com/rc0r/afl-utils). | ||
## Tips or FAQs | ||
GCC 7 in Ubuntu 18.04 LTS has a | ||
[defect](https://bugs.launchpad.net/ubuntu/+source/afl/+bug/1774816). Upgrade | ||
GCC 7 for AFL to work. GCC version `Ubuntu 7.3.0-27ubuntu1~18.04` works OK. |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.