Ignored scope matching working with multiple NNCFGraphs

[kshpv]

Changes

Add support of IgnoredScope matching many NNCFGraphs.
Change unmatched error logging when strict=True:
  before: the first unmatched rule is logged
  after: all unmatched rules are logged.
IgnoredScope validation for OpenVINO models with IF operation was updated with new changes - the validation is put before running the quantization pipeline.

Reason for changes

IgnoredScope validation failure for a OV model with IF operation.

Related tickets

135110

Tests

TBD

[codecov]

Codecov Report

Attention: Patch coverage is 95.23810% with 4 lines in your changes missing coverage. Please review.

Project coverage is 90.67%. Comparing base (85b3263) to head (1c88dee).
Report is 4 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2723      +/-   ##
===========================================
- Coverage    91.18%   90.67%   -0.51%     
===========================================
  Files          483      483              
  Lines        46363    46443      +80     
===========================================
- Hits         42274    42111     -163     
- Misses        4089     4332     +243     

Files	Coverage Δ
nncf/common/graph/graph.py	96.56% <100.00%> (+0.01%)	⬆️
nncf/openvino/quantization/quantize_ifmodel.py	100.00% <100.00%> (ø)
nncf/quantization/quantize_model.py	80.82% <75.00%> (+0.26%)	⬆️
nncf/scopes.py	95.65% <98.07%> (+4.08%)	⬆️
nncf/openvino/quantization/quantize_model.py	65.03% <90.47%> (+3.43%)	⬆️

... and 23 files with indirect coverage changes

Flag	Coverage Δ
COMMON	41.93% <62.50%> (-0.10%)	⬇️
ONNX	34.18% <55.95%> (-0.01%)	⬇️
OPENVINO	40.96% <95.23%> (+0.11%)	⬆️
TENSORFLOW	29.39% <13.09%> (-0.04%)	⬇️
TORCH	64.84% <53.57%> (-0.59%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
common	93.55% <100.00%> (+<0.01%)	⬆️
torch	92.00% <ø> (-1.65%)	⬇️
tensorflow	93.26% <ø> (ø)
onnx	93.06% <ø> (ø)
openvino	94.48% <91.66%> (+<0.01%)	⬆️
ptq	90.50% <75.00%> (+0.06%)	⬆️

[alexsu52]

Please provide the output for ignored scope before and after this PR, because this PR changes logging for the ignored scope.

[kshpv]

For the https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/openvino/yolov8/main.py with the following ignored_scope:

ignored_scope=nncf.IgnoredScope(
            patterns=[".bad_pattern"],
            names=["bad_name", "/model.22/Concat"],
            types=["Subtract", "bad_type"],
            subgraphs=[
                nncf.Subgraph(
                    inputs=["/model.22/Concat", "/model.22/Concat_1", "/model.22/Concat_2"],
                    outputs=["output0/sink_port_0"],
                ),
                nncf.Subgraph(
                    inputs=["/model.22/Concat"],
                    outputs=["/model.22/Concat"],
                ),
            ],
        ),

The resulted logs are obtained:
Old log:

New log:

[alexsu52]

For the https://github.com/openvinotoolkit/nncf/blob/develop/examples/post_training_quantization/openvino/yolov8/main.py with the following ignored_scope:

ignored_scope=nncf.IgnoredScope(
            patterns=[".bad_pattern"],
            names=["bad_name", "/model.22/Concat"],
            types=["Subtract", "bad_type"],
            subgraphs=[
                nncf.Subgraph(
                    inputs=["/model.22/Concat", "/model.22/Concat_1", "/model.22/Concat_2"],
                    outputs=["output0/sink_port_0"],
                ),
                nncf.Subgraph(
                    inputs=["/model.22/Concat"],
                    outputs=["/model.22/Concat"],
                ),
            ],
        ),

The resulted logs are obtained: Old log: New log:

I have found original_graph.dot in the message. It looks like is not valid. I would suggest to remove it for PTQ.

[kshpv]

I have found original_graph.dot in the message. It looks like is not valid. I would suggest to remove it for PTQ.

Updated:

[alexsu52]

LGTM

[KodiaqQ]

@kshpv, can we merge it? Were all fixes included?