[Good First Issue][NNCF]: Dump actual_subset_size to ov.Model

[l-bat]

Context

After applying quantization to the ov.Model in Neural Network Compression Framework (NNCF), the quantization parameters, including subset_size, are dumped to the meta section of the OpenVINO IR. subset_size represents the size of the dataset used for calibration.

nncf/nncf/openvino/quantization/quantize_model.py

Line 102 in 09960b9

"subset_size": subset_size,

However, inconsistencies arise when the dataset size is less than the provided or default 'subset_size'. To address this confusion, it is proposed to also dump the actual_subset_size, which denotes the number of data samples used to calculate activation statistics. This addition will improve clarity and accuracy in managing quantization parameters and assist in reproducing quantization results.

What needs to be done?

• Dump actual_subset_size parameter to ov.Model meta section.
• Add tests

Example Pull Requests

No response

Resources

• Contribution guide - start here!

Contact points

@l-bat

Ticket

No response

[AiGaf1]

.take

[github-actions]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[andrey-churkin]

@AiGaf1 Hi, are you still working on this task? Do you need any help? Please inform us if you do not plan to continue working on this task. Thanks!

[RitikaxShakya]

Hello! is there any update on this issue? If not i wish to work on this issue.

[p-wysocki]

@l-bat could you please reassign the issue to @RitikaxShakya? I lack the permissions for NNCF repository.

[RitikaxShakya]

.take

[github-actions]

Thanks for being interested in this issue. It looks like this ticket is already assigned to a contributor. Please communicate with the assigned contributor to confirm the status of the issue.

[p-wysocki]

Hello @RitikaxShakya, are you still working on that issue? Do you need any help?

[awayzjj]

.take

[github-actions]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[awayzjj]

Hi @l-bat I created a PR after testing locally, and the XML output is as expected:

I ran the pytest in tests/openvino, and the original tests did not break. But I have 2 questions

1. which file I should edit to add my unit test.
2. how to implement the unit test, should I check the output XML to verify whether the actual_subset_size property exists?

Thank you very much!

[l-bat]

Hi @awayzjj!
Thanks for your contribution!

calibration_dataset.get_length() returns the size of the dataset that was provided to the nncf.quantize method, however actual_subset_size should show the number of data samples that were used to calculate the activation statistics.
In the case of calibration_dataset.get_length() >= subset_size, actual_subset_size is equal to subset_size. Otherwise, actual_subset_size must be equal to calibration_dataset.get_length(). But it is not possible to use the get_length() method if __len__() is not implemented. Please take a look at

nncf/nncf/common/tensor_statistics/aggregator.py

Lines 50 to 53 in e8ea252

	dataset_length = self.dataset.get_length()
	if dataset_length and self.stat_subset_size:
	return min(dataset_length, self.stat_subset_size)
	return dataset_length or self.stat_subset_size

. You can implement the get_actual_subset_size() function.

[l-bat]

I ran the pytest in tests/openvino, and the original tests did not break. But I have 2 questions

1. which file I should edit to add my unit test.
2. how to implement the unit test, should I check the output XML to verify whether the actual_subset_size property exists?

1. You can add test to https://github.com/openvinotoolkit/nncf/blob/e8ea2521663de807d654ae4f375d20c904755061/tests/openvino/native/quantization/test_quantization_pipeline.py
2. You can use the test as an example

   nncf/tests/openvino/native/quantization/test_quantization_pipeline.py

   Lines 178 to 199 in e8ea252

   	def test_ignored_scope_dump(ignored_options, expected_dump, tmp_path):
   	ignored_scope_path = ["nncf", "quantization", "ignored_scope"]
   	model = WeightsModel().ov_model
   	dataset = get_dataset_for_test(model)
   	quantize_parameters = {
   	"preset": QuantizationPreset.PERFORMANCE,
   	"target_device": TargetDevice.CPU,
   	"subset_size": 1,
   	"fast_bias_correction": True,
   	"ignored_scope": ignored_options,
   	}
   	quantized_model = quantize_impl(model, dataset, **quantize_parameters)
   	ov.save_model(quantized_model, tmp_path / "ov_model.xml")
   	core = ov.Core()
   	dumped_model = core.read_model(tmp_path / "ov_model.xml")
   	for key, value in expected_dump.items():
   	rt_path = ignored_scope_path + [key] if key else ignored_scope_path
   	if value:
   	assert dumped_model.get_rt_info(rt_path) == value
   	else:
   	assert dumped_model.has_rt_info(rt_path) is False

[awayzjj]

@l-bat Hi, I've been really busy lately, so I've decided to unassign myself for now. I apologize for any inconvenience this may cause.

[zina-cs]

.take

[github-actions]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.