Implementation of strict mode

[PINTO0309]

Issue Type

Others

onnx2tf version number

1.7.x

onnx version number

1.13.0

tensorflow version number

2.10.0

Download URL for ONNX

N/A

Parameter Replacement JSON

N/A

Description

1. Personal
2. Add an ultra-slow conversion mode to strictly avoid shape unmatch errors and precision errors.

Background

Since the internal processing has already implemented a number of fairly complex workarounds to avoid conversion errors, we have not dared to implement a correction process for accuracy errors at this time in order not to further increase the complexity of the logic. Instead, I have implemented a mechanism whereby the user visually identifies which operation's channel transpositions cause problems, and the user changes the behavior of the tool himself.

However, I cannot overlook the situation where accuracy errors remain despite successful model transformations, and I hope to eradicate them in the future. The work to change the behavior of the tool by the users themselves with the users' visibility of the problem areas is very costly.

Since the internal processing mechanism needs to be significantly revised, we intend to implement many additional test modifications gradually in minor version upgrades to the extent that they do not affect the existing processing.

Idea

• Add all OPs of ONNX to the output OP of the graph.
• Run ONNX inference with dummy tensor only once to get the sample inference results for all OPs.
• Always compare the contents of the ONNX output tensor and TensorFlow output tensor before converting each OP.
• Channel placement will always be inconsistent between ONNX and TensorFlow, so the following implementation can be called from all OPs at any time.
  

  onnx2tf/onnx2tf/utils/common_functions.py

  Lines 2937 to 3062 in 33e65dc

  	def onnx_tf_tensor_validation(
  	*,
  	onnx_tensor_infos: Dict[str, np.ndarray],
  	tf_tensor_infos: Dict[str, np.ndarray],
  	rtol: float=1e-05,
  	atol: float=1e-05,
  	) -> Dict[str, List]:
  	"""Check if the ONNX tensor and the TF tensor are approximate.
  	Parameters
  	----------
  	onnx_tensor_infos: Dict[str, np.ndarray]
  	ONNX tensor to be verified
  	{
  	output_name: np.ndarray,
  	output_name: np.ndarray,
  	:
  	}
  	tf_tensors: List[np.ndarray]
  	TF tensor to be verified
  	[
  	np.ndarray,
  	np.ndarray,
  	:
  	]
  	rtol: float=1e-05
  	The relative tolerance parameter
  	atol: float=1e-05
  	The absolute tolerance parameter
  	Returns
  	----------
  	check_results: Dict[str, List[np.ndarray, int]]
  	Tensor Comparison Results
  	{
  	onnx_output_name: [
  	onnx_tensor,
  	matched_flg, <--- 0: Unmatched, 1: Matched, 2: Skipped (Deleted or Shape Unmatched),
  	max_abs_err,
  	]
  	}
  	"""
  	check_results = {
  	onnx_output_name: [onnx_tensor, False, 0.0] \
  	for onnx_output_name, onnx_tensor in onnx_tensor_infos.items()
  	}
  	for onnx_output_name, onnx_check_info in check_results.items():
  	onnx_tensor: np.ndarray = onnx_check_info[0] # onnx_tensor
  	tf_tensor: np.ndarray = tf_tensor_infos[onnx_output_name] # tf_tensor
  	onnx_tensor_shape = onnx_tensor.shape
  	max_abs_err = ONNX_INF_INDEX_VALUE
  	"""
  	onnx_dummy_data: np.random.random_sample([1,3,224,224])
  	tf_dummy_data : onnx_dummy_data.transpose([0,2,3,1]), len(tf_tensor.shape) == 4
  	tf_shape_transpose_perms:
  	[
  	(0, 1, 2, 3), (0, 1, 3, 2), (0, 2, 1, 3), (0, 2, 3, 1), (0, 3, 1, 2),
  	(0, 3, 2, 1), (1, 0, 2, 3), (1, 0, 3, 2), (1, 2, 0, 3), (1, 2, 3, 0),
  	(1, 3, 0, 2), (1, 3, 2, 0), (2, 0, 1, 3), (2, 0, 3, 1), (2, 1, 0, 3),
  	(2, 1, 3, 0), (2, 3, 0, 1), (2, 3, 1, 0), (3, 0, 1, 2), (3, 0, 2, 1),
  	(3, 1, 0, 2), (3, 1, 2, 0), (3, 2, 0, 1), (3, 2, 1, 0)
  	]
  	tf_target_transpose_perms:
  	[(0, 3, 1, 2), (0, 3, 2, 1)]
  	"""
  	tf_shape_transpose_perms = list(itertools.permutations(range(len(tf_tensor.shape))))
  	tf_target_transpose_perms = [
  	tf_shape_transpose_perm \
  	for tf_shape_transpose_perm in tf_shape_transpose_perms \
  	if tf_tensor.transpose(tf_shape_transpose_perm).shape == onnx_tensor_shape
  	]
  	# Validation
  	"""
  	tf_check_infos:
  	{
  	[
  	tf_target_transpose_perm, <--- tf_target_transpose_perms[idx]
  	matched_flg, <--- True: Matched, False: Unmatched
  	]
  	}
  	"""
  	validate_result = False
  	tf_check_infos = [
  	[tf_target_transpose_perm, 0] for tf_target_transpose_perm in tf_target_transpose_perms
  	]
  	for tf_check_info in tf_check_infos:
  	if len(onnx_tensor_shape) > 1:
  	tf_transposed_tensor = tf_tensor.transpose(tf_check_info[0])
  	if np.allclose(a=onnx_tensor, b=tf_transposed_tensor, rtol=rtol, atol=atol, equal_nan=True):
  	# Matched
  	tf_check_info[1] = 1
  	max_abs_err = 0.0
  	break
  	else:
  	# Unmatched
  	if onnx_tensor.shape == tf_transposed_tensor.shape:
  	error_value = np.max(np.abs(onnx_tensor - tf_transposed_tensor))
  	max_abs_err = error_value if error_value < max_abs_err else max_abs_err
  	else:
  	tf_check_info[1] = 2
  	max_abs_err = 0.0
  	# Validation results check
  	for tf_check_info in tf_check_infos:
  	if tf_check_info[1]:
  	validate_result = tf_check_info[1]
  	break
  	if not validate_result and max_abs_err == ONNX_INF_INDEX_VALUE:
  	# Tensors deleted from the TensorFlow model structure during
  	# the model optimization process are not comparable,
  	# so the status is rewritten to Skip.
  	# If there was no match between ONNX and TensorFlow output shapes.
  	check_results[onnx_output_name][1] = 2
  	check_results[onnx_output_name][2] = max_abs_err
  	else:
  	check_results[onnx_output_name][1] = validate_result
  	check_results[onnx_output_name][2] = max_abs_err
  	return check_results

• Search for the channel arrangement with the lowest accuracy error.
• The Transpose OP is excluded from the search because the value to be compared does not change.
• Not implemented for Constant and ConstantOfShape.
• Other than the above, OPs that do not involve conversion of values are excluded from processing.
• For error correction, the same method as for pre_process_transpose_perm is used to check all combinations that pre-transpose the input tensor to the corresponding OP.
• If the target OP is specified in parameter_replacement.json, omit processing.
• Remove any intrinsic processing already implemented in MatMul.
• OPs with both attribute and tensor values, such as Softmax, attempt correction in the following order: attribute value change first, then tensor transposition.

[PINTO0309]

I close because I now know how to address this issue.

• [InSPyReNet] Swin Transformer Support question #312
• [DN-DAB-DETR] The output of ONNX's Mul OP is different from the TFLite's output. #327
• [TODO] Implement a process to reduce accuracy degradation due to transposition errors in the Transformer's MatMul input values. #317