Tracking issue for Fortran frontend test failures on multi_sdfg branch.

[pratyai]

This is a tracking bug for the many Fortran frontend tests that are currently failing on multi_sdfg branch. I'll add more to the list as I find them.

Things to consider

• Many tests programs are invalid Fortran code. We need to verify first that what we parse are compilable Fortran in the first place (e.g. with gfortran -Wall -c).
• For the same reason as above, even if the test is passing, there is a small possibility that the test would fail if a valid Fortran code is given.

At commit d0025e3

• tests/fortran/intrinsic_all_test.py: 8 out of 9 tests fail. There are three failure points, which might also be related themselves:

>       return ast_internal_classes.Array_Subscript_Node(name=node.name, type=node.type, indices=newer_indices, line_number=node.line_number)
E       AttributeError: 'Array_Subscript_Node' object has no attribute 'type'
../../dace/frontend/fortran/ast_transforms.py:1269: AttributeError

>           offsets = scope_vars.get_var(node.parent, node.name.name).offsets
E           AttributeError: 'NoneType' object has no attribute 'parent'
../../dace/frontend/fortran/ast_transforms.py:1804: AttributeError

>           offsets = scope_vars.get_var(node.parent, node.name.name).offsets
E           AttributeError: 'NoneType' object has no attribute 'parent'
../../dace/frontend/fortran/ast_transforms.py:1804: AttributeError

• tests/fortran/intrinsic_any_test.py: 8 out of 9 tests fail. Same as above.
• tests/fortran/intrinsic_count_test.py: 8 out of 9 tests fail. Same as above.
• tests/fortran/intrinsic_merge_test.py: 5 out of 6 tests fail. Same as above.
• tests/fortran/intrinsic_minmaxval_test.py: 4 out of 4 tests fail. Same as above.
• tests/fortran/intrinsic_product_test.py: 2 out of 3 tests fail. Same as above.
• tests/fortran/intrinsic_sum_test.py: 4 out of 4 tests fail. Same as above.
• tests/fortran/intrinsic_blas_test.py: 2 out of 2 tests fail. Both at the same failure point:

>                       data_path = edge.data.data.split(".")
E                       AttributeError: 'NoneType' object has no attribute 'split'
../../dace/sdfg/replace.py:203: AttributeError

• tests/fortran/intrinsic_math_test.py: 1 out of 18 tests fail. Reasons:

>               self.nodes[child.name_pointer.name] = child.name_target
E               AttributeError: 'Data_Ref_Node' object has no attribute 'name'
../../dace/frontend/fortran/ast_transforms.py:2409: AttributeError

• tests/fortran/prune_test.py: 2 out of 3 tests fail. These will pass if normalizing_offset parameter is set to True.

>       assert (a[2] == 40)
E       assert 42.0 == 40
prune_test.py:97: AssertionError

• tests/fortran/parent_test.py: 1 out of 2 tests fail. Reason:

>       specification = module.specification_part.specifications[0]
E       IndexError: list index out of range
parent_test.py:84: IndexError

• tests/fortran/array_attributes_test.py: 1 out of 10 tests fail. Reason: it generates an invalid SDFG that does not compile.
• tests/fortran/pointer_removal_test.py: 3 out of 4 tests fail. Reasons:

>                       for symbol in var_decl.sizes:
E                       TypeError: 'NoneType' object is not iterable
../../dace/frontend/fortran/ast_transforms.py:2452: TypeError

>       result = super(NestedDict, self).__getitem__(token)
E       KeyError: 's->val1'
../../dace/sdfg/sdfg.py:60: KeyError

>       assert (a[2, 0] == 1410)
E       assert 1.0 == 1410
pointer_removal_test.py:147: AssertionError

[pratyai]

Another set of failures, this time mostly related to structs.

At commit d0025e3

• tests/fortran/type_test.py: 8 of 13 tests pass. Each of them fails in a different way! Reasons:

1. Segmentation fault.

>                   raise ValueError("Unknown type " + type)
E                   ValueError: Unknown type simple_type (__f2dace_SA_w_d_0_s_3: Scalar (dtype=int), __f2dace_SOA_w_d_0_s_3: Scalar (dtype=int), __f2dace_SA_w_d_1_s_4: Scalar (dtype=int), __f2dace_SOA_w_d_1_s_4: Scalar (dtype=int), w: Array (dtype=float, shape=(__f2dace_SA_w_d_0_s_3, __f2dace_SA_w_d_1_s_4)))
../../dace/frontend/fortran/fortran_parser.py:318: ValueError

>           raise RuntimeError(
                f'Structure attribute {node.attr} is not a member of the structure {struct.name} type definition'
            )
E           RuntimeError: Structure attribute wx is not a member of the structure simple_type type definition
../../dace/transformation/passes/lift_struct_views.py:43: RuntimeError

>                   actually_used_pointer_node_finder=ast_transforms.StructPointerChecker(i,cycle[(cycle.index(i)+1)%len(cycle)],point_name)
E                   TypeError: StructPointerChecker.__init__() missing 3 required positional arguments: 'structs_lister', 'struct_dep_graph', and 'analysis'
../../dace/frontend/fortran/fortran_parser.py:2720: TypeError

>           raise FortranSyntaxError(string, "")
E           fparser.two.utils.FortranSyntaxError: at line 27
E           >>>                    st.a.shape=[st.a_size]
../../../../dace/venv/lib/python3.12/site-packages/fparser/two/Fortran2003.py:270: FortranSyntaxError

E               dace.sdfg.validation.InvalidSDFGNodeError: Node validation failed: Missing symbols on nested SDFG: ['n'] (at state stateinternal_function1, node internal_function1)
E               Originating from source code at File "/Users/pmz/gitspace/tmpdace/dace/dace/frontend/fortran/fortran_parser.py", line 1722
../../dace/sdfg/validation.py:393: InvalidSDFGNodeError

>       raise KeyError("Variable {} has not been defined".format(name))
E       KeyError: 'Variable st->a has not been defined'
../../dace/codegen/dispatcher.py:84: KeyError

>       return self.typename or self.type.__name__
E       AttributeError: 'pointer' object has no attribute 'typename'
../../dace/dtypes.py:416: AttributeError

• tests/fortran/type_array_test.py: 3 out of 3 tests fail.

1. Fails to compile:

E               /Users/pmz/gitspace/tmpdace/dace/tests/fortran/.dacecache/type2_array_test_function0/src/cpu/type2_array_test_function0.cpp:34:46: error: '__f2dace_SA_w_d_0_s_2_pprog_p_1' was not declared in this scope
E                  34 |             d_out_1 = my_arr_0_in_pprog_w[(((__f2dace_SA_w_d_0_s_2_pprog_p_1 * (1 - __f2dace_SOA_w_d_1_s_3_pprog_p_1)) - __f2dace_SOA_w_d_0_s_2_pprog_p_1) + 1)];
E                     |                                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

>                   raise ValueError("Unknown type " + type)
E                   ValueError: Unknown type simple_type (w: Array (dtype=float, shape=(5, 5)))
../../dace/frontend/fortran/fortran_parser.py:318: ValueError

>                   raise ValueError("Unknown type " + type)
E                   ValueError: Unknown type simple_type (__f2dace_SA_w_d_0_s_2: Scalar (dtype=int), __f2dace_SOA_w_d_0_s_2: Scalar (dtype=int), __f2dace_SA_w_d_1_s_3: Scalar (dtype=int), __f2dace_SOA_w_d_1_s_3: Scalar (dtype=int), w: Array (dtype=float, shape=(__f2dace_SA_w_d_0_s_2_pprog_p_3, __f2dace_SA_w_d_1_s_3_pprog_p_3)))
../../dace/frontend/fortran/fortran_parser.py:318: ValueError

• tests/fortran/struct_test.py: 2 out of 2 tests fail. Reasons:

>       return self.typename or self.type.__name__
E       AttributeError: 'pointer' object has no attribute 'typename'
../../dace/dtypes.py:416: AttributeError

>       size_lhs = lhs_(0).itemsize
E       TypeError: abstract class
../../dace/dtypes.py:574: TypeError

• tests/fortran/nested_array_test.py: 1 out of 2 tests fail. Reason:

>           raise FortranSyntaxError(string, "")
E           fparser.two.utils.FortranSyntaxError: at line 5
E           >>>                     simple_type
../../../../dace/venv/lib/python3.12/site-packages/fparser/two/Fortran2003.py:270: FortranSyntaxError

• tests/fortran/missing_func_test.py: 1 out of 2 tests fail. Reason:

>       assert (a[1, 0] == 6.5)
E       assert 5.5 == 6.5
missing_func_test.py:102: AssertionError

• tests/fortran/cond_type_test.py: 1 out of 2 tests fail. Reason:

>       raise KeyError("Variable {} has not been defined".format(name))
E       KeyError: 'Variable ptr_patch->id has not been defined'
../../dace/codegen/dispatcher.py:84: KeyError

• tests/fortran/global_test.py: 1 out of 1 test fails. Reason:

>               raise InvalidSDFGInterstateEdgeError(
                    f'Undefined symbols in edge: {undef_syms}. Add those with '
                    '`sdfg.add_symbol()` or define outside with `dace.symbol()`', sdfg, eid)
E               dace.sdfg.validation.InvalidSDFGInterstateEdgeError: Undefined symbols in edge: {'__f2dace_SOA_w_d_2_s_17'}. Add those with `sdfg.add_symbol()` or define outside with `dace.symbol()` (at edge "tmp_parfor_2=__f2dace_SOA_w_d_2_s_17" (BeginFOR_l_19_c_19 -> GuardFOR_l_19_c_19)
../../dace/sdfg/validation.py:82: InvalidSDFGInterstateEdgeError

• tests/fortran/advanced_optional_args_test.py: 1 out of 1 test fails. This goes away with normalize_offset = True.

>       assert res[0] == 5
E       assert 42 == 5
advanced_optional_args_test.py:87: AssertionError

[mcopik]

@pratyai Thanks for the hard work of collecting those!

@acalotoiu Should just we make normalize_offset = True the default case everywhere?

[mcopik]

The first problem with intrinsic should be fixable by adding VOID; I tested it on one example and it works.

I also found the reason why the second one fails. The problem is that we need a very specific order of transformations: we need to apply intrinsic before running call argument extraction. Otherwise, the argument is replaced with a temporary one, and I have no chance of understanding what the input is. So, ArgumentExtractor needs to run after intrinsic.

@pratyai @acalotoiu I'm 100% sure I fixed this problem at some point, and I see now that we sometimes call this transformation twice - likely, the fixed version was broken again. I will try to fix it, but it would be good to finally merge all versions of create_sdfg_from_* into a single one. There are multiple dependencies on the order of transformations, and it's not good to keep multiple definitions of it.

So far, I pushed three commits that make intrinsic's all test work. I will retest it on other as well, but again - we should just have one parsing function.