Get use- and colorful output from compile.sh

[JBludau]

This originated from a discussion with @tylerjereddy on slack.
In short: We would like to have a better error handling if stuff does not compile. Ideally we want to even have warnings before the compilation stage fails but this is a goal for the future. As a starter I like to propose that we capture the output of the compilation script as good as we can (this includes warnings and errors in the right order and colors if the compiler does a colored and highlighted output).

Apparently scripts detect if the output is piped into a file or a shell and restrict the coloring/hinting if it goes to a file. The linux standard program script can run a command and force the output to be logged as if it were in a shell thus capturing the coloring and other highlights.

If I artificially introduce an error in the generated cpp file I get something like this when running a pykokkos script that compiles the broken cpp file:

Total size S = 262144 N = 256 M = 1024 E = 1024
Total size S = 262144 N = 256 M = 1024

C++ compilation in pk_cpp/pykokkos/examples/kokkos-tutorials/functor/04/04_Workload/Cuda failed. For colored compiler output run 'cat pk_cpp/pykokkos/examples/kokkos-tutorials/functor/04/04_Workload/Cuda/compile.out'

When pasting the newly written .out file with the suggested command I get this rendered in my terminal:

Which is the same as I would get from directly running the compile line on the generated .cpp file

[tylerjereddy]

Looks like the CI is failing for the pre-compilation shims used for array API conformance.

Ignoring that, I tried the problem workunit I described in gh-168 with and without this patch, but I ended up with the same error message because it seems to be a runtime error that isn't capable of providing the actual C++ line that overflows or whatever?

On branch treddy_arange with this reproducer test:

import pykokkos as pk


def main():
    pk.arange(0, 2147483648, 100000, dtype=pk.uint32)


if __name__ == "__main__":
    main()

Before/after I get this:

  File "/home/tyler/github_projects/pykokkos/test.py", line 9, in <module>
    main()
  File "/home/tyler/github_projects/pykokkos/test.py", line 5, in main
    pk.arange(0, 2147483648, 100000, dtype=pk.uint32)
  File "/home/tyler/github_projects/pykokkos/pykokkos/lib/create.py", line 62, in arange
    _ufunc_kernel_dispatcher(tid=size,
  File "/home/tyler/github_projects/pykokkos/pykokkos/lib/ufuncs.py", line 50, in _ufunc_kernel_dispatcher
    ret = sub_dispatcher(tid, desired_workunit, **kwargs)
  File "/home/tyler/github_projects/pykokkos/pykokkos/interface/parallel_dispatch.py", line 175, in parallel_for
    func(**args)
RuntimeError: Unable to cast Python instance of type <class 'int'> to C++ type 'int'
terminate called after throwing an instance of 'std::runtime_error'
  what():  Kokkos allocation "" is being deallocated after Kokkos::finalize was called

Aborted (core dumped)

Let me see if I can dig up a compilation-specific issue that didn't have a good error message.

[tylerjereddy]

Well, I don't have another compilation failure case handy at the moment, but another problem error message scenario is a failed translation like the one in gh-161. For example, doing this on latest develop branch to declare a variable without initializing it (pytest array_api_tests/test_operators_and_elementwise_functions.py::test_exp):

index 89b190c..31ba284 100644
--- a/pykokkos/lib/ufunc_workunits.py
+++ b/pykokkos/lib/ufunc_workunits.py
@@ -3,6 +3,7 @@ import pykokkos as pk
 
 @pk.workunit
 def exp_impl_1d_double(tid: int, view: pk.View1D[pk.double], out: pk.View1D[pk.double]):
+    mod: float
     out[tid] = exp(view[tid])
 

Produces a huge mess of output, some of which is sent to stdout instead of stderr, which doesn't make sense to me:

======================================================================================================================================================================================= FAILURES =======================================================================================================================================================================================
_______________________________________________________________________________________________________________________________________________________________________________________ test_exp _______________________________________________________________________________________________________________________________________________________________________________________

self = <hypothesis.core.StateForActualGivenExecution object at 0x7fed6b49e770>, data = ConjectureData(INTERESTING, 14 bytes, frozen)

    def _execute_once_for_engine(self, data):
        """Wrapper around ``execute_once`` that intercepts test failure
        exceptions and single-test control exceptions, and turns them into
        appropriate method calls to `data` instead.
    
        This allows the engine to assume that any exception other than
        ``StopTest`` must be a fatal error, and should stop the entire engine.
        """
        try:
            trace = frozenset()
            if (
                self.failed_normally
                and not self.failed_due_to_deadline
                and Phase.shrink in self.settings.phases
                and Phase.explain in self.settings.phases
                and sys.gettrace() is None
                and not PYPY
            ):  # pragma: no cover
                # This is in fact covered by our *non-coverage* tests, but due to the
                # settrace() contention *not* by our coverage tests.  Ah well.
                tracer = Tracer()
                try:
                    sys.settrace(tracer.trace)
                    result = self.execute_once(data)
                    if data.status == Status.VALID:
                        self.explain_traces[None].add(frozenset(tracer.branches))
                finally:
                    sys.settrace(None)
                    trace = frozenset(tracer.branches)
            else:
>               result = self.execute_once(data)

../../python_310_pykokkos_work/lib/python3.10/site-packages/hypothesis/core.py:879: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <hypothesis.core.StateForActualGivenExecution object at 0x7fed6b49e770>, data = ConjectureData(INTERESTING, 14 bytes, frozen), print_example = False, is_final = False, expected_failure = None

    def execute_once(
        self, data, print_example=False, is_final=False, expected_failure=None
    ):
        """Run the test function once, using ``data`` as input.
    
        If the test raises an exception, it will propagate through to the
        caller of this method. Depending on its type, this could represent
        an ordinary test failure, or a fatal error, or a control exception.
    
        If this method returns normally, the test might have passed, or
        it might have placed ``data`` in an unsuccessful state and then
        swallowed the corresponding control exception.
        """
    
        self.ever_executed = True
        data.is_find = self.is_find
    
        text_repr = None
        if self.settings.deadline is None:
            test = self.test
        else:
    
            @proxies(self.test)
            def test(*args, **kwargs):
                self.__test_runtime = None
                initial_draws = len(data.draw_times)
                start = time.perf_counter()
                result = self.test(*args, **kwargs)
                finish = time.perf_counter()
                internal_draw_time = sum(data.draw_times[initial_draws:])
                runtime = datetime.timedelta(
                    seconds=finish - start - internal_draw_time
                )
                self.__test_runtime = runtime
                current_deadline = self.settings.deadline
                if not is_final:
                    current_deadline = (current_deadline // 4) * 5
                if runtime >= current_deadline:
                    raise DeadlineExceeded(runtime, self.settings.deadline)
                return result
    
        def run(data):
            # Set up dynamic context needed by a single test run.
            with local_settings(self.settings):
                with deterministic_PRNG():
                    with BuildContext(data, is_final=is_final):
    
                        # Generate all arguments to the test function.
                        args, kwargs = data.draw(self.search_strategy)
                        if expected_failure is not None:
                            nonlocal text_repr
                            text_repr = repr_call(test, args, kwargs)
    
                        if print_example or current_verbosity() >= Verbosity.verbose:
                            output = StringIO()
    
                            printer = RepresentationPrinter(output)
                            if print_example:
                                printer.text("Falsifying example:")
                            else:
                                printer.text("Trying example:")
    
                            if self.print_given_args:
                                printer.text(" ")
                                printer.repr_call(
                                    test.__name__,
                                    args,
                                    kwargs,
                                    force_split=True,
                                )
                            report(printer.getvalue())
                        return test(*args, **kwargs)
    
        # Run the test function once, via the executor hook.
        # In most cases this will delegate straight to `run(data)`.
>       result = self.test_runner(data, run)

../../python_310_pykokkos_work/lib/python3.10/site-packages/hypothesis/core.py:818: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = ConjectureData(INTERESTING, 14 bytes, frozen), function = <function StateForActualGivenExecution.execute_once.<locals>.run at 0x7fed6aa42320>

    def default_new_style_executor(data, function):
>       return function(data)

../../python_310_pykokkos_work/lib/python3.10/site-packages/hypothesis/executors.py:47: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = ConjectureData(INTERESTING, 14 bytes, frozen)

    def run(data):
        # Set up dynamic context needed by a single test run.
        with local_settings(self.settings):
            with deterministic_PRNG():
                with BuildContext(data, is_final=is_final):
    
                    # Generate all arguments to the test function.
                    args, kwargs = data.draw(self.search_strategy)
                    if expected_failure is not None:
                        nonlocal text_repr
                        text_repr = repr_call(test, args, kwargs)
    
                    if print_example or current_verbosity() >= Verbosity.verbose:
                        output = StringIO()
    
                        printer = RepresentationPrinter(output)
                        if print_example:
                            printer.text("Falsifying example:")
                        else:
                            printer.text("Trying example:")
    
                        if self.print_given_args:
                            printer.text(" ")
                            printer.repr_call(
                                test.__name__,
                                args,
                                kwargs,
                                force_split=True,
                            )
                        report(printer.getvalue())
>                   return test(*args, **kwargs)

../../python_310_pykokkos_work/lib/python3.10/site-packages/hypothesis/core.py:814: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x = <pykokkos.interface.views.View object at 0x7fed6a9d74f0>

    @given(xps.arrays(dtype=xps.floating_dtypes(), shape=hh.shapes()))
    def test_exp(x):
>       out = xp.exp(x)

array_api_tests/test_operators_and_elementwise_functions.py:828: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

view = <pykokkos.interface.views.View object at 0x7fed6a9d74f0>

    def exp(view):
        """
        Element-wise exp of the view.
    
        Parameters
        ----------
        view : pykokkos view
               Input view.
    
        Returns
        -------
        out : pykokkos view
               Output view.
    
        """
        dtype = view.dtype
        ndims = len(view.shape)
        if ndims > 2:
            raise NotImplementedError("exp() ufunc only supports up to 2D views")
        if view.size == 0:
            return view
        out = pk.View([*view.shape], dtype=dtype)
        if view.shape == ():
            tid = 1
        else:
            tid = view.shape[0]
>       _ufunc_kernel_dispatcher(tid=tid,
                                 dtype=dtype,
                                 ndims=ndims,
                                 op="exp",
                                 sub_dispatcher=pk.parallel_for,
                                 out=out,
                                 view=view)

../pykokkos/pykokkos/lib/ufuncs.py:2138: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

tid = 1, dtype = <class 'pykokkos.interface.data_types.float64'>, ndims = 1, op = 'exp', sub_dispatcher = <function parallel_for at 0x7fed74d36cb0>, kwargs = {'out': <pykokkos.interface.views.View object at 0x7fed6a9d44c0>, 'view': <pykokkos.interface.views.View object at 0x7fed6a9d74f0>}, dtype_extractor = re.compile('.*(?:dtype|data_types|DataType)\\.(\\w+)')
res = <re.Match object; span=(0, 45), match="<class 'pykokkos.interface.data_types.float64">, dtype_str = 'double', function_name_str = 'exp_impl_1d_double', desired_workunit = <function exp_impl_1d_double at 0x7fed74db7910>

    def _ufunc_kernel_dispatcher(tid,
                                 dtype,
                                 ndims,
                                 op,
                                 sub_dispatcher,
                                 **kwargs):
        dtype_extractor = re.compile(r".*(?:dtype|data_types|DataType)\.(\w+)")
        if ndims == 0:
            ndims = 1
        res = dtype_extractor.match(str(dtype))
        dtype_str = res.group(1)
        if dtype_str == "float32":
            dtype_str = "float"
        elif dtype_str == "float64":
            dtype_str = "double"
        function_name_str = f"{op}_impl_{ndims}d_{dtype_str}"
        desired_workunit = kernel_dict[function_name_str]
        # call the kernel
>       ret = sub_dispatcher(tid, desired_workunit, **kwargs)

../pykokkos/pykokkos/lib/ufuncs.py:41: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (1, <function exp_impl_1d_double at 0x7fed74db7910>), kwargs = {'out': <pykokkos.interface.views.View object at 0x7fed6a9d44c0>, 'view': <pykokkos.interface.views.View object at 0x7fed6a9d74f0>}, args_to_hash = [<pykokkos.interface.views.View object at 0x7fed6a9d44c0>, <pykokkos.interface.views.View object at 0x7fed6a9d74f0>, 'exp_impl_1d_double']
args_not_to_hash = {}, k = 'view', v = <pykokkos.interface.views.View object at 0x7fed6a9d74f0>, a = <function exp_impl_1d_double at 0x7fed74db7910>, to_hash = frozenset({<pykokkos.interface.views.View object at 0x7fed6a9d74f0>, 'exp_impl_1d_double', <pykokkos.interface.views.View object at 0x7fed6a9d44c0>}), cache_key = -6272884706014322896

    def parallel_for(*args, **kwargs) -> None:
        """
        Run a parallel for loop
    
        :param *args:
            :param name: (optional) name of the kernel
            :param policy: the execution policy, either a RangePolicy,
                TeamPolicy, TeamThreadRange, ThreadVectorRange, or an
                integer representing the number of threads
            :param workunit: the workunit to be run in parallel
            :param view: (optional) the view being initialized
    
        :param **kwargs: the keyword arguments passed to a standalone
            workunit
        """
    
        args_to_hash: List = []
        args_not_to_hash: Dict = {}
        for k, v in kwargs.items():
            if not isinstance(v, int):
                args_to_hash.append(v)
            else:
                args_not_to_hash[k] = v
    
        # Hash the workunit
        for a in args:
            if callable(a):
                args_to_hash.append(a.__name__)
                break
    
        to_hash = frozenset(args_to_hash)
        cache_key: int = hash(to_hash)
    
        if cache_key in workunit_cache:
            dead_obj = 0
            func, newargs = workunit_cache[cache_key]
            for arg in newargs.values():
                # see gh-34
                # reject cache retrieval when an object in the
                # cache has a reference count of 0 (presumably
                # only possible because of the C++/pybind11 infra;
                # normally a refcount of 1 is the lowest for pure
                # Python objects)
                # NOTE: is the cache genuinely useful now though?
                ref_count = len(gc.get_referrers(arg))
                if ref_count == 0:
                    dead_obj += 1
                    break
            if not dead_obj:
                args = newargs
                args.update(args_not_to_hash)
                func(**args)
                return
    
        handled_args: HandledArgs = handle_args(True, args)
>       func, args = runtime_singleton.runtime.run_workunit(
            handled_args.name,
            handled_args.policy,
            handled_args.workunit,
            "for",
            **kwargs)

../pykokkos/pykokkos/interface/parallel_dispatch.py:167: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pykokkos.core.runtime.Runtime object at 0x7feda89cb8b0>, name = None, policy = <pykokkos.interface.execution_policy.RangePolicy object at 0x7fed6a9d4af0>, workunit = <function exp_impl_1d_double at 0x7fed74db7910>, operation = 'for', initial_value = 0
kwargs = {'out': <pykokkos.interface.views.View object at 0x7fed6a9d44c0>, 'view': <pykokkos.interface.views.View object at 0x7fed6a9d74f0>}, members = <pykokkos.core.translators.members.PyKokkosMembers object at 0x7fed6b0dca30>, module_setup = <pykokkos.core.module_setup.ModuleSetup object at 0x7fed6b0dce50>

    def run_workunit(
        self,
        name: Optional[str],
        policy: ExecutionPolicy,
        workunit: Callable[..., None],
        operation: Optional[str] = None,
        initial_value: Union[float, int] = 0,
        **kwargs
    ) -> Optional[Union[float, int]]:
        """
        Run the workunit
    
        :param name: the name of the kernel
        :param policy: the execution policy of the operation
        :param workunit: the workunit function object
        :param kwargs: the keyword arguments passed to the workunit
        :param operation: the name of the operation "for", "reduce", or "scan"
        :param initial_value: the initial value of the accumulator
        :returns: the result of the operation (None for parallel_for)
        """
    
        if self.is_debug(policy.space):
            if operation is None:
                raise RuntimeError("ERROR: operation cannot be None for Debug")
            return run_workunit_debug(policy, workunit, operation, initial_value, **kwargs)
    
        members: Optional[PyKokkosMembers] = self.precompile_workunit(workunit,policy.space)
        if members is None:
            raise RuntimeError("ERROR: members cannot be none")
    
        module_setup: ModuleSetup = self.get_module_setup(workunit, policy.space)
>       return self.execute(workunit, module_setup, members, policy.space, policy=policy, name=name, **kwargs)

../pykokkos/pykokkos/core/runtime.py:102: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pykokkos.core.runtime.Runtime object at 0x7feda89cb8b0>, entity = <function exp_impl_1d_double at 0x7fed74db7910>, module_setup = <pykokkos.core.module_setup.ModuleSetup object at 0x7fed6b0dce50>, members = <pykokkos.core.translators.members.PyKokkosMembers object at 0x7fed6b0dca30>, space = <ExecutionSpace.OpenMP: 'OpenMP'>
policy = <pykokkos.interface.execution_policy.RangePolicy object at 0x7fed6a9d4af0>, name = None, kwargs = {'out': <pykokkos.interface.views.View object at 0x7fed6a9d44c0>, 'view': <pykokkos.interface.views.View object at 0x7fed6a9d74f0>}
module_path = 'pk_cpp/home/tyler/python_310_pykokkos_work/bin/pyt/ufunc_workunits_exp_impl_1d_double/OpenMP/kernel.cpython-310-x86_64-linux-gnu.so'

    def execute(
        self,
        entity: Union[object, Callable[..., None]],
        module_setup: ModuleSetup,
        members: PyKokkosMembers,
        space: ExecutionSpace,
        policy: Optional[ExecutionPolicy] = None,
        name: Optional[str] = None,
        **kwargs
    ) -> Optional[Union[float, int]]:
        """
        Imports the module containing the bindings and executes the necessary function
    
        :param entity: the workload or workunit object
        :param module_path: the path to the compiled module
        :param members: a collection of PyKokkos related members
        :param space: the execution space
        :param policy: the execution policy for workunits
        :param name: the name of the kernel
        :param kwargs: the keyword arguments passed to the workunit
        :returns: the result of the operation (None for "for" and workloads)
        """
    
        module_path: str
        if is_host_execution_space(space) or not km.is_multi_gpu_enabled():
            module_path = module_setup.path
        else:
            device_id: int = km.get_device_id()
            module_path = module_setup.gpu_module_paths[device_id]
    
>       module = self.import_module(module_setup.name, module_path)

../pykokkos/pykokkos/core/runtime.py:145: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pykokkos.core.runtime.Runtime object at 0x7feda89cb8b0>, module_name = 'pk_cpp_home_tyler_python_310_pykokkos_work_bin_pyt_ufunc_workunits_exp_impl_1d_double_OpenMP_kernel_cpython_310_x86_64_linux_gnu_so', module_path = 'pk_cpp/home/tyler/python_310_pykokkos_work/bin/pyt/ufunc_workunits_exp_impl_1d_double/OpenMP/kernel.cpython-310-x86_64-linux-gnu.so'

    def import_module(self, module_name: str, module_path: str):
        """
        Import a compiled module
    
        :param module_name: the name of the compiled module
        :param module_path: the path to the compiled module
        :returns: the imported module
        """
    
        hashed_name: str = module_name.replace("kernel", f"kernel_{km.get_device_id()}")
    
        if hashed_name in sys.modules:
            return sys.modules[hashed_name]
    
        spec = importlib.util.spec_from_file_location(module_name, module_path)
>       module = importlib.util.module_from_spec(spec)

../pykokkos/pykokkos/core/runtime.py:176: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

spec = ModuleSpec(name='pk_cpp_home_tyler_python_310_pykokkos_work_bin_pyt_ufunc_workunits_exp_impl_1d_double_OpenMP_kernel_c...ler/python_310_pykokkos_work/bin/pyt/ufunc_workunits_exp_impl_1d_double/OpenMP/kernel.cpython-310-x86_64-linux-gnu.so')

>   ???

<frozen importlib._bootstrap>:571: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_frozen_importlib_external.ExtensionFileLoader object at 0x7fed6a9d6980>, spec = ModuleSpec(name='pk_cpp_home_tyler_python_310_pykokkos_work_bin_pyt_ufunc_workunits_exp_impl_1d_double_OpenMP_kernel_c...ler/python_310_pykokkos_work/bin/pyt/ufunc_workunits_exp_impl_1d_double/OpenMP/kernel.cpython-310-x86_64-linux-gnu.so')

>   ???

<frozen importlib._bootstrap_external>:1176: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

f = <built-in function create_dynamic>, args = (ModuleSpec(name='pk_cpp_home_tyler_python_310_pykokkos_work_bin_pyt_ufunc_workunits_exp_impl_1d_double_OpenMP_kernel_...r/python_310_pykokkos_work/bin/pyt/ufunc_workunits_exp_impl_1d_double/OpenMP/kernel.cpython-310-x86_64-linux-gnu.so'),), kwds = {}

>   ???
E   ImportError: /home/tyler/github_projects/array-api-tests/pk_cpp/home/tyler/python_310_pykokkos_work/bin/pyt/ufunc_workunits_exp_impl_1d_double/OpenMP/kernel.cpython-310-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

<frozen importlib._bootstrap>:241: ImportError

During handling of the above exception, another exception occurred:

    @given(xps.arrays(dtype=xps.floating_dtypes(), shape=hh.shapes()))
>   def test_exp(x):

array_api_tests/test_operators_and_elementwise_functions.py:827: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <hypothesis.internal.conjecture.datatree.TreeRecordingObserver object at 0x7fed6a9d6590>, status = Status.INTERESTING, interesting_origin = (<class 'ImportError'>, '<frozen importlib._bootstrap>', 241, (), ())

    def conclude_test(self, status, interesting_origin):
        """Says that ``status`` occurred at node ``node``. This updates the
        node if necessary and checks for consistency."""
        if status == Status.OVERRUN:
            return
        i = self.__index_in_current_node
        node = self.__current_node
    
        if i < len(node.values) or isinstance(node.transition, Branch):
            inconsistent_generation()
    
        new_transition = Conclusion(status, interesting_origin)
    
        if node.transition is not None and node.transition != new_transition:
            # As an, I'm afraid, horrible bodge, we deliberately ignore flakiness
            # where tests go from interesting to valid, because it's much easier
            # to produce good error messages for these further up the stack.
            if isinstance(node.transition, Conclusion) and (
                node.transition.status != Status.INTERESTING
                or new_transition.status != Status.VALID
            ):
>               raise Flaky(
                    f"Inconsistent test results! Test case was {node.transition!r} "
                    f"on first run but {new_transition!r} on second"
                )
E               hypothesis.errors.Flaky: Inconsistent test results! Test case was Conclusion(status=Status.INTERESTING, interesting_origin=(<class 'SystemExit'>, '/home/tyler/github_projects/pykokkos/pykokkos/core/translators/static.py', 223, (<class 'AttributeError'>, '/usr/lib/python3.10/ast.py', 252, (), ()), ())) on first run but Conclusion(status=Status.INTERESTING, interesting_origin=(<class 'ImportError'>, '<frozen importlib._bootstrap>', 241, (), ())) on second

../../python_310_pykokkos_work/lib/python3.10/site-packages/hypothesis/internal/conjecture/datatree.py:406: Flaky
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Translation of <ast.FunctionDef object at 0x7fed6b131420> failed
=================================================================================================================================================================================== warnings summary ===================================================================================================================================================================================
../../python_310_pykokkos_work/lib/python3.10/site-packages/hypothesis/extra/array_api.py:925
  /home/tyler/python_310_pykokkos_work/lib/python3.10/site-packages/hypothesis/extra/array_api.py:925: HypothesisWarning: Could not determine whether module pykokkos is an Array API library
    warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=============================================================================================================================================================================== short test summary info ================================================================================================================================================================================
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_exp - hypothesis.errors.Flaky: Inconsistent test results! Test case was Conclusion(status=Status.INTERESTING, interesting_origin=(<class 'SystemExit'>, '/home/tyler/github_projects/pykokkos/pykokkos/core/translators/static.py', 223, (<class 'AttributeError'>, '/usr/lib/python3.10/ast.py', 252, (), (...
============================================================================================================================================================================ 1 failed, 1 warning in 11.12s =============================================================================================================================================================================
terminate called after throwing an instance of 'std::runtime_error'
  what():  Kokkos allocation "" is being deallocated after Kokkos::finalize was called

Aborted (core dumped)

Obviously, that output doesn't change with this patch since it is a translation issue.

[JBludau]

import pykokkos as pk


def main():
    pk.arange(0, 2147483648, 100000, dtype=pk.uint32)


if __name__ == "__main__":
    main()

Before/after I get this:

  File "/home/tyler/github_projects/pykokkos/test.py", line 9, in <module>
    main()
  File "/home/tyler/github_projects/pykokkos/test.py", line 5, in main
    pk.arange(0, 2147483648, 100000, dtype=pk.uint32)
  File "/home/tyler/github_projects/pykokkos/pykokkos/lib/create.py", line 62, in arange
    _ufunc_kernel_dispatcher(tid=size,
  File "/home/tyler/github_projects/pykokkos/pykokkos/lib/ufuncs.py", line 50, in _ufunc_kernel_dispatcher
    ret = sub_dispatcher(tid, desired_workunit, **kwargs)
  File "/home/tyler/github_projects/pykokkos/pykokkos/interface/parallel_dispatch.py", line 175, in parallel_for
    func(**args)
RuntimeError: Unable to cast Python instance of type <class 'int'> to C++ type 'int'
terminate called after throwing an instance of 'std::runtime_error'
  what():  Kokkos allocation "" is being deallocated after Kokkos::finalize was called

Aborted (core dumped)

hmm, this looks like a runtime error thrown by bounds checking before casting python variables into c++ types. It looks like even though you requested pk.uint32 something is trying to do a conversion into an int. With 2147483648 being to large for an integer a useful bounds check would throw an error here. This might be hard to find, as it might depend on some configure time specific defaults for template arguments in kokkos core. I can try looking into this next week

[JBludau]

Well, I don't have another compilation failure case handy at the moment, but another problem error message scenario is a failed translation like the one in gh-161. For example, doing this on latest develop branch to declare a variable without initializing it (pytest array_api_tests/test_operators_and_elementwise_functions.py::test_exp):

index 89b190c..31ba284 100644
--- a/pykokkos/lib/ufunc_workunits.py
+++ b/pykokkos/lib/ufunc_workunits.py
@@ -3,6 +3,7 @@ import pykokkos as pk
 
 @pk.workunit
 def exp_impl_1d_double(tid: int, view: pk.View1D[pk.double], out: pk.View1D[pk.double]):
+    mod: float
     out[tid] = exp(view[tid])
 

Produces a huge mess of output, some of which is sent to stdout instead of stderr, which doesn't make sense to me:

Obviously, that output doesn't change with this patch since it is a translation issue.

hmm this looks like we try to load a .so but fail to do so ... I will try to look into better error handling or retriggering a compilation if the module is not found. Nevertheless, if the module is not able to compile (running in a clean state, e.g. without the pk_cpp folder), it should raise an error stating that it can not be compiled. With the patch here you should be able to look at the colored compiler output afterwards.

[JBludau]

This now checks for ANY .so file in the path. Timing with a 1000 iterations shows, that asking if the dir exists has the same cost as checking for the .so file (if only one is there). This also triggers a recompile if the .so was not found. And if the recompile fails gives the instructions for the colored compiler output.

Nevertheless, we do not check for the name, as this would require a larger refactor (the method could no longer be static or we would need to pass in the module name from outside). For the different backends we should be fine, as the backend name is contained in the folder path. But we might still need the module names, as we have this multigpu thing where multiple .so files with different names exist in the same folder. But someone first using only one gpu and then multiple without clearing the cache might be an edge case we do not care about...

[JBludau]

Well, I don't have another compilation failure case handy at the moment, but another problem error message scenario is a failed translation like the one in gh-161. For example, doing this on latest develop branch to declare a variable without initializing it (pytest array_api_tests/test_operators_and_elementwise_functions.py::test_exp):

index 89b190c..31ba284 100644
--- a/pykokkos/lib/ufunc_workunits.py
+++ b/pykokkos/lib/ufunc_workunits.py
@@ -3,6 +3,7 @@ import pykokkos as pk
 
 @pk.workunit
 def exp_impl_1d_double(tid: int, view: pk.View1D[pk.double], out: pk.View1D[pk.double]):
+    mod: float
     out[tid] = exp(view[tid])
 

Produces a huge mess of output, some of which is sent to stdout instead of stderr, which doesn't make sense to me:
Obviously, that output doesn't change with this patch since it is a translation issue.

hmm this looks like we try to load a .so but fail to do so ... I will try to look into better error handling or retriggering a compilation if the module is not found. Nevertheless, if the module is not able to compile (running in a clean state, e.g. without the pk_cpp folder), it should raise an error stating that it can not be compiled. With the patch here you should be able to look at the colored compiler output afterwards.

@tylerjereddy could you retry if this gives you a compilation error? If not we might need to generate a hash from the generated source we send to the compiler after translation in order to check if the source has changed between runs ... this would be a major redesign thing

[tylerjereddy]

This should always just be linux for any version of Python we support now.

[tylerjereddy]

Why are we printing instead of using Python error classes/handling here?

[tylerjereddy]

For the mod: float example above, I see no improvement on this branch:

Translation of <ast.FunctionDef object at 0x7f53c88238b0> failed

Which failure exactly do you want me to try on develop vs. this branch?

[tylerjereddy]

I think I'm a bit confused--can't we just use standard Python error handling/output machinery to form a nicely-constructed error without requiring a command-line callout? If we really need color there is portable stuff like https://github.com/Textualize/rich, but first I want to know which errors I should compare on develop vs. this branch to see an improvement? (i.e., do you have a reproducer I can try?)