add SimpleArray caster #266

tigercosmos · 2023-12-26T15:57:37Z

Introduced new SimpleArrayPlex and its wrapper WrapSimpleArrayPlex. The interface of SimpleArray and WrapSimpleArrayPlex should be the same as SimpleArray and WrapSimpleArray, but this should be a to-do for the next PR.
Implemented the customized caster for SimpleArray to allow SimpleArrayPlex implicit casting.
add a testhelper module for testing

the first step for #27

tigercosmos · 2023-12-26T15:58:59Z

cpp/modmesh/CMakeLists.txt

@@ -39,6 +39,7 @@ add_subdirectory(python)
 add_subdirectory(spacetime)
 add_subdirectory(view)
 add_subdirectory(inout)
+add_subdirectory(testhelper)


some tests require python binding functions or classes. We can put those test helpers in this testhelper module in the future.

tigercosmos · 2023-12-26T16:01:46Z

cpp/modmesh/buffer/SimpleArray.hpp

@@ -82,13 +84,17 @@ inline size_t buffer_offset(small_vector<size_t> const & stride, small_vector<si
 namespace detail
 {

+using shape_type = small_vector<size_t>;


the shape_type is not related to T, I think it's better to put it outside.

tigercosmos · 2023-12-26T16:02:04Z

cpp/modmesh/buffer/SimpleArray.hpp

 template <typename T>
 struct SimpleArrayInternalTypes
 {
+    using shape_type = detail::shape_type;


backward compatibility

Please do not reorder them. The diff looks ugly.

the order should be back.

tigercosmos · 2023-12-26T16:02:43Z

cpp/modmesh/buffer/SimpleArray.hpp

@@ -654,6 +659,86 @@ using is_simple_array = std::is_same<
 template <typename S>
 inline constexpr bool is_simple_array_v = is_simple_array<S>::value;

+using SimpleArrayBool = SimpleArray<bool>;


for later macro usage in SimpleArrayCaster.hpp

This is not beautiful, but I can live with it.

tigercosmos · 2023-12-26T16:03:25Z

cpp/modmesh/buffer/SimpleArray.hpp

+        return m_instance_ptr;
+    }
+
+    /// TODO: add all SimpleArray public methods


to let the PR be simple, no other methods for now.

tigercosmos · 2023-12-26T16:04:37Z

cpp/modmesh/buffer/pymod/SimpleArrayCaster.hpp

+
+#define ARRAYPLEX_TYPE_CASTER(DATATYPE)                                                                                                                       \
+    template <> /* NOLINTNEXTLINE(bugprone-macro-parentheses) */                                                                                              \
+    struct type_caster<modmesh::SimpleArray##DATATYPE> : public type_caster_base<modmesh::SimpleArray##DATATYPE>                                              \


the most important part for casting.

It should use a comment to highlight the importance.

added comment at the beginning of this header file.

tigercosmos · 2023-12-26T16:05:19Z

cpp/modmesh/buffer/pymod/wrap_SimpleArrayPlex.cpp

+                    { return wrapped_type(make_shape(shape), datatype); }),
+                pybind11::arg("shape"),
+                pybind11::arg("dtype"))
+            /// TODO: should have the same interface as WrapSimpleArray


the python module should have more methods, but keep it simple for now.

tigercosmos · 2023-12-26T16:06:00Z

cpp/modmesh/python/module.cpp

@@ -49,6 +50,10 @@ void initialize(pybind11::module_ mod)
    initialize_spacetime(spacetime_mod);
    pybind11::module_ onedim_mod = mod.def_submodule("onedim", "onedim");
    initialize_onedim(onedim_mod);
+
+    pybind11::module_ testhelper_mod = mod.def_submodule("testhelper", "testhelper");


the helper module for testing python binding.

tigercosmos · 2023-12-26T16:07:33Z

.clang-tidy

@@ -16,6 +16,7 @@ Checks: >
    -cppcoreguidelines-non-private-member-variables-in-classes,
    -cppcoreguidelines-macro-usage,
    -fuchsia-overloaded-operator,-fuchsia-default-arguments*,
+    -cppcoreguidelines-pro-type-reinterpret-cast,


there are too many reinterpret_cast operations due to our low-level implementation manner, so disable the warning directly.

While it is not the devastating, I would like to ask you to try suppressing this in only the file having the new reinterpret_cast. I am a little bit worried by the relaxation on the free casting.

tigercosmos · 2023-12-26T17:45:23Z

@yungyuc the PR should be ready for review. There are CI errors for the debug build on macos/windows; however the release works for both platforms. I am still figuring out the root cause.

yungyuc

Thanks for the great patch. Please leave a global comment when you are ready for the next review. Please make sure all CI runs are clean for review.

Try to relax reinterpret_cast on the file-basis.
Keep the buffer_type alias in SimpleArrayInternalTypes mixin.
Review and add back all end marking.
Rename argument name data_type to type_string or something including string.
Python.h always need to be included in the first place.
Use your name for the new files of your major contribution.
Make a plan to remove testing code from production build.
Also assert the type name string.

yungyuc · 2024-01-03T13:41:23Z

.clang-tidy

@@ -16,6 +16,7 @@ Checks: >
    -cppcoreguidelines-non-private-member-variables-in-classes,
    -cppcoreguidelines-macro-usage,
    -fuchsia-overloaded-operator,-fuchsia-default-arguments*,
+    -cppcoreguidelines-pro-type-reinterpret-cast,


While it is not the devastating, I would like to ask you to try suppressing this in only the file having the new reinterpret_cast. I am a little bit worried by the relaxation on the free casting.

yungyuc · 2024-01-03T13:43:14Z

cpp/modmesh/buffer/SimpleArray.hpp

@@ -82,13 +84,17 @@ inline size_t buffer_offset(small_vector<size_t> const & stride, small_vector<si
 namespace detail
 {

+using shape_type = small_vector<size_t>;


yungyuc · 2024-01-03T13:44:04Z

cpp/modmesh/buffer/SimpleArray.hpp

@@ -82,13 +81,17 @@ inline size_t buffer_offset(small_vector<size_t> const & stride, small_vector<si
 namespace detail
 {

+using shape_type = small_vector<size_t>;
+using sshape_type = small_vector<ssize_t>;
+using buffer_type = ConcreteBuffer;


Please keep the buffer_type alias in SimpleArrayInternalTypes mixin. It's not impossible to use other buffer types in the (distant) future.

yungyuc · 2024-01-03T13:44:54Z

cpp/modmesh/buffer/SimpleArray.hpp

 template <typename T>
 struct SimpleArrayInternalTypes
 {
+    using shape_type = detail::shape_type;


Please do not reorder them. The diff looks ugly.

yungyuc · 2024-01-03T13:45:54Z

cpp/modmesh/buffer/SimpleArray.hpp

@@ -654,6 +659,86 @@ using is_simple_array = std::is_same<
 template <typename S>
 inline constexpr bool is_simple_array_v = is_simple_array<S>::value;

+using SimpleArrayBool = SimpleArray<bool>;


This is not beautiful, but I can live with it.

yungyuc · 2024-01-03T13:56:49Z

cpp/modmesh/testhelper/pymod/wrap_TestSimpleArrayHelper.cpp

+            .def("test_cast_to_arrayint32", []() -> modmesh::SimpleArrayInt32
+                 {  
+                    SimpleArrayInt32 arr(100);
+                    return arr; })


I doubt this passes clang-format. Let me know I am correct or not.

I don't like it as well, this is formatted by clang-format...

I think I can add // clang-format off for this line to make it pretty.

Oh. If clang-format does it, then let it be so. It's just a small part and perhaps clang-format will correct it in the future.

yungyuc · 2024-01-03T13:58:31Z

modmesh/core.py

@@ -73,6 +74,7 @@
    'Bezier3dFp64',
    'WorldFp32',
    'WorldFp64',
+    'testhelper',


I do not like to have testing code shipped with the production code. Please come up with a plan to make it disappear from production build before we can merge it. Once we conclude the plan it should be put in an issue.

I added a USE_PYTEST_HELPER_BINDING flag for this. will have explanation comments on the code.

yungyuc · 2024-01-03T14:04:13Z

tests/test_buffer.py

+    def test_SimpleArray_casting(self):
+        # first check the caster works if the argument is exact the same type
+        array_float64 = modmesh.SimpleArrayFloat64((2, 3, 4))
+        self.assertEqual(modmesh.testhelper.TestSimpleArrayHelper.


Multi-lines like this look ugly. Please beautify.

@yungyuc current 80 length limit of flake8 makes it difficult if any variable has a long name. 80 sounds too short for a mordern monitor, maybe consider to 120?

It brings more pain to violate PEP-8. We should stick with it.

yungyuc · 2024-01-03T14:05:28Z

tests/test_buffer.py

+        arrayplex_uint64 = modmesh.SimpleArray((2, 3, 4), dtype="uint64")
+        arrayplex_float64 = modmesh.SimpleArray((2, 3, 4), dtype="float64")
+
+        # check the type is the same with different data types


Please also assert the type name string. It will make the test code more descriptive.

Thanks for checking the type strings

yungyuc · 2024-01-03T14:08:51Z

tests/test_buffer.py

+            .TestSimpleArrayHelper \
+            .test_cast_to_arrayint32()  # SimpleArray32 from casting
+        self.assertTrue(type(array_int32) is type(array_int32_2))
+


The following does not seem to work (I am ok with it now):

# Obtain a plex from typed array array_int32 = modmesh.SimpleArrayInt32((2, 3, 4)) arrayplex_int32 = array_int32.plex # Obtain the typed array from a plex arrayplex_int32 = modmesh.SimpleArray((2, 3, 4), dtype='int32') array_int32 = arrayplex_int32.typed

How can we make the above work?

yes, we can do it. already added in the PR. will leave explanation comments later.

tigercosmos · 2024-01-27T04:50:35Z

Try to relax reinterpret_cast on the file-basis.
Keep the buffer_type alias in SimpleArrayInternalTypes mixin.
Review and add back all end marking.
Rename argument name data_type to type_string or something including string.
Python.h always need to be included in the first place.
Use your name for the new files of your major contribution.
Make a plan to remove testing code from production build.
Also assert the type name string.

tigercosmos · 2024-01-27T04:51:53Z

.github/workflows/devbuild.yml

@@ -159,7 +159,7 @@ jobs:
          VERBOSE=1 USE_CLANG_TIDY=OFF \
          BUILD_QT=OFF \
          CMAKE_BUILD_TYPE=${{ matrix.cmake_build_type }} \
-          CMAKE_ARGS="-DPYTHON_EXECUTABLE=$(which python3)"
+          CMAKE_ARGS="-DPYTHON_EXECUTABLE=$(which python3) -DUSE_PYTEST_HELPER_BINDING=ON"


DUSE_PYTEST_HELPER_BINDING flag is used to build the pytest helper.

most of the time we need this flag, since almost all builds are used for testing.

tigercosmos · 2024-01-27T04:52:28Z

.github/workflows/devbuild.yml

@@ -177,6 +177,17 @@ jobs:
          CMAKE_ARGS="-DPYTHON_EXECUTABLE=$(which python3)"
        make buildext VERBOSE=1

+    # build with pytest helper binding for testing


we first build the version without the helper, then build again with the helper, since later on we will run pytest.

tigercosmos · 2024-01-27T04:59:43Z

CMakeLists.txt

@@ -82,8 +89,10 @@ message(STATUS "DEBUG_SYMBOL: ${DEBUG_SYMBOL}")
 if(DEBUG_SYMBOL)
    if(MSVC)
        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /DEBUG")
+        add_compile_options(/O2 /Ob2)


unfortunately, pybind11 has bug for -O0 on mac/windows, customized caster will not work.
explicitly set -O2 for debug build.

What is the pybind11 bug with O0 and debug build?

Setting O2 may be a workaround but it may not be merged without describing the root cause clearly. A clear description must be given when the issue is found, or it will be overly difficult for reproduction later.

tigercosmos · 2024-01-27T05:01:37Z

cpp/modmesh/buffer/pymod/SimpleArrayCaster.hpp

+#include <modmesh/buffer/buffer.hpp>
+
+/*
+ * The purpose of including this header is to facilitate implicit casting of


add the comment to point out the purpose of the header.

Please follow the doxygen format /** * */

tigercosmos · 2024-01-27T05:02:27Z

cpp/modmesh/buffer/pymod/TypeBroadcast.hpp

@@ -1,7 +1,7 @@
 #pragma once

 /*
- * Copyright (c) 2019, Yung-Yu Chen <[email protected]>
+ * Copyright (c) 2024, An-Chi Liu <[email protected]>


the author of the file is also me, so just changed the name at this time as well.

tigercosmos · 2024-01-27T05:03:01Z

cpp/modmesh/buffer/pymod/wrap_SimpleArray.cpp

@@ -27,8 +27,9 @@
 */

 #include <modmesh/buffer/pymod/buffer_pymod.hpp> // Must be the first include.
-#include <modmesh/buffer/pymod/TypeBroadcast.hpp>
+


a space here can prevent clang-format to reorder the includes.

Good to know

tigercosmos · 2024-01-27T05:03:46Z

cpp/modmesh/buffer/pymod/wrap_SimpleArray.cpp

@@ -156,6 +157,7 @@ class MODMESH_PYTHON_WRAPPER_VISIBILITY WrapSimpleArray
            .def_property_readonly("has_ghost", &wrapped_type::has_ghost)
            .def_property("nghost", &wrapped_type::nghost, &wrapped_type::set_nghost)
            .def_property_readonly("nbody", &wrapped_type::nbody)
+            .def_property_readonly("plex", &get_arrayplex)


this is for

array_plex = array_typed.plex

Please use a lambda for plex.

tigercosmos · 2024-01-27T05:07:18Z

cpp/modmesh/testhelper/pymod/wrap_TestSimpleArrayHelper.cpp

+        (*this)
+            .def("test_cast_to_arrayint32", []() -> modmesh::SimpleArrayInt32
+                 {
+                     // clang-format off


clang-format is stupid here, added this to make the code pretty

tigercosmos · 2024-01-27T05:10:33Z

modmesh/core.py

@@ -73,6 +74,9 @@
    'Bezier3dFp64',
    'WorldFp32',
    'WorldFp64',
+
+    # TODO: have a way to enable this according to `USE_PYTEST_HELPER_BINDING`


didn't find out a good way to enable testhelper in python when USE_PYTEST_HELPER_BINDING is set on C++.
there are some complicated ways, such as doing file editing in CMake, but I think it isn't worth it.

tigercosmos · 2024-01-27T05:19:41Z

cpp/modmesh/buffer/pymod/wrap_SimpleArrayPlex.cpp

+                    { return wrapped_type(make_shape(shape), datatype); }),
+                pybind11::arg("shape"),
+                pybind11::arg("dtype"))
+            .def_property_readonly("typed", &get_typed_array)


this is for array_typed = array_plex.typed

This is great

tigercosmos · 2024-01-27T05:26:32Z

@yungyuc the PR is ready for review. all aforementioned issues are addressed. There are also comments for more details.

yungyuc · 2024-01-27T05:39:00Z

@yungyuc the PR is ready for review. all aforementioned issues are addressed. There are also comments for more details.

Thanks, @tigercosmos . You added a lot of changesets and it is a little bit confusing to me. If you have time, could you please reorganize them into fewer commits and rebase?

tigercosmos · 2024-01-27T06:07:01Z

@yungyuc the old commits are like spaghetti, not easy to rebase them into a few commits, so I just rebase them into a single commit. hope this will help you.

yungyuc

The updated patch is very good! We need to pin-point the pybind11 issue discovered.

IMPORTANT Clarify why the custom pybind11 caster does not work with O0 and debug build with mac/windows
Use doxygen format for file header comments.
Use initialization list for SimpleArrayPlex::m_has_instance_ownership.
Please use a lambda to wrap plex.
Make sure there is a build action without USE_PYTEST_HELPER_BINDING=ON

yungyuc · 2024-01-27T08:16:54Z

cpp/modmesh/buffer/pymod/SimpleArrayCaster.hpp

+#include <modmesh/buffer/buffer.hpp>
+
+/*
+ * The purpose of including this header is to facilitate implicit casting of


Please follow the doxygen format /** * */

yungyuc · 2024-01-27T08:43:55Z

cpp/modmesh/buffer/SimpleArray.cpp

+        return; // other does not have instance
+    }
+
+    m_has_instance_ownership = true;


Use initialization list

@yungyuc in this case is m_has_instance_ownership already in default member initializer as false, here 236 line changes to true if the previous if statement is true. Or, do you mean I should use initialization list to declare m_has_instance_ownership as false and not to use default member initializer?

Oh! I overlooked it. In this case we should not move this to initialization list. Please keep it here. Thanks for pointing it out.

yungyuc · 2024-01-27T08:48:17Z

cpp/modmesh/buffer/SimpleArray.hpp

@@ -40,7 +40,6 @@ typedef SSIZE_T ssize_t;

 namespace modmesh
 {
-


If change of blank lines is not by clang-format, please revert the unnecessary change.

yungyuc · 2024-01-27T11:50:57Z

cpp/modmesh/buffer/pymod/TypeBroadcast.hpp

@@ -1,7 +1,7 @@
 #pragma once

 /*
- * Copyright (c) 2019, Yung-Yu Chen <[email protected]>
+ * Copyright (c) 2024, An-Chi Liu <[email protected]>


yungyuc · 2024-01-27T11:51:18Z

cpp/modmesh/buffer/pymod/wrap_SimpleArray.cpp

@@ -27,8 +27,9 @@
 */

 #include <modmesh/buffer/pymod/buffer_pymod.hpp> // Must be the first include.
-#include <modmesh/buffer/pymod/TypeBroadcast.hpp>
+


Good to know

yungyuc · 2024-01-27T11:51:51Z

cpp/modmesh/buffer/pymod/wrap_SimpleArray.cpp

@@ -156,6 +157,7 @@ class MODMESH_PYTHON_WRAPPER_VISIBILITY WrapSimpleArray
            .def_property_readonly("has_ghost", &wrapped_type::has_ghost)
            .def_property("nghost", &wrapped_type::nghost, &wrapped_type::set_nghost)
            .def_property_readonly("nbody", &wrapped_type::nbody)
+            .def_property_readonly("plex", &get_arrayplex)


Please use a lambda for plex.

yungyuc · 2024-01-27T11:53:36Z

cpp/modmesh/buffer/pymod/wrap_SimpleArrayPlex.cpp

+                    { return wrapped_type(make_shape(shape), datatype); }),
+                pybind11::arg("shape"),
+                pybind11::arg("dtype"))
+            .def_property_readonly("typed", &get_typed_array)


This is great

yungyuc · 2024-01-27T11:58:46Z

tests/test_buffer.py

+        arrayplex_uint64 = modmesh.SimpleArray((2, 3, 4), dtype="uint64")
+        arrayplex_float64 = modmesh.SimpleArray((2, 3, 4), dtype="float64")
+
+        # check the type is the same with different data types


Thanks for checking the type strings

yungyuc · 2024-01-27T12:00:19Z

.github/workflows/devbuild.yml

Is there a build action without USE_PYTEST_HELPER_BINDING=ON?

yes, please check USE_PYTEST_HELPER_BINDING=OFF case in CI

generally, I made a build without the helper first, and then had a build with the helper for pytest.

yungyuc · 2024-01-27T12:03:51Z

CMakeLists.txt

@@ -82,8 +89,10 @@ message(STATUS "DEBUG_SYMBOL: ${DEBUG_SYMBOL}")
 if(DEBUG_SYMBOL)
    if(MSVC)
        set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /DEBUG")
+        add_compile_options(/O2 /Ob2)


What is the pybind11 bug with O0 and debug build?

Setting O2 may be a workaround but it may not be merged without describing the root cause clearly. A clear description must be given when the issue is found, or it will be overly difficult for reproduction later.

tigercosmos · 2024-02-15T15:44:08Z

here is a short report for early studying of the behavoiur of different -O levels

I inserted some stub by using asm like asm("brk #0x99"), to check if pybind11::detail::type_caster<modmesh::SimpleArrayInt32>::load()(in SimpleArrayCaster.hpp) is used, and use objdump -D -S -g -m _modmesh.cpython-311-darwin.so to dump the assembly code.

for example, we can see asm("brk #0x99") in -O0 build assembly code.

__ZN8pybind116detail11type_casterIN7modmesh11SimpleArrayIiEEvE4loadENS_6handleEb:
    345bbc:	ff 43 02 d1	sub	sp, sp, #144
    345bc0:	fd 7b 08 a9	stp	x29, x30, [sp, #128]
    345bc4:	fd 03 02 91	add	x29, sp, #128
    345bc8:	a1 03 1f f8	stur	x1, [x29, #-16]
    345bcc:	a0 83 1e f8	stur	x0, [x29, #-24]
    345bd0:	28 00 80 52	mov	w8, #1
    345bd4:	48 00 08 0a	and	w8, w2, w8
    345bd8:	a8 73 1e 38	sturb	w8, [x29, #-25]
    345bdc:	a0 83 5e f8	ldur	x0, [x29, #-24]
    345be0:	e0 0f 00 f9	str	x0, [sp, #24]
    345be4:	20 13 20 d4	brk	#0x99

-O0 build

asm("brk #0x99") is found in the following symbol (called S1 later), but it is never used in -g0 build code.

__ZN8pybind116detail11type_casterIN7modmesh11SimpleArrayIiEEvE4loadENS_6handleEb

pybind11::detail::type_caster<modmesh::SimpleArray<int>, void>::load(pybind11::handle, bool))

also, this S1 symbol doesn't exis in -O1, -O2, -O3 build

-O1, -O2, -O3 build

on the other hand, for -O1, -O2, -O3 build, asm("brk #0x99") are all found in the following symbol (called S2 later).

__ZZN8pybind1112cpp_function10initializeIRPFbRN7modmesh11SimpleArrayIiEEEbJS5_EJNS_4nameENS_9is_methodENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE_8__invokeESP_

void pybind11::cpp_function::initialize<bool (*&)(modmesh::SimpleArray<int>&), bool, modmesh::SimpleArray<int>&, pybind11::name, pybind11::is_method, pybind11::sibling>(bool (*&)(modmesh::SimpleArray<int>&), bool (*)(modmesh::SimpleArray<int>&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::'lambda'(pybind11::detail::function_call&)::__invoke(pybind11::detail::function_call&)

Summary

The S2 symbol is the entrance of casting SimpleArray, and for -O1, -O2, -O3 build, they all have asm("brk #0x99") in S2.

For -O0 build, the S2 symbol also exists, but it never calls S1 (where the asm("brk #0x99") exists), and that's why the casting failed.

Somehow the compiler just ignores the customized caster template in -O0 build, but the reason is still unknown.

@yungyuc I would like to have your advice for further investigation.

yungyuc · 2024-02-17T12:35:04Z

I am guessing -O1 -O2 -O3 inlines load while -O0 does not. Perhaps you can try forcing clang to inline it and see what may happen.

tigercosmos · 2024-02-17T14:26:23Z

@yungyuc very interesting, if I force inline for pybind11::detail::type_caster<modmesh::SimpleArray<int>, void>::load(pybind11::handle, bool)), the function is not complied. I cannot find #0x99 in the assembly in -O0 build.

tigercosmos · 2024-02-17T17:18:13Z

could be a related issue: pybind/pybind11#1153

tigercosmos · 2024-02-17T19:03:48Z

also a possible related issue: pybind/pybind11#2763

yungyuc · 2024-02-18T07:24:28Z

@yungyuc very interesting, if I force inline for pybind11::detail::type_caster<modmesh::SimpleArray<int>, void>::load(pybind11::handle, bool)), the function is not complied. I cannot find #0x99 in the assembly in -O0 build.

By "not compiled", do you mean it is totally optimized away? Please try tricks like https://programfan.github.io/blog/2015/04/27/prevent-gcc-optimize-away-code/ . We need the asm showing up to know what's going on.

tigercosmos · 2024-02-19T14:39:31Z

@yungyuc with using asm volatile trick:

inline __attribute__((always_inline)) bool load(pybind11::handle src, bool convert), I cannot find asm code in the compiled file
inline bool load(pybind11::handle src, bool convert), the asm is compiled, but the result is the same as the above report, and the function is never used.

the asm code is like this:

        inline bool load(pybind11::handle src, bool convert)                                                                                                         \
        {                                      
            asm volatile("mov w8, #0");
            asm volatile("" : : : "memory");
            asm("brk #0x99");

tigercosmos · 2024-02-19T15:17:33Z

aslo tried to use latest LLVM 17 toolchain to build ... the result is the same

yungyuc · 2024-02-20T00:21:12Z

Perhaps compiler explorer may help you find a way to insert the asm code.

tigercosmos · 2024-02-20T02:18:14Z

@yungyuc please let me clarify again. By using asm volatile trick, I confirmed the following result in both modmesh, and a simple code on compiler explorer.

inline __attribute__((always_inline)) foo(): by using this syntax for inline under -O0, the code needs to be called somewhere, otherwise the compiler will do nothing for this foo. Therefore, if I set inline __attribute__((always_inline)) for the customized caster load, we cannot find the inserted asm code due to the function is never used.
inline foo(): by using this syntax for inline -O0, seems the compiler will just ignore the inline keyword and still compile foo as a function. In modmesh case, the customized caster inline load is compiled as a function but the function is never called by others.

To sum up, the compiler decides not to use the pybind11 customized caster under -O0 (it decides to take the generic caster for unknown reasons), and it doesn't matter whether the load function is inline, force inline or not inline.

yungyuc · 2024-02-20T02:58:37Z

Can we say that, when we use LLVM, the custom caster is called only when it is inlined? If that statement is (or may be) true, it may be used to create a smaller test case for studying.

tigercosmos · 2024-03-05T12:58:42Z

IMPORTANT Clarify why the custom pybind11 caster does not work with O0 and debug build with mac/windows
Use doxygen format for file header comments.
~~Use initialization list for SimpleArrayPlex::m_has_instance_ownership.~~ (no need)
Please use a lambda to wrap plex.
Make sure there is a build action without USE_PYTEST_HELPER_BINDING=ON
Add a cmake switch to control optimization level for both debug and release build (DEBUG_BUILD_OPTIMIZATION_ONE)
In Github Actions turn O1 for the problematic compiler setup
Draft a Github issue to track the compiler issue and file it on GitHub (Bad assembly with SimpleArray caster #283)

tigercosmos · 2024-03-05T13:03:05Z

.github/workflows/devbuild.yml

@@ -423,11 +445,26 @@ jobs:
          Get-Command clang-tidy
          Get-Command flake8

-      - name: cmake ALL_BUILD
+      - name: cmake ALL_BUILD USE_PYTEST_HELPER_BINDING=OFF DEBUG_BUILD_OPTIMIZATION_ONE=ON


Here's an instance where USE_PYTEST_HELPER_BINDING is set to OFF. While there are additional scenarios, I'm highlighting one for reference.

DEBUG_BUILD_OPTIMIZATION_ONE is ON for both the debug build and the release build, but it only works for the debug build. I don't think we need to have a special condition branch to separate the two builds.

tigercosmos · 2024-03-05T13:04:02Z

.github/workflows/lint.yml

@@ -148,7 +148,7 @@ jobs:
    strategy:
      matrix:
        # https://github.com/actions/runner-images/blob/main/images/macos/macos-13-Readme.md
-        os: [macos-13]
+        os: [macos-13, macos-14]


add missing macos-14

tigercosmos · 2024-03-05T13:04:38Z

CMakeLists.txt

@@ -31,6 +31,17 @@ option(LINT_AS_ERRORS "clang-tidy warnings as errors" OFF)
 # - Address Sanitization: Protecting against address-related vulnerabilities. See https://clang.llvm.org/docs/AddressSanitizer.html
 option(USE_SANITIZER "use sanitizer (undefined, leak, address)" OFF)

+# FIXME: A workaround flag, check more on https://github.com/solvcon/modmesh/issues/283


the issue #283 is pointed out here.

tigercosmos · 2024-03-05T13:05:07Z

CMakeLists.txt

+            # We found that only /O2 can have the correct behavior, so we set /O2 here.
+            # /Ob2 is the default value of /O2, but we found that it need to explicitly set /Ob2 to have the correct behavior.
+            # See more: https://learn.microsoft.com/en-us/cpp/build/reference/o1-o2-minimize-size-maximize-speed
+            add_compile_options(/O2 /Ob2)


a little complicated here, but please check the comment in the code.

tigercosmos · 2024-03-05T13:15:37Z

IMPORTANT Clarify why the custom pybind11 caster does not work with O0 and debug build with mac/windows

Use doxygen format for file header comments.

~~Use initialization list for SimpleArrayPlex::m_has_instance_ownership.~~ (no need)

Please use a lambda to wrap plex.

Make sure there is a build action without USE_PYTEST_HELPER_BINDING=ON

Add a cmake switch to control optimization level for both debug and release build (DEBUG_BUILD_OPTIMIZATION_ONE)

In Github Actions turn O1 for the problematic compiler setup

Draft a Github issue to track the compiler issue and file it on GitHub (SimpleArray caster issue #283)

@yungyuc All comments have been addressed. Please have a look at your convenience.

yungyuc

LGTM. I'd like to merge it. I guess there are some rough edges to round, but we can do it against master.

yungyuc · 2024-03-05T14:28:29Z

CMakeLists.txt

@@ -31,6 +31,17 @@ option(LINT_AS_ERRORS "clang-tidy warnings as errors" OFF)
 # - Address Sanitization: Protecting against address-related vulnerabilities. See https://clang.llvm.org/docs/AddressSanitizer.html
 option(USE_SANITIZER "use sanitizer (undefined, leak, address)" OFF)

+# FIXME: A workaround flag, check more on https://github.com/solvcon/modmesh/issues/283
+# For macOS and Windows development, please turn on this flag to avoid the issue.
+option(DEBUG_BUILD_OPTIMIZATION_ONE "use -O1 for debug build. This is a workaround for issue#283." OFF)


This is great.

tigercosmos commented Dec 26, 2023

View reviewed changes

yungyuc added the enhancement New feature or request label Jan 3, 2024

yungyuc requested changes Jan 3, 2024

View reviewed changes

tigercosmos commented Jan 27, 2024

View reviewed changes

tigercosmos requested a review from yungyuc January 27, 2024 05:26

tigercosmos force-pushed the simple_array_caster20231227 branch from 8157111 to 451b4f3 Compare January 27, 2024 06:03

yungyuc requested changes Jan 27, 2024

View reviewed changes

add SimpleArray caster

6bb764e

tigercosmos force-pushed the simple_array_caster20231227 branch from 451b4f3 to 6bb764e Compare February 17, 2024 17:54

tigercosmos added 2 commits February 26, 2024 17:47

some minor fixes

7d93fb4

add DEBUG_BUILD_OPTIMIZATION_ONE

32e0f2c

tigercosmos mentioned this pull request Mar 5, 2024

Bad assembly with SimpleArray caster #283

Closed

tigercosmos commented Mar 5, 2024

View reviewed changes

yungyuc approved these changes Mar 5, 2024

View reviewed changes

yungyuc merged commit 2787aa2 into solvcon:master Mar 5, 2024
13 checks passed

tigercosmos deleted the simple_array_caster20231227 branch March 5, 2024 15:55

tigercosmos mentioned this pull request Apr 21, 2024

Single Python wrapper for SimpleArray #27

Closed

		@@ -40,7 +40,6 @@ typedef SSIZE_T ssize_t;

		namespace modmesh
		{

add SimpleArray caster #266

add SimpleArray caster #266

Conversation

tigercosmos commented Dec 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tigercosmos commented Dec 26, 2023

yungyuc left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tigercosmos commented Jan 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tigercosmos Jan 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tigercosmos commented Jan 27, 2024

yungyuc commented Jan 27, 2024

tigercosmos commented Jan 27, 2024

yungyuc left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tigercosmos commented Dec 26, 2023 •

edited

Loading

yungyuc left a comment •

edited

Loading

tigercosmos Jan 27, 2024 •

edited

Loading

yungyuc left a comment •

edited

Loading