advice as to why the same cppyy code can not be run multiple times in succession in the same run #269
Comments
|
more to the about error - as i step through using a debugger, on the cppyy call in Test002, i get the following from the debugger: "Test001 so i then copied the Python_001.py into Python_002.py and used that in Test002. I get the same error. has anyone any ideas - any advice - any help - please |
|
more to the error - if it is an error if i start up python from the command line, ensuring the virtual env is activative. so the issues seems to be running python whilst embedded is the problem, using pybind11 as the glue between the two. But pybind11 runs fine for multiple test uses of embedded python in one execution. confused. pybind11 states its closes down python once its guard goes out of scope. i even tried changing the hello c++ functions in both python_001 & 002 to 'hello1()' and 'hello2()' - same error as before. really confused. any advice or help please. |
|
I don't understand what you are trying to do here? cppyy is a bindings generator between Python and C++. As such, it has state in both languages. With the Note that Python itself never offloads dynamic libraries, as the dynamic linker will put a reference on any library into which a symbol was resolved. Python has no control over that. As such, you might get this to work for small, self-contained cases, but not if you intend any connection to some large library such as Qt. Again, I don't know what the goal is here, but one thing that might work is to import cppyy separately such that it isn't offloaded by the |
|
thanks for the reply. this is part of my dissertation for my MSc in Cyber Security. i have developed a visual plug-and-play environment that allows a user to both develop their own code and utilise others solutions (encapsulating their entire command line application). its a bit like metasploit but far more visual, far more extendable and far more experimentable in terms of playing around with captured and created data. i am embedding python into my QT / C++ solution. i would like to call new c++ code from within this environment, without going through all the issues of compiling and then bring an '.so' into my system explicitly. the python part of this equation is easy. i understand the issues of scope and am banking on that. the python works well. starting up, stopping and restarting, all under user control. each time python is called with a job of work, there will be no other python running - GIL issues - but this may change in the future with 3.13. i would like to use cppyy within my embedded python environment to do the above. either take some existing c / c++ code and run it with a set of wrappers, or allow the user to write their own. so i simulated this isolated environment. first with pybind11 and then just python / c api to utilise cppyy. so i tried something new. now the second run in a new function in my code (first function was 'Test001', this second function 'Test002' - both in c++ in my code). Test001 runs as expected but Test02 fails. i stripped Test002 down to just importing cppyy, leaving Test001 as it was (cppdef a c++ function), with no other code, and it still fails so i went further. now both Test001 and Test002 (totally independent functions, each starting and stopping their own embedded python sessions as before, running one after each other, not in parallel), each setting up their own sys.paths in python, each only containing "import cppyy". fails when it gets the Test002. i am totally stumped as to what to do to fix it. i have tested my code multiple times and am happy with all the non-cppyy parts, but i must be doing something wrong somewhere. i just can not see it. // import cppyy } any advice would be very much appreciated please. |
|
As said above, the Python side isn't the problem, so that's not where the fix can be, it's on the C++ side. As such, I really don't think any of the above approaches have a chance of working. I tried with cppyy kept alive globally (by running pybind11 from cppyy through C++), but found that that indeed doesn't work with Two other options: p3.12 subinterpreters (with the main intepreter keeping Note that Cling isn't setup to be transactional (that's why Also, if the code isn't meant to be interactively used (ie. if it's always correct rather than typed on some in-application prompt), you can just embed OrcJIT. |
|
again thanks for the reply. you are right about performing c++ scripting within Python. it just that your solution held out so many possiblities for this approach and so many more. plus there was little development need to move data back and forward when from python to my solution. in my day research job i use matlab, which is both very expensive and is not interactive where c++ comes into play. my client was quite keen on my approach and the use of cppyy + python all within a dynamic visual + controlling framework. the university was also. i will have to rethink some of my concepts. you mentioned using cling (root??). is there any chance of pointing me to a simple guide to both building cling and embedding it (initially for linux and later for windows)?? many thanks for all your advice and carry on with the excellant cppyy project. |
|
Yes, you can use Cling w/o cppyy to get the same features, but yes, the marshaling is then yours to do (of course that's true now as well from Python, but a bit harder since Python's reflection is easier to use). Cling is here: https://github.com/root-project/cling I'm still not 100% sure on the use case and in particular not on how calls happen, but one way to start experimenting today, is by taking function pointers in your own code, with The following is a bit self-referencing, but you can execute it in C++ as well using With just Cling Or Clang-Repl, you can also spin up completely fresh interpreters, wiping away old state (Cling will be a bit slower). Neither truly allows resource tracking (AFAIK), like eg. OrcJIT would, but since you can have multiple interpreters active at the same time, the only limitation is the startup time, which if you don't use the PCH and include C++ headers explicitly instead, isn't all that much. |
|
again, thanks for all the advice. i had based all my assumptions on using cppyy as i would in any python session life. question 1. question 2. many thanks for any advice that can be given. |
|
For the first, using cppyy is just a convenience (if not hack), to allow you in essence to Then no, there really isn't any way to shutdown/restart For the second, yes, that's why I suggested sticking to You can share the |
|
again, thanks for the reply - these really do help my use-case for wanting to use c++& python as a scripting features in my code, alone with many other future potential language encapsulations, is to allow a user to experiment whilst in my environment and do this visually, to capture results, to save results, to feed results from one solution into another. Python was easy embed. c++ can be easy but the process can be quite brittle and not always for the faint-hearted, plus the need for many specialist support functions to take data backup and forward. the user may create a solution in a piece meal approach, stop, start, debug, repeat, hence my questions about playing around with the session element of cppyy (a similar idea to that of the life-time in python sessions). i am not particularly worried about the time element yet, but its always nice to have a good functioning flow. core to my dissertation is not only to allow a user to create their own solutions from raw source code, but to allow them to play around with other's code and to allow them to encapsulate entire command line applications from others. so hence my fixation on python and c++ - others will come later. i have been playing around with a number of repl solutions about allow the illusion of an interpreter, but this still does not address all of the support issues. cling seems to have a rich set of features, which you use in cppyy, for this. so it would seem cling or even that clangrepl, may be a direction, if only i can develop a simple demonstrator in a standalone manor (embedding that interpreter in my code). so here is my cling build recipe to date: after spending 2 days trying to get cling to embed, i have in utter frustation deleted my play area with my cmake + c++ files, swearing i would never touch cling again - but then i calmed down and regreted my actions. at its simpliest, the embedding to the cling interpreter would like this in c++: int main(int argc, char** argv) interp.declare("int p=0;"); what does 'LLVMRESDIR' point to?? i got something to compile, but fell over at the 'interp' statement. many thanks for at least allowing me to ask all this and if you can help, it would be very much appreciated and save me from tearing any more of my hair out. |
|
Pinging @aaronj0 here, in case he has a cmake fragment to embed clang-repl? Yes, building LLVM can take a significant amount of time; recommend to always to a full parallel build by allowing 2x the jobs as you have cores available. As for the LLVM resources dir, it's the directory to the Clang include files for CUDA, AVX, etc. Search for eg. Beyond that, interactive C++ is significantly different in use that interactive Python. Two important considerations here are the easy way of redefining entities in the latter (or even modifying them); but also the fact that preceding C++ code affects code that comes later and hence combining two pieces of C++ code interactively isn't guaranteed to be deterministic. |
|
thanks for the reply i understand, or at least some undertanding, of the issues with respect to the interactivity lose - its never going to be anything like that of python or Visual studio or QT in terms of auto-complete - there is the potential for many stop / restarts in the process - but it will allow a user to taken code from what-ever source they choose or code themself to fit into the flow of the visual concept. a session recipe may be made of many languages, and in the visual flow - one node feeding another node, in turn feeding another node, they do not have to fit as in a normal programming flow, just in the data they expect (flow based programming (FBP) is a close example) - an example, so i could capture some raw network traffic, save it and then feed that captured traffic into another node (a piece of code or another encapsulated command-line program) for futher processing - so the flow could be a functional flow, as in most programs we write, or a data flow, with different progams extracting some further meaning that another connected node can further perform processing - so i design a recipe to a soution from many different sub-solutions to achieve some desired objective the c++ element in an intrepreter is just another node. it could be a teaching aid for students wan to learn some concept - the number of python and c++ code snippets given to me during my MSc i wished i could have quickly tried out - in my clients world we extract meaning from multiple souces of data, which leads to huge monothlic code chunks, which if could be broken down into small parts but still bolten together to achieve some desired objective, would save massive amounts of time. again, given the cling recipe i posted above, how i could create a simple standalone demo, from which i could then make new solutions in terms of the cmake - i had hoped cppyy would fill this role through using python, but that seems not possible - i do have other potential approachs but nothing with the support functions for data transfer in and out of using cling if you can provide any advice or know of anyone how could advise, i would very much appreciate it - many thanks |
|
@vgvassilev do you have an example code of the most basic Cling setup? |
|
I think the demo @SimonDev666 mentioned should be the easiest start. We have built tutorials for embedding clang-repl https://github.com/compiler-research/pldi-tutorials-2023 maybe that’d help? |
|
Hello vgvassilev, i will have a look at the web link you sent many thanks |
|
Hello vgvassilev, so here is my cling build recipe to date: at its simpliest, the embedding to the cling interpreter would like this in c++: int main(int argc, char** argv) interp.declare("int p=0;"); i will then later embed this code approach into a shared object for loading on demand within my code. i was hoping, given the above build recipe and code, you might be able to construction me a simple cmake file, if that would at all be possible please. i had a look at your provided link - looks very good, just way beyond me at the moment - i am suffering from using qt creator so many of the cmake complex features just look like another language and plus i am more of a windows coder than linux. many thanks for any help that you could provide. |
|
Isn't that what you need: https://github.com/root-project/cling/tree/master/tools/demo You can simplify the example to your needs. |
|
Hello vgvassilev, I've tried many combinations of that approach and still nothing but errors. as per that web page you pointed me too, i tried the following: cmake -Dcling_DIR=/home/simon/Experimental/cling-build/lib/cmake/clang /home/simon/Experimental/cling/tools/demo there is no cling-build/lib/cmake/clang directory created after the build process. here i get the following errors: Could not find a package configuration file provided by "cling" with any of Add the installation prefix of "cling" to CMAKE_PREFIX_PATH or set i checked and there is no files for "cling*.cmake" anywhere in Experiemental (in any of the 3 directories within it - cling, cling-build, llvm-project) can you advise me as to what i am doing incorrect please |
Hello,
i have embedded the python interpreter within my c++ program / app (QT) using pybind11.
i setup a simple python script file which internally sets up and then calls a simple c++ function using cppyy.
Runs without a problem.
now copy the same code, keeping to scope rules, so it does the same thing in an excution twice, then it fails on the second call that worked on the first call.
here is the error:
"Test001
Hello, World!
['/home/simon/Desktop/Code/VirtEnv/lib/python3.12/site-packages/', '/home/simon/Code/QT/Test_pybind11/Test_pybind11_cppyy_001/', '', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/usr/local/lib/python3.12/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.12/dist-packages']
here
done
free(): invalid size
Test002
*** Break *** abort
#0 0x00007f08d6d10893 in __GI___wait4 (pid=10578, stat_loc=stat_loc
entry=0x7ffc9b684228, options=options
entry=0, usage=usage
entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#1 0x00007f08d6d109e7 in _GI___waitpid (pid=, stat_loc=stat_loc.........
#51 0x00007f08d737f94f in PyObject_CallFunction () from /lib/x86_64-linux-gnu/libpython3.12.so.1.0
#52 0x00007f08d74cf10a in PyImport_Import () from /lib/x86_64-linux-gnu/libpython3.12.so.1.0
#53 0x00007f08d74cf31f in PyImport_ImportModule () from /lib/x86_64-linux-gnu/libpython3.12.so.1.0
#54 0x000055dee4fe9934 in pybind11::module::import (name=0x55dee501113e "Python_001") at /usr/include/pybind11/pybind11.h:1279
#55 0x000055dee4fe85f6 in main (argc=1, argv=0x7ffc9b688888) at /home/simon/Code/QT/Test_pybind11/Test_pybind11_cppyy_001/main.cpp:65"
here is the main c++ program / app (this is in my code and is compiled within QT creator):
" std::cout << "Test001\n";
{
//
py::scoped_interpreter guard{};
//
QString qPythonDir("/home/simon/Code/QT/Test_pybind11/Test_pybind11_cppyy_001/");
//
QString qPythonCommand("");
auto locals = py::dict();
//
qPythonCommand += "import sys\n";
qPythonCommand += "sys.path.insert(0, '";
qPythonCommand += qPythonDir.toStdString().c_str();
qPythonCommand += "')\n";
qPythonCommand += "sys.path.insert(0, '/home/simon/Desktop/Code/VirtEnv/lib/python3.12/site-packages/')\n";
qPythonCommand += "print(sys.path)\n";
py::exec(qPythonCommand.toStdString().c_str(), py::globals(), locals);
//
py::module module = py::module_::import("Python_001");
//
module.attr("runCPP")();
}
std::cout << "Test002\n";
{
//
py::scoped_interpreter guard{};
//
QString qPythonDir("/home/simon/Code/QT/Test_pybind11/Test_pybind11_cppyy_001/");
//
QString qPythonCommand("");
auto locals = py::dict();
//
qPythonCommand += "import sys\n";
qPythonCommand += "sys.path.insert(0, '";
qPythonCommand += qPythonDir.toStdString().c_str();
qPythonCommand += "')\n";
qPythonCommand += "sys.path.insert(0, '/home/simon/Desktop/Code/VirtEnv/lib/python3.12/site-packages/')\n";
qPythonCommand += "print(sys.path)\n";
py::exec(qPythonCommand.toStdString().c_str(), py::globals(), locals);
//
py::module module = py::module_::import("Python_001");
//
module.attr("runCPP")();
}
"
here is the python script (Python_001.py) called by pybind11:
"import cppyy
cppyy.cppdef(r"""
#include
void hello() {
std::cout << "Hello, World!" << std::endl;
}
""")
def runCPP():
print("here")
cppyy.gbl.hello()
print("done")
"
my cmake file looks like:
"cmake_minimum_required(VERSION 3.16)
project(Test_pybind11_cppyy_001 LANGUAGES CXX)
set(CMAKE_AUTOUIC ON)
set(CMAKE_AUTOMOC ON)
set(CMAKE_AUTORCC ON)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
find_package(QT NAMES Qt6 Qt5 REQUIRED COMPONENTS Core)
find_package(Qt${QT_VERSION_MAJOR} REQUIRED COMPONENTS Core)
find_package(pybind11 REQUIRED)
add_executable(Test_pybind11_cppyy_001
main.cpp
../../EMB.h
)
#target_link_libraries(Test_pybind11_cppyy_001 Qt${QT_VERSION_MAJOR}::Core)
target_link_libraries(Test_pybind11_cppyy_001 Qt${QT_VERSION_MAJOR}::Core pybind11::embed)
include(GNUInstallDirs)
install(TARGETS Test_pybind11_cppyy_001
LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}"
can anyone tell me what i am doing incorrect please.
this is all running on the latest Kali Linux, using the latest QT, with all python libaries uptodate as possible.
The text was updated successfully, but these errors were encountered: