Segmentation fault when running `test_get_runtime_model` test #18388

p-wysocki · 2023-07-05T13:21:24Z

Context

One of our tests results in a segfault error when ran by itself. This is probably related to Model object destruction. This task regards finding the error and fixing it. This task allows for quick familiarization with the internals of OpenVINO Python API.

Reproduction

The issue is reproducible with the following command:

python3 -m pytest tests/test_runtime/test_compiled_model.py::test_get_runtime_model -sv

which needs to be ran from openvino/src/bindings/python.

Resources

Contact points

@jiwaszki
@akuporos
@p-wysocki

Ticket: 113294

The text was updated successfully, but these errors were encountered:

abhamedewar · 2023-07-19T03:34:32Z

Hi,
I want to start contributing to OpenVINO.
Could you please assign me this issue?
Do let me know if I need to go through something else as well.
Thank you

mlukasze · 2023-07-19T05:15:23Z

hey @abhamedewar
welcome and thank you :)

at first please try to install openvino and reproduce the problem following the instruction.

abhamedewar · 2023-07-19T18:42:50Z

Thank you for assigning me the issue.
I will start working on it!!

abhamedewar · 2023-07-19T21:25:41Z

Hi @mlukasze,

I followed the installation instructions from the following link: Build OpenVINO™ Runtime for Linux systems.

Ran the command provided for reproduction, but the test passed. I do not get any error.
Refer the screenshot below:

jiwaszki · 2023-07-20T08:58:21Z

Hi @abhamedewar , I am very happy to see that issue is being looked at!

As far as I remember the problem was only reproducible under specific circumstances (like running all tests together). @akuporos as a reporter, do you remember a case?

akuporos · 2023-07-20T12:30:39Z

Hi @abhamedewar,

I caught the segfault with next cmake options:
cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DENABLE_WHEEL=OFF -DENABLE_OV_PADDLE_FRONTEND=OFF -DENABLE_TESTS=OFF -DENABLE_OV_TF_FRONTEND=OFF -DENABLE_MULTI=OFF -DENABLE_HETERO=OFF -DENABLE_FUNCTIONAL_TESTS=OFF -DENABLE_SAMPLES=OFF -DCMAKE_BUILD_TYPE=Release -DENABLE_INTEL_GPU=OFF -DENABLE_SAMPLES=ON ..
python version is 3.8.10

abhamedewar · 2023-07-20T14:23:18Z

Hi @jiwaszki,
I ran all the tests that passed too!

@akuporos Sure, I will try with python 3.8.10 and cmake command

abhamedewar · 2023-07-20T18:28:12Z

Hi @akuporos,

To build the OpenVINO Runtime Python API we need to enable the -DENABLE_WHEEL=ON option in the CMake step, this will generate a wheel file in the build folder. (As per the installation instructions)

But in the command that you have provided above the -DENABLE_WHEEL flag is OFF which will not generate openvino wheel file for python.

To run the file test_compiled_model.py we need to install openvino using the wheel file right? Am I missing anything here??

akuporos · 2023-07-21T11:23:08Z

Hi @abhamedewar,

It's possible to run python applications without building&installation of the wheel.
After building the project using mentioned CMake command, please, just set next variables:

export PYTHONPATH=PYTHONPATH:<openvino_repo>/bin/intel64/Release/python/
export LD_LIBRARY_PATH=LD_LIBRARY_PATH:<openvino_repo>/bin/intel64/Release

p-wysocki · 2023-08-23T10:39:45Z

I'm returning the task to being open due to current assignee's inactivity.

@jiwaszki @akuporos can we make a decision whether this error is even reproducible? Maybe we should close the issue.

PiwkoWazzap · 2023-09-27T12:08:10Z

Hi,
I want to start contributing to OpenVINO. I'd be glad to try reproducing (and hopefully fix) the error if you assign the issue to me, thank you.

@p-wysocki

p-wysocki · 2023-10-02T07:58:06Z

Hello @PiwkoWazzap,

Sorry for the late reply, I have been out of office. I assigned you, thanks for taking a look!

p-wysocki · 2023-10-17T14:14:00Z

Hi @PiwkoWazzap, are you still working on it? I'm updating the tasks' statuses.

@akuporos is this task still valid?

sanbuphy · 2023-12-07T15:17:00Z

Hi , Can i try this?

p-wysocki · 2023-12-08T08:46:43Z

Hi @sanbuphy, of course! Please let us know if you have any questions. If you wish you can use our work in progress roadmap for new contributors - it could help you get started. If you do, please let us know if it was useful and how it can be improved.

p-wysocki · 2023-12-15T09:55:45Z

Hello @sanbuphy, do you have any questions or need any help?

sanbuphy · 2023-12-15T10:16:43Z

Hello @sanbuphy, do you have any questions or need any help?

HI , I'm sorry, but I will have to wait until the weekend to start contributing to projects. I usually finish work very late during weekdays(in china you know,I usually don't have the energy to continue my research) . I will take a look at it over the weekend.

p-wysocki · 2023-12-15T10:45:58Z

Sure, no worries, just checking in. :) There's no rush, take your time - I'm just periodically checking our board for activity in the tasks, sometimes they get abandoned.

p-wysocki · 2024-01-03T15:11:37Z

Hi @sanbuphy! Please let us know if we can help with the task. :)

I am happy to announce that we have created a channel dedicated to Good First Issues support on our Intel DevHub Discord server! Join it to receive support, engage in discussions, ask questions and talk to OpenVINO developers.

sanbuphy · 2024-01-04T05:57:46Z

Hi @sanbuphy! Please let us know if we can help with the task. :)

I am happy to announce that we have created a channel dedicated to Good First Issues support on our Intel DevHub Discord server! Join it to receive support, engage in discussions, ask questions and talk to OpenVINO developers.

I'm not sure if it's the error:

p-wysocki · 2024-01-04T13:18:44Z

Did you follow https://github.com/openvinotoolkit/openvino/blob/master/src/bindings/python/docs/test_examples.md?

Can you please share your PYTHONPATH environment variable?

FredBill1 · 2024-01-28T01:53:49Z

Hi, I was testing the issue on windows 11, the std::shared_ptr<ov::Model> holding the variable runtime_model created from runtime_model = compiled_model.get_runtime_model() becomes Inaccessible immediately after the openvino_intel_cpu_plugin.dll is unloaded by FreeLibrary in

openvino/src/common/util/src/os/win/win_shared_object_loader.cpp

Line 161 in d553d4c

FreeLibrary(reinterpret_cast<HMODULE>(shared_object));

which happens during the destruction of compiled_model.

If the device is specified as "CPU" or "GPU", the CompiledModel class will hold a shared_ptr to the loaded device plugin in

openvino/src/inference/include/openvino/runtime/compiled_model.hpp

Line 38 in 635bfd8

std::shared_ptr<void> _so;

If the device is None, the auto plugin will be used, the CompiledModel class will hold a reference to the ov::Core through a chain of std::shared_ptrs, and the destruction of ov::Core unloads the loaded plugins.

openvino/src/inference/include/openvino/runtime/compiled_model.hpp

Line 37 in 635bfd8

std::shared_ptr<ov::ICompiledModel> _impl;

openvino/src/plugins/auto/src/auto_compiled_model.hpp

Line 14 in e40e5b8

class AutoCompiledModel : public CompiledModel {

openvino/src/plugins/auto/src/compiled_model.hpp

Line 29 in e40e5b8

ScheduleContext::Ptr m_context;

openvino/src/plugins/auto/src/common.hpp

Line 202 in e40e5b8

std::shared_ptr<ov::ICore> m_ov_core;

openvino/src/inference/src/dev/core_impl.hpp

Line 58 in 66d792d

mutable std::map<std::string, ov::Plugin> plugins;

A simple walkaround would be to keep the dynamic library alive by making the compiled_model object or a global Core object out lives the runtime_model object.

import gc

def test_get_runtime_model(device):
    compiled_model = generate_relu_compiled_model(device)
    runtime_model = compiled_model.get_runtime_model()
    assert isinstance(runtime_model, Model)

    del runtime_model
    gc.collect()

But I'm not sure if this counts as a proper fix to the issue

p-wysocki · 2024-01-31T10:06:31Z

Hi @FredBill1, thanks for taking a look!

cc @jiwaszki

FredBill1 · 2024-01-31T14:12:30Z

Hi, thanks for assigning me the issue. I found that the previous comment had a mistake and I've updated it.

I also wrote a piece of C++ code that can reproduce the issue, which makes it a bit easier to debug.

#include <iostream>
#include <memory>

#include "openvino/core/model.hpp"
#include "openvino/opsets/opset8.hpp"
#include "openvino/runtime/core.hpp"

std::shared_ptr<ov::Model> create_simple_model() {
    auto data = std::make_shared<ov::opset8::Parameter>(ov::element::f32, ov::Shape{3, 1, 2});
    auto mul_constant = ov::opset8::Constant::create(ov::element::f32, ov::Shape{1}, {1.5});
    auto mul = std::make_shared<ov::opset8::Multiply>(data, mul_constant);
    auto res = std::make_shared<ov::opset8::Result>(mul);
    return std::make_shared<ov::Model>(ov::ResultVector{res}, ov::ParameterVector{data});
}

int main(int argc, char* argv[]) {
    auto model = create_simple_model();
    auto core = std::make_shared<ov::Core>();
    auto compiled_model = std::make_shared<ov::CompiledModel>(core->compile_model(model, "CPU"));
    auto runtime_model = compiled_model->get_runtime_model();
    std::cout << compiled_model << ' ' << runtime_model << std::endl;
    core.reset();
    compiled_model.reset();  // FreeLibrary happens here
    runtime_model.reset();   // segmentation fault happens here
    std::cout << compiled_model << ' ' << runtime_model << std::endl;
}

…ence to the CompiledModel

FredBill1 · 2024-02-16T19:27:22Z

Hi, I've created 2 PRs:

fix #18388 by manually del runtime_model in python code #22874 (simple walkaround described above)
fix #18388 by recreate ov::Model using the main runtime and assign the shared objects to the recreated object #22900

…ime and assign the shared objects to the recreated object

@ilya-lavrenov

…the shared objects to the recreated object (#22900) ### Details: The detailed cause of the issue is described in #18388 (comment). Details of this PR are in #22874 (comment) @ilya-lavrenov. I have tested the changes against the following code and it worked as expected: ```cpp #include <iostream> #include <memory> #include "openvino/core/model.hpp" #include "openvino/opsets/opset8.hpp" #include "openvino/runtime/core.hpp" std::shared_ptr<ov::Model> create_simple_model() { auto data = std::make_shared<ov::opset8::Parameter>(ov::element::f32, ov::Shape{3, 1, 2}); auto mul_constant = ov::opset8::Constant::create(ov::element::f32, ov::Shape{1}, {1.5}); auto mul = std::make_shared<ov::opset8::Multiply>(data, mul_constant); auto res = std::make_shared<ov::opset8::Result>(mul); return std::make_shared<ov::Model>(ov::ResultVector{res}, ov::ParameterVector{data}); } int main(int argc, char* argv[]) { const auto devices = {"CPU", "GPU", "AUTO", "MULTI:GPU,CPU", "BATCH:GPU", "HETERO:CPU,GPU"}; for (auto&& device : devices) { std::cout << "Device: " << device << std::endl; auto model = create_simple_model(); auto core = std::make_shared<ov::Core>(); auto compiled_model = std::make_shared<ov::CompiledModel>(core->compile_model(model, device)); auto runtime_model = compiled_model->get_runtime_model(); std::cout << compiled_model << ' ' << runtime_model << std::endl; core.reset(); compiled_model.reset(); // FreeLibrary happens here runtime_model.reset(); // segmentation fault happens here std::cout << compiled_model << ' ' << runtime_model << std::endl; } } ``` ``` Device: CPU 000002327F6F63E0 000002327EF0A950 0000000000000000 0000000000000000 Device: GPU 0000023213B85510 000002327F6F39B0 0000000000000000 0000000000000000 Device: AUTO 000002327F234920 000002327F14B720 0000000000000000 0000000000000000 Device: MULTI:GPU,CPU 0000023212DFDEF0 000002327FB153B0 0000000000000000 0000000000000000 Device: BATCH:GPU 75 warnings generated. 0000023212DE6C60 000002327F14D340 0000000000000000 0000000000000000 Device: HETERO:CPU,GPU 0000023212DAFC80 000002327F14BA40 0000000000000000 0000000000000000 ``` ### Tickets: - Closes #18388 --------- Co-authored-by: Roman Kazantsev <[email protected]> Co-authored-by: Ilya Lavrenov <[email protected]>

p-wysocki added good first issue Good for newcomers no_stale Do not mark as stale labels Jul 5, 2023

p-wysocki mentioned this issue Jul 3, 2023

Contributions Welcome #17502

Open

mlukasze assigned abhamedewar Jul 19, 2023

p-wysocki unassigned abhamedewar Aug 23, 2023

siddhant-0707 mentioned this issue Sep 30, 2023

Segmentation fault when running test_get_runtime_model test #20163

Closed

p-wysocki assigned PiwkoWazzap Oct 2, 2023

p-wysocki added this to Good first issues Oct 3, 2023

p-wysocki moved this to Contrubutors needed in Good first issues Oct 3, 2023

p-wysocki moved this from Contrubutors needed to Assigned in Good first issues Oct 3, 2023

p-wysocki unassigned PiwkoWazzap Nov 13, 2023

p-wysocki moved this from Assigned to Contributors Needed in Good first issues Nov 13, 2023

p-wysocki assigned sanbuphy Dec 8, 2023

p-wysocki moved this from Contributors Needed to Assigned in Good first issues Dec 8, 2023

p-wysocki unassigned sanbuphy Jan 19, 2024

p-wysocki moved this from Assigned to Contributors Needed in Good first issues Jan 19, 2024

p-wysocki assigned FredBill1 Jan 31, 2024

p-wysocki moved this from Contributors Needed to Assigned in Good first issues Jan 31, 2024

FredBill1 added a commit to FredBill1/openvino that referenced this issue Feb 15, 2024

fix openvinotoolkit#18388 by using a custom holder that keeps a refer…

a78a1c0

…ence to the CompiledModel

FredBill1 mentioned this issue Feb 15, 2024

fix #18388 by using a custom holder that keeps a reference to the CompiledModel #22872

Closed

FredBill1 added a commit to FredBill1/openvino that referenced this issue Feb 15, 2024

fix openvinotoolkit#18388 by manually del runtime_model in python code

084f6fc

FredBill1 mentioned this issue Feb 15, 2024

fix #18388 by manually del runtime_model in python code #22874

Closed

FredBill1 added a commit to FredBill1/openvino that referenced this issue Feb 17, 2024

fix openvinotoolkit#18388 by recreate ov::Model using the main runt…

8e32a92

…ime and assign the shared objects to the recreated object

FredBill1 mentioned this issue Feb 17, 2024

fix #18388 by recreate ov::Model using the main runtime and assign the shared objects to the recreated object #22900

Merged

mlukasze added this to the 2024.1 milestone Feb 23, 2024

ilya-lavrenov closed this as completed in #22900 Feb 23, 2024

github-project-automation bot moved this from Assigned to Closed in Good first issues Feb 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segmentation fault when running `test_get_runtime_model` test #18388

Segmentation fault when running `test_get_runtime_model` test #18388

p-wysocki commented Jul 5, 2023

abhamedewar commented Jul 19, 2023

mlukasze commented Jul 19, 2023 •

edited

Loading

abhamedewar commented Jul 19, 2023

abhamedewar commented Jul 19, 2023

jiwaszki commented Jul 20, 2023

akuporos commented Jul 20, 2023

abhamedewar commented Jul 20, 2023

abhamedewar commented Jul 20, 2023

akuporos commented Jul 21, 2023

p-wysocki commented Aug 23, 2023

PiwkoWazzap commented Sep 27, 2023

p-wysocki commented Oct 2, 2023

p-wysocki commented Oct 17, 2023

sanbuphy commented Dec 7, 2023

p-wysocki commented Dec 8, 2023

p-wysocki commented Dec 15, 2023

sanbuphy commented Dec 15, 2023 •

edited

Loading

p-wysocki commented Dec 15, 2023 •

edited

Loading

p-wysocki commented Jan 3, 2024

sanbuphy commented Jan 4, 2024

p-wysocki commented Jan 4, 2024

FredBill1 commented Jan 28, 2024 •

edited

Loading

p-wysocki commented Jan 31, 2024

FredBill1 commented Jan 31, 2024 •

edited

Loading

FredBill1 commented Feb 16, 2024 •

edited

Loading

Segmentation fault when running test_get_runtime_model test #18388

Segmentation fault when running test_get_runtime_model test #18388

Comments

p-wysocki commented Jul 5, 2023

Context

Reproduction

Resources

Contact points

abhamedewar commented Jul 19, 2023

mlukasze commented Jul 19, 2023 • edited Loading

abhamedewar commented Jul 19, 2023

abhamedewar commented Jul 19, 2023

jiwaszki commented Jul 20, 2023

akuporos commented Jul 20, 2023

abhamedewar commented Jul 20, 2023

abhamedewar commented Jul 20, 2023

akuporos commented Jul 21, 2023

p-wysocki commented Aug 23, 2023

PiwkoWazzap commented Sep 27, 2023

p-wysocki commented Oct 2, 2023

p-wysocki commented Oct 17, 2023

sanbuphy commented Dec 7, 2023

p-wysocki commented Dec 8, 2023

p-wysocki commented Dec 15, 2023

sanbuphy commented Dec 15, 2023 • edited Loading

p-wysocki commented Dec 15, 2023 • edited Loading

p-wysocki commented Jan 3, 2024

sanbuphy commented Jan 4, 2024

p-wysocki commented Jan 4, 2024

FredBill1 commented Jan 28, 2024 • edited Loading

p-wysocki commented Jan 31, 2024

FredBill1 commented Jan 31, 2024 • edited Loading

FredBill1 commented Feb 16, 2024 • edited Loading

Segmentation fault when running `test_get_runtime_model` test #18388

Segmentation fault when running `test_get_runtime_model` test #18388

mlukasze commented Jul 19, 2023 •

edited

Loading

sanbuphy commented Dec 15, 2023 •

edited

Loading

p-wysocki commented Dec 15, 2023 •

edited

Loading

FredBill1 commented Jan 28, 2024 •

edited

Loading

FredBill1 commented Jan 31, 2024 •

edited

Loading

FredBill1 commented Feb 16, 2024 •

edited

Loading