Handler Registry callbacks are not placed into the global callback queue

[cromachina]

Version of Dear PyGui

Version: 1.10.1
Operating System: Windows 10

My Issue/Question

I was testing out using DearPyGui with asyncio by handling async callbacks myself. This seems to work fine for item callbacks, however callbacks registered with a Handler Registry do not seem to get submitted to DearPyGui's global callback queue, thus I am unable to make these particular callbacks async. I'm guessing that the Handler Registry callbacks are submitted to a separate queue that is inaccessible from DearPyGui's API, or maybe they are just run immediately. I suppose a workaround for my particular goal would be to wrap callbacks before passing them to items or handlers so that when called by DearPyGui, they are submitted to my own queue for processing.

To Reproduce

I've constructed a minimal example (without asyncio) below that prints the contents of the DearPyGui callback queue. You can observe that when clicking on the button, you see the button's callback in the callback queue, but never the mouse handler's callback.

Possible sample output:

mouse click
((<function on_button_click at 0x0000014DDCBF7EC0>, 22, None, None),)
button click
mouse click

Expected behavior

A callback registered with the Handler Registry should always be submitted to the global callback registry when a user has opted for manual callback management: dpg.configure_app(manual_callback_management=True).

Possible expected output:

((<function on_mouse_handler_click at 0x000001D423EF7EC0>, 24, None, None), (<function on_button_click at 0x0000014DDCBF7EC0>, 22, None, None))
mouse click
button click
((<function on_mouse_handler_click at 0x000001D423EF7EC0>, 24, None, None),)
mouse click

Standalone, minimal, complete and verifiable example

import inspect
import dearpygui.dearpygui as dpg

dpg.create_context()
dpg.configure_app(manual_callback_management=True)
dpg.create_viewport(title='psd-export', width=600, height=200)
dpg.setup_dearpygui()

def run_callbacks():
    jobs = dpg.get_callback_queue()
    if jobs is not None:
        print(jobs)
        for job, *args in jobs:
            if job is not None:
                arg_slice = args[:len(inspect.signature(job).parameters)]
                job(*arg_slice)

def on_button_click():
    print('button click')
    dpg.later

with dpg.window(tag='window'):
    dpg.add_button(label='Export', callback=on_button_click)

def on_mouse_handler_click(sender):
    print('mouse click')

with dpg.handler_registry():
    dpg.add_mouse_click_handler(callback=on_mouse_handler_click)

def main():
    while dpg.is_dearpygui_running():
        run_callbacks()
        dpg.render_dearpygui_frame()

dpg.show_viewport()
dpg.set_primary_window('window', True)
main()
dpg.destroy_context()

Here is the minimal version with asyncio, in case you are curious. It's mostly the same, except async callbacks are forwarded to asyncio.

import asyncio
import inspect
import dearpygui.dearpygui as dpg

dpg.create_context()
dpg.configure_app(manual_callback_management=True)
dpg.create_viewport(title='psd-export', width=600, height=200)
dpg.setup_dearpygui()

pending_tasks = set()

def run_callbacks():
    jobs = dpg.get_callback_queue()
    if jobs is not None:
        for job, *args in jobs:
            if job is not None:
                arg_slice = args[:len(inspect.signature(job).parameters)]
                if inspect.iscoroutinefunction(job):
                    task = asyncio.create_task(job(*arg_slice))
                    pending_tasks.add(task)
                    task.add_done_callback(pending_tasks.discard)
                else:
                    job(*arg_slice)

async def on_button_click():
    print('button click')
    await asyncio.sleep(1)
    print('button click delay')

with dpg.window(tag='window'):
    dpg.add_button(label='Export', callback=on_button_click)

async def on_mouse_handler_click():
    print('mouse click')

with dpg.handler_registry():
    dpg.add_mouse_click_handler(callback=on_mouse_handler_click)

async def main():
    while dpg.is_dearpygui_running():
        run_callbacks()
        await asyncio.sleep(0)
        dpg.render_dearpygui_frame()

dpg.show_viewport()
dpg.set_primary_window('window', True)
asyncio.run(main())
dpg.destroy_context()

[cromachina]

And here is the wrapper workaround that I suggested above.

import asyncio
import inspect
import dearpygui.dearpygui as dpg

dpg.create_context()
dpg.create_viewport(title='psd-export', width=600, height=200)
dpg.setup_dearpygui()

loop = None
pending_tasks = set()

def callback(func):
    def wrapped(sender, app_data, user_data):
        args = (sender, app_data, user_data)
        arg_slice = args[:len(inspect.signature(func).parameters)]
        task = asyncio.run_coroutine_threadsafe(func(*arg_slice), loop)
        pending_tasks.add(task)
        task.add_done_callback(pending_tasks.discard)
    return wrapped

async def on_button_click():
    print('button click')
    await asyncio.sleep(1)
    print('button click delay')

with dpg.window(tag='window'):
    dpg.add_button(label='Export', callback=callback(on_button_click))

async def on_mouse_handler_click():
    print('mouse click')

with dpg.handler_registry():
    dpg.add_mouse_click_handler(callback=callback(on_mouse_handler_click))

async def main():
    global loop
    loop = asyncio.get_event_loop()
    while dpg.is_dearpygui_running():
        await asyncio.sleep(0)
        dpg.render_dearpygui_frame()

dpg.show_viewport()
dpg.set_primary_window('window', True)
asyncio.run(main())
dpg.destroy_context()

[v-ein]

This is because item handlers (mvItemHandlers.cpp) explicitly schedule callbacks using mvSubmitCallback() + mvRunCallback(), instead of using mvAddCallback() like regular callbacks do. Anything that goes around mvAddCallback will not honor manual_callback_management.

Not sure if it was done that way on purpose (probably not), but I agree that the "manual" flag must affect both callbacks and handlers. One way to do that is to make sure all scheduling goes through mvAddCallback.

I've done a substantial rework on callbacks in my local build of DPG, and this issue is one of the things I fixed there. However, to push a PR I'd need to rework it a little bit more, which will take some time - so don't expect my fix soon. Maybe somebody else can fix this particular issue before I push my PR (but honestly, this rabbit hole might be way deeper than one would think).

[v-ein]

For future reference, here's a list of callbacks in the current version of DPG that do not honor manual_callback_management:

• set_exit_callback
• callback on add_drag_ZZZ
• callback on table
• on_close on dpg.window
• cancel_callback on file_dialog
• all global handlers and item handlers (added to dpg.handler_registry or item_handler_registry)

[danfleck]

I ran into this today. The file_dialog.callback (not just the cancel_callback) also seems to not honor manual_callback_management.

[v-ein]

Re-checked it - yes, both callback and cancel_callback on file_dialog ignore the manual management flag (I somehow missed the "callback" if branch :)).