use_ref with canvas

[slauko]

hey 👋 ,

im trying to render a canvas with frequently changing drawdata like that:

    @component
    def Overlay(self):
        canvas = use_ref(None)

        @use_effect
        async def draw_loop():
            while True:
                data = self.draw()
                if data:
                    # print(data.get("circles"))
                    # ctx = canvas.current.get_context('2d')
                    # if ctx:
                    pass
                await asyncio.sleep(0.01)

        return html.div(
            {
                "id": "container",
            },
            html.style("""
                            * { 
                                margin: 0;
                                padding: 0;
                                box-sizing: border-box;
                            }
                            html {
                                height: 100vh;
                                width: 100vw;
                            }
                            .app, body, #container, #overlay {
                                height: 100vh;
                                width: 100vw;
                                overflow: hidden;
                                background-color: transparent !important;
                            }
                        """),
            html.canvas(
                {
                    "id": "overlay",
                    "ref": canvas,
                }
            )
        )

    def draw_backend_loop(self):
        run(self.Overlay, port=3001)

but i get an error:
TypeError: Object of type Ref is not JSON serializable

when trying to use the ref, i guess im doing something wrong. 😅

can anyone help me out?

[slauko]

for explaining, i already implemented it in ReactJS like this,
just want to switch to ReactPy 😄

export default function Overlay() {
	const canvas = useRef<HTMLCanvasElement>(null);
	const timeoutRef = useRef<NodeJS.Timeout | undefined>(undefined);

	const draw = () => {
		const api = window.pywebview?.api;
		if (!api) {
			return;
		}

		api.draw().then((data: DrawData) => {
			if (data) {
				const ctx = canvas.current?.getContext('2d');
				if (ctx) {
					ctx.canvas.width = data.width;
					ctx.canvas.height = data.height;
					ctx.clearRect(0, 0, data.width, data.height);

					for (const text of data.texts) {
						ctx.font = '30px Arial';
						ctx.fillStyle = text.color;
						ctx.fillText(text.text, text.pos[0], text.pos[1] + 30);
					}
					for (const line of data.lines) {
						ctx.beginPath();
						ctx.strokeStyle = line.color;
						ctx.moveTo(line.start[0], line.start[1]);
						ctx.lineTo(line.end[0], line.end[1]);
						ctx.stroke();
					}
					for (const circle of data.circles) {
						ctx.beginPath();
						for (let i = 0; i < circle.points.length - 1; i++) {
							ctx.strokeStyle = circle.points[i].color;
							ctx.moveTo(circle.points[i].pos[0], circle.points[i].pos[1]);
							ctx.lineTo(circle.points[i + 1].pos[0], circle.points[i + 1].pos[1]);
						}
						ctx.stroke();
					}
					for (const sprite of data.sprites) {
						const img = new Image();
						img.src = sprite.url;
						ctx.drawImage(img, sprite.pos[0], sprite.pos[1], sprite.size[0], sprite.size[1]);
					}
				}
			}
		});
	};

	const drawLoop = useCallback(() => {
		draw();
		timeoutRef.current = setTimeout(drawLoop, 10);
	}, []);

	useEffect(() => {
		drawLoop();
		return () => {
			if (timeoutRef.current) {
				clearTimeout(timeoutRef.current);
			}
		};
	}, [drawLoop]);

	return (
		<div style={{height: '100vh', width: '100vw', backgroundColor: 'transparent !important'}}>
			<canvas id='overlay' ref={canvas} style={{height: '100%', width: '100%', backgroundColor: 'transparent !important'}} />
		</div>
	);
}

[Archmonger]

It seems like this might require us to add some more features to reactpy.html.canvas.

With ReactJS, it will "fill in" the canvas context into the ref=... you provide. We currently do not have an equivalent feature in ReactPy. Currently, all props provided to ReactPy are serialized and sent to the client as-is. The issue you're seeing is that a Ref object is not serializable, which is true since it's a Python object.

Currently we have no mechanism that would enable ReactJS/ReactPy to intelligently "fill in" this ref. To add compatibility for this feature, ReactPy would need to start treating reactpy.html.canvas["ref"] as a unique prop.

In all likelihood, creating an equivalent API to ReactJS <canvas ref=... /> would rely on our future plans for a use_messenger hook to allow for arbitrary client->server real-time communication.

There's also a chance we develop this fully server-sided, but that would require some investigation if that's technologically feasible.

cc: @rmorshea