(newbie question) dynamically add Signal as new data comes in from an EventSource?

[choucavalier]

I have a very specific question regarding a real-time medical research data visualisation app that I am building in a French hospital.

I have a Python backend that consumes data with a high temporal resolution (high frequency) from a Kafka broker, calculates some lower-frequency signals from this data, and then exposes it through an EventSource server.

My Fresh server runs on the same machine as the Python backend, and exposes the real-time visualisation application to doctors.

On the client side, I want the client to directly connect to the Python backend's EventSource server to subscribe to streams of data.

Each data point that is coming through has a "label". Each plot has a specific set of labels that it's going to subscribe to, and use for its particular visualisation.

Plots might share labels (e.g. PlotA and PlotB will both subscribe to label xyz).

I am wondering how to structure this with Fresh. And in particular:

1. Where should my state, that's going to store and cache incoming data, be located? I want the state to be shared from one plot to another, such that switching from one plot to another is not triggering a reload of the data.
2. Should each plot be an island? I have looked at this project but I don't really understand whether the plot components are islands or not.
3. How can I dynamically create new preact Signal as new labels are being subscribed to? My thinking is that a plot should be updated only when data with the labels it's subscribing to received new data. i.e. if label y comes in but the current plot being displayed doesn't care about label y, then it should not be updated (useless recalculation).

In general, being new to (p)react and Fresh, I am still a bit confused by all these concepts.

If someone more experienced with Fresh and preact could give her two cents that would be great!

Thanks :)

[marvinhagemeister]

1. It sounds like the state is mostly global from the app's point of view. In that case you could add it somewhere above the plot components in the tree.
2. Yes, it's each plot should be an island as you want them to react to changes on the client and not just be rendered on the server.
3. You can either create new signals inside components via the useSignal hook or outside of components using the signal function. There are multiple ways to go about modeling your data in a way that fulfils your use case. It's less about the underlying tech like whether signals are used or not, but rather how you structure the data. It often helps to tackle these problems one at a time rather than trying to solve everything at once. A first iteration could be to just pipe the data from the event source into the plots directly. A second iteration could be to store the filter values of the plot somewhere, where the code reading the event source can react to. That way when an event comes in, you could check if there is any plot that cares about that kind of label to begin with. A third iteration could be to now try to cache past event data, etc.

[choucavalier]

thanks!

so here is a piece of code I came up with in ./islands/DataAware.tsx which creates a DataAware component, responsible for updating a global context that contains my data.

it uses fancy shallow state updates to only update the part of the data corresponding to the label for the message it receives.

import * as d3 from "https://cdn.skypack.dev/d3?dts";
import { useEffect, useState } from "preact/hooks";
import DataContext from "../contexts/DataContext.tsx";
import { META_SPECIAL_LABEL } from "../types/data.ts";
import {
  isRawPatientMetaMessage,
  isRawPointMessage,
  isRawTimeSeriesMessage,
  isRawVectorMessage,
  RawMessage,
} from "../types/message.ts";
import { maybe_convert_to_date, normalise_bed_label } from "../common.ts";

function state_modify_factory(msg: RawMessage) {
  if (isRawPointMessage(msg) || isRawVectorMessage(msg)) {
    return (
      (prev_state) => ({
        ...prev_state,
        [msg.pid]: {
          ...prev_state[msg.pid],
          [msg.lbl]: [
            ...((msg.pid in prev_state) && (msg.lbl in prev_state[msg.pid])
              ? prev_state[msg.pid][msg.lbl]
              : []),
            { val: msg.val, ts: d3.isoParse(msg.ts_recv) },
          ],
        },
      })
    );
  } else if (isRawPatientMetaMessage(msg)) {
    const meta = {
      "gender": msg["gender"],
      "ipp": msg["ipp"],
      "first_name": msg["first_name"],
      "last_name": msg["last_name"],
      "nbl": normalise_bed_label(msg["nbl"]),
      "dob": maybe_convert_to_date(msg["dob"]),
      "dtbegin": maybe_convert_to_date(msg["dtbegin"]),
    };
    return (
      (prev_state) => ({
        ...prev_state,
        [msg.pid]: {
          ...prev_state[msg.pid],
          [META_SPECIAL_LABEL]: [meta],
        },
      })
    );
  } else if (isRawTimeSeriesMessage(msg)) {
    console.warn(
      `got time series message which is not expected lbl=${msg.lbl}`,
    );
    return ((prev_state) => prev_state);
  } else {
    throw Error(`got unknown msg: ${msg}`);
  }
}

export default function DataAware({ children }) {
  const [data, setData] = useState({});
  let es: EventSource;

  useEffect(() => {
    es = new EventSource("http://localhost:8888/data");
    es.onopen = () => {
      console.log("connected to server-sent-event data stream");
    };
    es.onmessage = (event) => {
      const msg = JSON.parse(event.data);
      setData(state_modify_factory(msg));
      console.log(data);
    };
  }, []);

  return (
    <DataContext.Provider value={data}>
      {children}
    </DataContext.Provider>
  );
}

and the corresponding DataContext code

import { createContext } from "preact";

const DataContext = createContext({});

export default DataContext;

and the _app.tsx file

import { AppProps } from "$fresh/server.ts";
import Header from "../components/Header.tsx";
import { Head } from "$fresh/runtime.ts";
import DataAware from "../islands/DataAware.tsx";

export default function App({ Component, route }: AppProps) {
  return (
    <DataAware>
      <div class="wrapper">
        <Header route={route} />
        <Component />
      </div>
    </DataAware>
  );
}

and the actual component being displayed

import { Head } from "$fresh/runtime.ts";
import { useContext } from "preact/hooks";
import DataContext from "../contexts/DataContext.tsx";

export default function MyComponent() {
  const data = useContext(DataContext);
  return (
    <>
      <div class="p-4 mx-auto max-w-screen-lg">
        Plot will be here
        <p>
          Current data : {data.value}
        </p>
      </div>
    </>
  );
}

my data state is not being updated it seems. i checked the messages are coming in, and the setData code is indeed being called... i think i messed up something within my setData function.

any idea?

[choucavalier]

hmm, actually the data seems to be correctly set

when adding this to my DataAware function component

useEffect(() => {
    console.log(data);
}, [data]);

i see that the state is indeed being updated...

i'm wondering why displaying the data with data.value is not triggering the state updates...

(again, i'm new to all of this, so i'm probably missing some basic understanding of how (p)react works)

thanks a ton @marvinhagemeister for answering my newbie questions!

[marvinhagemeister]

Can you share a repo we can check out? This makes it a little easier for me to inspect what's going on rather than manually copying a bunch of snippets from here and hoping I have the same file structure.