Real-time BigQuery data loading

[Prokos]

Hi!

I just started messing around with Observable Framework, but quickly ran into an issue. I'm trying to implement a graph that gives me real-time data directly queried from BigQuery which updates every x seconds. Something like this:

const counts = Generators.observe(notify => {
	const rows = [];

	const get = async () => {
		const results = await FileAttachment('data/counts.json').json();

		rows.push(results);

		notify(rows);
	}

	get();
	window.setInterval(get, 1000);
});

It seems like I cannot use a data loader like this, because they are cached at build time and so will not update even if I load it more than once.

However, the proposed solution I found elsewhere is to simply not use a data loader but do it in the notebook directly. This is not possible because the BigQuery library does not function in the browser.

Is this something that's simply impossible within the framework design unless I figure out a different way (maybe some in between API) to query BigQuery? Or is there for example some way to invalidate data loader cache during run time (and thus having the server regenerate the data source on-demand, or run/serve it directly)

[CobusT]

Hi @Prokos
One way to do this is to actually run the data loader on a schedule (every few seconds) and pipe stdout to the cache directory on your hosted platform. That way the data file refreshes on a schedule, and you would then use your Generator method in the markdown file to poll the data from that data file.

[Prokos]

@CobusT thanks for the reply! Yeah, I suppose that works :) also running yarn clean works (though it's a bit more intense).

Ideally I was hoping for a way to invalidate the cache from the client side, so the app code could determine whether a data source needs to be invalidated or not. Something like passing a cache=false or invalidate_at=timestamp parameter to the FileAttachment constructor. That'd allow somebody writing a "notebook" to have control over this without needing an additional server side implementation.

Anyway, I see a path forward if this proved to be truly important to us, thanks 👍