docs(tutorial): add a tutorial for the Flink backend

[chloeh13q]

Description of changes

Add a tutorial for the Flink backend.

[ncclementi]

@chloeh13q for thelinting error you are getting, you can run the pre-commit and that should fix them. https://ibis-project.org/contribute/03_style

[ncclementi]

The remaining CI failure is because pyflink is missing in the docs dependencies. We should add it to the group using poetry

poetry add --group docs pyflink

Then we will want to run

poetry lock --no-upgrade

to update the lock file, then push all those changes up. I think we also might need to re-create the requirements-dev.txt but I'm not sure, maybe @gforsyth can chime in here?

[gforsyth]

If you have https://github.com/casey/just installed, you can run just lock to handle the relock and regeneration of the requirements dev file, Otherwise, these are the commands to run:

poetry lock --no-update
poetry export --extras all --with dev --with test --with docs --without-hashes --no-ansi > requirements-dev.txt

[cpcloud]

This is unlikely to work, due to docs being built with nix. I suggest freezing the output for now until we can get this working properly.

[gforsyth]

Suggested change

	# Getting started
	---
	execute:
	freeze: auto
	---
	# Getting started

to freeze the output as it gets generated on your machine you can add this to the top of each of the qmd files

[chloeh13q]

We're hosting the example repo under our github account right now, but will most likely need to move this somewhere else (under ibis-project?), clean up some of the code, and figure out how best to structure multiple tutorials and blog posts.

[lostmygithubaccount]

this can now be updated to https://github.com/ibis-project/ibis-flink-example

[ibis-docs-bot]

Docs preview: https://pr-8085-97438faf0c1ee09b9389920e4aecc39da094697f--ibis-quarto.netlify.app

[lostmygithubaccount]

in general, wrap lines. I would keep the tutorial document a bit more simple and focused on getting the user up and running

[lostmygithubaccount]

this can now be updated to https://github.com/ibis-project/ibis-flink-example

[chloeh13q]

@lostmygithubaccount Thanks for the review, I addressed all of your comments!

I would keep the tutorial document a bit more simple and focused on getting the user up and running

The intention is to get to the point where the user can write a transformation. If this is too long, I can split it into two and have the first tutorial go up to the point where we write a simple over aggregation, and the second post cover window aggregation and what the differences are (but the setup code would be identical, i.e. connect to a source, connect to a sink). Alternatively, we can just focus on window aggregation and get rid of the code for over aggregation. We can provide the code for over aggregation in the example repo but not explicitly talk about the syntax in this tutorial. What do you think?

P.S. Still pending porting the complete code example to the new ibis-flink-example repo.

[ibis-docs-bot]

Docs preview: https://pr-8085-197cc82920e67be7ede264dca7d8814708613b98--ibis-quarto.netlify.app

[ibis-docs-bot]

Docs preview: https://pr-8085-87e961371d1b98745be8ddfbcda69c363f0fe77c--ibis-quarto.netlify.app

[ibis-docs-bot]

Docs preview: https://pr-8085-916b9ab966868adc79baa781aa40587506b0131f--ibis-quarto.netlify.app

[lostmygithubaccount]

thanks!

[lostmygithubaccount]

this should be good to merge. #8197 is a follow up

[chloeh13q]

@lostmygithubaccount BTW, we're also working on adding a visualization dashboard to the demo. We think it adds some fun elements and could help make the demo more interactive and tangible. We will provide the complete code for the dashboard so that all that the user needs to do is to spin up the service for the dashboard (via one line command) and then they should be able to see plots on a local port. Does that sound reasonable to you? (We can put this in a separate PR because we're still tweaking some parts)

[lostmygithubaccount]

sounds great! follow up PR seems fine for that

what kind of dashboard is this? Quarto?

[chloeh13q]

No it's a Dash app. It's not a part of the Quarto rendering - in the tutorial it would tell users to go to http://127.0.0.1:8050/ in the browser to see the dashboard. We're also thinking about adding a grafana dashboard for more complex visualizations, but won't tackle that now.

[lostmygithubaccount]

why dash? if it's quarto we can put it right in the website (like the backend support matrix)

[chloeh13q]

From @mfatihaktas :

For the following reasons:

• As much as I could understand it, Quarto is for reproducible technical reporting. It integrates very well with the notebooks and allows for quick implementation of beautiful graphs. However, I could not find much information on how to use it as a "web service" in a containerized setting. The alternative would to implement a "wrapper python module" around Quarto API and insert calls to it in the demo notebook. I thought that this would "pollute/inflate" and might deviate/conflate the goal of the notebook: introducing Ibis/Flink.
• Quarto's functionality seems to be limited in terms of developing dynamic graphs. It allows for responding to user input (e.g., selecting a button on the graphs), however I could not find enough information on how to generate semi-runtime-plots with it.
• Dash requires very small amount of boilerplate code to turn data into a dashboard. Thus, we can easily replace it with any other tool that allows for serving dashboards over HTTP.

We should be able to switch to any other viz tool quickly as long as the dashboard tool works in a containerized setting. Otherwise, we would need to figure out how to bring the dashboard into the demo notebook.

[mfatihaktas]

From @mfatihaktas :

For the following reasons:

• As much as I could understand it, Quarto is for reproducible technical reporting. It integrates very well with the notebooks and allows for quick implementation of beautiful graphs. However, I could not find much information on how to use it as a "web service" in a containerized setting. The alternative would to implement a "wrapper python module" around Quarto API and insert calls to it in the demo notebook. I thought that this would "pollute/inflate" and might deviate/conflate the goal of the notebook: introducing Ibis/Flink.
• Quarto's functionality seems to be limited in terms of developing dynamic graphs. It allows for responding to user input (e.g., selecting a button on the graphs), however I could not find enough information on how to generate semi-runtime-plots with it.
• Dash requires very small amount of boilerplate code to turn data into a dashboard. Thus, we can easily replace it with any other tool that allows for serving dashboards over HTTP.

We should be able to switch to any other viz tool quickly as long as the dashboard tool works in a containerized setting. Otherwise, we would need to figure out how to bring the dashboard into the demo notebook.

Quick note. Dash apps can be embedded in Quarto easily as dash is powered by plotly.

[ibis-docs-bot]

Docs preview: https://pr-8085-5b7cfc17e41f4d7668a2acece7b0a268e697c6c2--ibis-quarto.netlify.app

[lostmygithubaccount]

thanks!