1. Introduction to Geoflow
Geoflow is a spatial ETL tool for processing 3D geo-information such as point clouds and 3D city models.
I started developing Geoflow while working on an algorithm for automatic 3D building reconstruction from point clouds (what currently underpins 3DBAG.nl). This turned out to be a complex process consisting of many small steps that are executed sequentially. The first step is to read the input datasets (eg. point cloud) and the last step to write the 3D building model to a file. And, most importantly, in between there are many steps that take some data, process it and output new data. At the beginning of my work I did not know what the final algorithm would look like. And as I was working on the reconstruction algorithm and trying different ideas I quickly realised that it was very important to me to be able to quickly
- (visually) inspect and validate the output of each step in the process,
- reconfigure the behaviour of one step and immediately see the effects on the rest of the pipeline, and
- replace the operations in one steps with an alternative implementation, or remove and add steps all-together.
And thus Geoflow was born. I built it to be fast, interactive, and modular. It consists of a GUI application that allows one to build and run 'flowcharts'. Flowcharts contains pipelines of 'nodes' (each node is a 'step' that performs a certain operation as described above) and connections between nodes (to direct how the data flows). A command line application also exists that can be used to run flowchart files (human readable JSON) on headless servers.
Geoflow is designed around the idea of the flowchart. You create a flowchart by building graphs of nodes and connections.
- A node is a processing unit. It takes data from its input terminals (on the left side of the node), performs some processing, and delivers the results on the output terminals (on the right side of the node). Nodes can be created thought the menu that appears when right clicking on the flowchart canvas.
- With connections one can connect output terminals to input terminals of other nodes. Connections can be created by dragging from one terminal to another terminal
You can start a node in Geoflow by clicking on the Run button in the menu that appears when you right click on a node. The node will then start processing. Once a node has finished processing it will turn green and the output results are pushed to the output terminals. When a terminal has data pushed to it, it wil also turn green. In case there is a connection on an output terminal, the input terminal on the other end of the connection will also be notified that data is available, and it will also turn green. A node can only start processing when all its input terminals have data/are green.
The example above shows a simple flowchart with three nodes. There are two Number nodes on the left and they are connected to an Adder node (which reads the numbers on the input terminals and outputs their sum). The two number nodes are special in the sense that they do not have input terminals. They are therefore called root nodes. In this example the Number nodes simply set a user-configurable number (a parameter) on the output terminal. In practice, root nodes often read data from disk and output that data.
You can see that the Adder node automatically turns green once we manually run the two Number nodes. This is the default behaviour: once its input terminals are green a node automatically starts processing. Only root nodes do not run automatically. You have to either manually start them in the right-click menu, or you can run all the root nodes at once through the Flowchart menu on the upper side of the window.
So, when the root nodes are ran all the dependent nodes (the children of the root node) automatically follow. Each node will simply start running when all its inputs are available.
Once an entire flowchart has completed running all its nodes are green. And if you now connect a new node to the available output terminals, it will immediately run as well, since it's inputs are already available. The parent nodes do not need to be ran again.
If you decide to manually re-run a node with eg. different parameters all the children of that node will be re-ran automatically as well.
You can disable the automatic execution of a node by disabling the Autorun option in its context menu.
Once you have set up a flowchart to your liking you can save it to disk as a JSON file using the Save button in the File menu. Conversely, you can load a flowchart from disk through the same menu.
As shown in the animation above each node has a context menu. In this menu you can run the node, change it's name or change the autorun setting on the node. You can also configures the node parameters in this menu. Parameters affect how a node does its processing. They could for example be integer of float thresholds in geometric algorithms or booleans that enable/disable a certain functionality in the node.
When you save a flowchart the current parameters are saved as well and will be restored when you load that flowchart later.
Globals are parameters that a user can create in the Globals menu on the top. After creating a global it can be linked to one or more node parameters.
Typically you would create globals for the most important parameters in the flowchart, and for values that are reused by several different nodes.
A big benefit of global parameters is that these are automatically exposed in the command line version of geoflow. You can thus override globals directly from the command line when you run the flowchart in production (which is typically done with the command line version). You could use globals for example for the file paths of the input/output files of the flowchart.
geof <flowchart file> [--config <TOML config file with globals>] [--GLOBAL1 <value> --GLOBAL2 <value> ...]
You can also simply print just information on the plugins that are loaded with:
geof info
Takes the same parameters as geof on the command line, but opens a GUI instead of simply running a flowchart.
Warning: the GUI is currently experimental.
- Right click to open the menu to create new nodes
- Drag from input/output terminals to make connections
- Right click on a node to access its context menu
- Translate in the 3D viewer by left-mouse dragging while holding
ctrl, faster zooming by holdingctrlwhile scrolling.
https://github.com/geoflow3d/gfc-lod13
The geoflow-bundle repository bundles geoflow together with the actively developed plugins and some example flowcharts (eg. for building reconstruction). With each release of this repository some binary packages become available:
Geoflow can be compiled for all major platforms (Linux, Mac, Windows). We prioritised the above packages, but we aim to provide readily usable binaries for other platforms too in the future.