1. Introduction to Geoflow
THIS PAGE IS OUTDATED
Geoflow is a spatial ETL tool for processing 3D geo-information such as point clouds and 3D city models. The primary use case for Geoflow at this moment is automatic 3D building reconstruction from point clouds.
This document will provide the necessary information to get started with the Geoflow software. Once you have mastered the basic concepts presented here, you can proceed with the tutorial LoD1.3 building reconstruction
I started developing Geoflow while working on an algorithm for automatic 3D building reconstruction from point clouds (what currently underpins 3DBAG.nl). This turned out to be a complex process consisting of many small steps that are executed sequentially. The first step is to read the input datasets (eg. point cloud) and the last step to write the 3D building model to a file. And, most importantly, in between there are many steps that take some data, process it and output new data. At the beginning of my work I did not know what the final algorithm would look like. And as I was working on the reconstruction algorithm and trying different ideas I quickly realised that it was very important to me to be able to quickly
- (visually) inspect and validate the output of each step in the process,
- reconfigure the behaviour of one step and immediately see the effects on the rest of the pipeline, and
- replace the operations in one steps with an alternative implementation, or remove and add steps all-together.
And thus Geoflow was born. I built it to be fast, interactive, and modular. It consists of a GUI application that allows one to build and run 'flowcharts'. Flowcharts contains pipelines of 'nodes' (each node is a 'step' that performs a certain operation as described above) and connections between nodes (to direct how the data flows). A command line application also exists that can be used to run flowchart files (human readable JSON) on headless servers.
Geoflow is designed around the idea of the flowchart. You create a flowchart by building graphs of nodes and connections. This most easy to do with the Geoflow GUI.
- A node is a processing unit. It takes data from its input terminals (on the left side of the node), performs some processing, and delivers the results on the output terminals (on the right side of the node). Nodes can be created thought the menu that appears when right clicking on the flowchart canvas.
- With connections one can connect output terminals to input terminals of other nodes. Connections can be created by dragging from one terminal to another terminal
You can start a node in Geoflow by clicking on the Run button in the menu that appears when you right click on a node. The node will then start processing. Once a node has finished processing it will turn green and the output results are pushed to the output terminals. When a terminal has data pushed to it, it wil also turn green. In case there is a connection on an output terminal, the input terminal on the other end of the connection will also be notified that data is available, and it will also turn green. A node can only start processing when all its input terminals have data/are green.
The example above shows a simple flowchart with three nodes. There are two Number nodes on the left and they are connected to an Adder node (which reads the numbers on the input terminals and outputs their sum). The two number nodes are special in the sense that they do not have input terminals. They are therefore called root nodes. In this example the Number nodes simply set a user-configurable number (a parameter) on the output terminal. In practice, root nodes often read data from disk and output that data.
You can see that the Adder node automatically turns green once we manually run the two Number nodes. This is the default behaviour: once its input terminals are green a node automatically starts processing. Only root nodes do not run automatically. You have to either manually start them in the right-click menu, or you can run all the root nodes at once through the Flowchart menu on the upper side of the window.
So, when the root nodes are ran all the dependent nodes (the children of the root node) automatically follow. Each node will simply start running when all its inputs are available.
Once an entire flowchart has completed running all its nodes are green. And if you now connect a new node to the available output terminals, it will immediately run as well, since it's inputs are already available. The parent nodes do not need to be ran again.
If you decide to manually re-run a node with eg. different parameters all the children of that node will be re-ran automatically as well.
You can disable the automatic execution of a node by disabling the Autorun option in its context menu.
Once you have set up a flowchart to your liking you can save it to disk as a JSON file using the Save button in the File menu. Conversely, you can load a flowchart from disk through the same menu.
Notice that any relative filepaths in the flowchart file are relative to the folder that holds the flowchart file.
As shown in the animation above each node has a context menu. In this menu you can run the node, change it's name or change the autorun setting on the node. You can also configures the node parameters in this menu. Parameters affect how a node does its processing. They could for example be integer of float thresholds in geometric algorithms or booleans that enable/disable a certain functionality in the node.
When you save a flowchart the current parameters are saved as well and will be restored when you load that flowchart later.
Globals are parameters that a user can create in the Globals menu on the top. After creating a global it can be linked to one or more node parameters.
Typically you would create globals for the most important parameters in the flowchart, and for values that are reused by several different nodes.
A big benefit of global parameters is that these are automatically exposed in the command line version of geoflow. You can thus override globals directly from the command line when you run the flowchart in production (which is typically done with the command line version). You could use globals for example for the file paths of the input/output files of the flowchart.
Geoflow has an experimental 3D viewer. This viewer can be used to visualise spatial data on the output terminal of any node. This works by connection an output terminal with a special Painter node that can be added like any other node through the right click menu in the flowchart canvas.
Geoflow can be operated through two interfaces: the command line interface (CLI) and the graphical user interface (GUI).
The CLI version (the geof executable) is mainly intended to quickly run an existing flowchart and works also on systems without graphical capability (servers etc). It therefore is more portable than the GUI version and it is the recommended way to run flowcharts in a production environment.
The GUI version (the geoflow executable) can be used to visually create new flowcharts and edit/inspect/modify existing flowcharts. This is therefore recommended when developing/finetuning/debugging flowcharts.
You can use the geof executable as follows:
geof <flowchart file> [--config <TOML config file with globals>] [--GLOBAL1 <value> --GLOBAL2 <value> ...]
A simple geoflow myflowchart.json will automatically start all the root node in the flowchart, which should also trigger the execution of the rest of the flowchart (see section Automatic node execution).
You can also override the values of global parameters in the flowchart on the command line. This is done by either
- appending
--GLOBAL1 valueto the command. HereGLOBAL1is the name of the global parameter andvalueshould be replaced by the value that you want to assign to the global, and/or - providing a configuration file with
--config myconfig.tmlThis is a more convenient way to provide a bunch of global values in a text file. See eg. this example config file.
Notice that globals provided through option 1 will take priority over the ones provided by option 2.
Takes the same parameters as geof on the command line, but opens a GUI (as shown in the screenshots above) instead of simply running a flowchart.
The GUI windows has a menu bar on the top, a flowchart panel that shows the graph of the current flowchart, and a 3D Viewer panel that is used to render the geometries that are routed to Painter nodes in the flowchart.
Can be used to
- open and save flowchart files (file menu),
- run all root nodes or remove all nodes (flowchart menu), and to
- create/edit/remove global parameters (globals menu).
This is where you create, configure, and manage flowcharts. The most important controls are:
- Right click to open the flowchart context menu to create new nodes
- Right click on a node to access its node context menu. This is where you can run an individual node and configure its parameters. Press ctrl-click to enter a numerical value on a slider.
- Shift-click to (de-)select a node,
- Ctrl-shift-click on the canvas to deselect all nodes
- Drag nodes to move a single unselected node or a group of selected nodes around on the flowchart canvas
- Press the
deletekey to remove the selected node(s) - Scroll to move around the canvas
- Drag between input/output terminals to make connections
- Double-click on a connection to remove it
The 3D viewer is accompanies by a panel with Painters. In the Painters panel you can manage the different available painter, eg. you can hide certain painters and change the appearance of the geometries in the 3D view.
The 3D viewer itself renders the geomteries as outputted by the active painters. On right click a menu shows up where you can reset the view and configure a number of other settings such as the background color and the field of view of the camera.
The controls for the viewer are:
- Rotate by left click and drag
- Translate by left-click dragging while holding
ctrl - Zoom with scroll wheel, faster zooming by holding
ctrlwhile scrolling.