dynamically add to dataset

[aakim26]

I'm trying to keep the data I display in kepler updated with my database in real time. The only way I've been able to do this so far is by deleting the current layers with the associated dataID, adding the data with the new row to the map with addDataToMap, and then restyling the layers.

This works with smaller datasets, but seems very inefficient with larger ones. It would be nice if it was possible to dynamically add (or delete) a row from a dataset currently displayed on the map- and see those changes reflected in the layers. I'm imagining something like addToDataset(dataId, newRow).

If there is a way to currently do this that I'm overlooking or other suggestions I'd love to hear them.
Thanks!

[macrigiuseppe]

thank you for using Kepler.gl.

Currently we don't have a straight way to update an existing dataset.

In the meanwhile, you can change your flow by skipping removeDataSet and call addDataToMap with your new data. Make sure the new dataset id that you are passing as addDataToMap call matches the existing one; by doing so you are going to override existing data with your new data.

[aakim26]

Hey, thanks for the response and suggestion!

I've tried not removing the layers associated with the dataset and just calling addDataToMap- while it does update the existing layers it also creates new ones (for example, when working with icon layers I end up with a point layer and an icon layer in addition to my original icon layer). Is there a way to set the configuration so that this doesn't happen (when I call addDataToMap I generally have config: {})?

[macrigiuseppe]

@aakim26 you need to pass the same configuration that is currently in your kepler state as part of your addDataToMap call.

You can use KeplerGlSchema to export the configuration before you call addDataToMap

[macrigiuseppe]

@aakim26 you can find a tutorial on how to dynamically update a dataset with new data here: http://vis.academy/#/kepler.gl/3-load-config

please see section "2. Replace an existing dataset with new data with the same format , keeping the current config."

[aakim26]

@macrigiuseppe thanks for the link! I'm going to leave the issue open because I think the ability to modify datasets without completely replacing them would still be very useful. Especially using large datasets it could really increase the efficiency.

[prune998]

any idea how to get the data from a remote URL instead of a file ?

[varunpant]

@prune998 You could fetch the csv and config, and then just do something like:
dataset = csv file
config = json map configuration file understood by kepler

loadSampleConfigurations(dataSet,dataConfig){ var dataId = dataConfig.config.visState.layers[0].config.dataId; const data = Processors.processCsvData(dataSet); const dataset = { data, info: { id: dataId } }; this.props.dispatch(addDataToMap({datasets: dataset, config: dataConfig})); };

[macrigiuseppe]

@prune998
We just added a new feature to load maps/datasets from remote URL. Please feel free to check out the pull request here with instructions on how to load maps/datasets from url: #260.

P.S. please do not use the same issue for multiple topics

[vararp25]

@aakim26
I have embed kepler.gl locally and want to fetch data directly from database which is not stored locally. I am new to react.js so having difficulty to resolve it. So can you tell me how can i fetch it from database directly.

[derek-nascent]

Is this going to re-render the tiles each time from the beginning or progressively add new data? My scenario is such that a 200mb dataset might be small. I'd like to be able to stream in the dataset in chunks and add them to the map as they become available, but not force a full re-render of the tiles each time I add a new chunk.

[heshan0131]

This is a bigger effort we will tackle later, to support data streaming in chucks, and not update the entire dataset

[piyushkandoi]

Hey, thanks for the response and suggestion!

I've tried not removing the layers associated with the dataset and just calling addDataToMap- while it does update the existing layers it also creates new ones (for example, when working with icon layers I end up with a point layer and an icon layer in addition to my original icon layer). Is there a way to set the configuration so that this doesn't happen (when I call addDataToMap I generally have config: {})?

Hi,

I've saved the Kepler map that I created and customized using Python as an HTML file (populated with some default data). I needed to update data in the map on an event and thus I'm updating data in __KeplerDataConfig using an API on the occurrence of that event. Could you please help me with calling addDataToMap function to refresh the map component so that it reads the updated __KeplerDataConfig (with updated data) and thus the updates the data reflected on the map.

Thanks!

[mmisbahali]

Hey, has this been worked on?

[ChristopherLyon]

Any update? I have a complicated setup, and it needs to be manually replicated every time I update the data...

[cdolan92]

Also would love the ability to do this in a simple way

[sinannoureddine]

Any update regarding this task? It's not very convenient to delete all data sets and call "AddDataToMap" every time i want to update my data.