dataDM 💬📊

DataDM is your private data assistant. A conversational interface for your data where you can load, clean, transform, and visualize without a single line of code. DataDM is open source and can be run entirely locally, keeping your juicy data secrets fully private.

Demo

datadm_happiness_qa_and_plots.mp4

Note: Demo above is GPT-4, which sends the conversation to OpenAI's API. To use in full local mode, be sure to select starchat-alpha-cuda or starchat-beta-cuda as the model. This will use the StarChat model, which is a bit less capable but runs entirely locally.

⚠️ LLMs are known to hallucinate and generate fake results. So, double-check before trusting their results blindly!

Join our discord to join the community and share your thoughts!

Features

• Persistent Juptyer kernel backend for data manipulation during conversation
• Run entirely locally, keeping your data private
• Natural language chat, visualizations/plots, and direct download of data assets
• Easy to use docker-images for one-line deployment
• Load multiple tables directly into the chat
• Search for data and load CSVs directly from github
• Option to use OpenAI's GPT-3.5 or GPT-4 (requires API key)
• WIP: GGML based mode (CPU only, no GPU required)
• WIP: Rollback kernel state when undo using criu (re-execute all cells)
• TODO: Support for more data sources (e.g. SQL, S3, PySpark etc.)
• TODO: Export a conversation as a notebook or html

Things you can ask DataDM

• Load data from a URL
• Clean data by removing duplicates, nulls, outliers, etc.
• Join data from multiple tables into a single output table
• Visualize data with plots and charts
• Ask whatever you want to your very own private code-interpreter

Quickstart

You can use docker, colab, or install locally.

1. Docker to run locally

docker run -e OPENAI_API_KEY={{YOUR_API_KEY_HERE}} -p 7860:7860 -it ghcr.io/approximatelabs/datadm:latest

For local-mode using StarChat model (requiring a CUDA device with at least 24GB of RAM)

docker run --gpus all -p 7860:7860 -it ghcr.io/approximatelabs/datadm:latest-cuda

2. Colab to run in the cloud

3. Use as a python package

⚠️ datadm used this way runs LLM generated code in your userspace

For local-data, cloud-model mode (no GPU required) - requires an OpenAI API key

$ pip install datadm
$ datadm

For local-mode using StarChat model (requiring a CUDA device with at least 24GB of RAM)

$ pip install "datadm[cuda]"
$ datadm

Special Thanks

• starchat-beta (starcoder with databricks-dolly and OpenAssistant/oasst1)
• Guidance
• HuggingFace
• OpenAI

Contributions

Contributions are welcome! Feel free to submit a PR or open an issue.

Community

Join the Discord to chat with the team

Check out our other projects: sketch and approximatelabs