diff --git a/network/hubble/README.mdx b/network/hubble/README.mdx index 9eacc67bc..af1f13bc4 100644 --- a/network/hubble/README.mdx +++ b/network/hubble/README.mdx @@ -7,7 +7,7 @@ sidebar_position: 0 Hubble is an open-source, publicly available dataset that provides a complete historical record of the Stellar network. Similar to Horizon, it ingests and presents the data produced by the Stellar network in a format that is easier to consume than the performance-oriented data representations used by Stellar Core. The dataset is hosted on BigQuery–meaning it is suitable for large, analytic workloads, historical data retrieval and complex data aggregation. **Hubble should not be used for real-time data retrieval and cannot submit transactions to the network.** For real time use cases, we recommend [running an API server](../horizon/admin-guide/README.mdx). -This guide describes when to use Hubble and how to connect. To view the underlying data structures, queries and examples, use the [Viewing Metadata](./viewing-metadata.mdx) and [Optimizing Queries](./optimizing-queries.mdx) tutorials. +This guide describes when to use Hubble and how to connect. To view the underlying data structures, queries and examples, use the [Viewing Metadata](./analyst-guide/viewing-metadata.mdx) and [Optimizing Queries](./analyst-guide/optimizing-queries.mdx) tutorials. ## Why Use Hubble? diff --git a/network/hubble/admin-guide/README.mdx b/network/hubble/admin-guide/README.mdx new file mode 100644 index 000000000..0716e2ff9 --- /dev/null +++ b/network/hubble/admin-guide/README.mdx @@ -0,0 +1,10 @@ +--- +title: Admin Guide +sidebar_position: 15 +--- + +import DocCardList from "@theme/DocCardList"; + +All you need to know about running a Hubble analytics platform. + + \ No newline at end of file diff --git a/network/hubble/admin-guide/data-curation/README.mdx b/network/hubble/admin-guide/data-curation/README.mdx new file mode 100644 index 000000000..599382bbf --- /dev/null +++ b/network/hubble/admin-guide/data-curation/README.mdx @@ -0,0 +1,10 @@ +--- +title: Data Curation +sidebar_position: 20 +--- + +import DocCardList from "@theme/DocCardList"; + +Running stellar-dbt-public to transform raw Stellar network data into something better. + + \ No newline at end of file diff --git a/network/hubble/admin-guide/data-curation/architecture.mdx b/network/hubble/admin-guide/data-curation/architecture.mdx new file mode 100644 index 000000000..e01f25ee3 --- /dev/null +++ b/network/hubble/admin-guide/data-curation/architecture.mdx @@ -0,0 +1,22 @@ +--- +title: Architecture +sidebar_position: 10 +--- + +import stellar_dbt_arch from '/img/hubble/stellar_dbt_architecture.png'; + +## Architecture Overview + + + +In general stellar-dbt-public runs by: + +* Selecting a dbt model to run +* Within the model run: + * Sources are referenced and used to create staging tables + * Staging tables then undergo various transformations and are stored in intermediate tables + * Finishing touches and joins are done on the intermediate tables which produce the final analytics friendly mart tables + +We try to adhere to the best practices set by the [dbt docs](https://docs.getdbt.com/docs/build/projects) + +More detailed information about stellar-dbt-public and examples can be found in the [stellar-dbt-public](https://github.com/stellar/stellar-dbt-public/tree/master) repo. \ No newline at end of file diff --git a/network/hubble/admin-guide/data-curation/getting-started.mdx b/network/hubble/admin-guide/data-curation/getting-started.mdx new file mode 100644 index 000000000..1dac9b086 --- /dev/null +++ b/network/hubble/admin-guide/data-curation/getting-started.mdx @@ -0,0 +1,140 @@ +--- +title: Getting Started +sidebar_position: 20 +--- + +[stellar-dbt-public GitHub repository](https://github.com/stellar/stellar-dbt-public/tree/master) + +[stellar/stellar-dbt-public docker images](https://hub.docker.com/r/stellar/stellar-dbt-public) + +## Recommended Usage + +### Docker Image + +Generally if you do not need to modify any of the stellar-dbt-public code, it is recommended that you use the [stellar/stellar-dbt-public docker images](https://hub.docker.com/r/stellar/stellar-dbt-public) + +Example to run locally with docker: + +``` +docker run --platform linux/amd64 -ti stellar/stellar-dbt-public:latest +``` + +### Import stellar-dbt-public as a dbt Package + +Alternatively, if you need to build your own models on top of stellar-dbt-public, you can import stellar-dbt-public as a dbt package into a separate dbt project. + +Example instructions: + +* Create a new file `packages.yml` in your dbt project (not the stellar-dbt-public project) with the yml below + +``` +packages: + - git: "https://github.com/stellar/stellar-dbt-public.git" + revision: v0.0.28 +``` + +* (Optional) Update your profiles.yml to include profile configurations for stellar-dbt-public + +``` +new_project: + target: test + outputs: + test: + project: + dataset: + + +stellar_dbt_public: + target: test + outputs: + test: + project: + dataset: + +``` + +* (Optional) Update your dbt_project.yml to include project configurations for stellar-dbt-public + +``` +name: 'stellar_dbt' +version: '1.0.0' +config-version: 2 + +profile: 'new_project' + +model-paths: ["models"] +analysis-paths: ["analyses"] +test-paths: ["tests"] +seed-paths: ["seeds"] +macro-paths: ["macros"] +snapshot-paths: ["snapshots"] + +target-path: "target" +clean-targets: + - "target" + - "dbt_packages" + +models: + new_project: + staging: + +materialized: view + intermediate: + +materialized: ephemeral + marts: + +materialized: table + + stellar_dbt_public: + staging: + +materialized: ephemeral + intermediate: + +materialized: ephemeral + marts: + +materialized: table +``` + +* Models from the stellar-dbt-public package/repo will now be available in your new dbt project + +## Building and Running Locally + +### Clone the repo + +``` +git clone https://github.com/stellar/stellar-dbt-public +``` + +### Install required python packages + +``` +pip install --upgrade pip && pip install -r requirements.txt + +``` + +### Install required dbt packages + +``` +dbt deps +``` + +### Running dbt + +* There are many useful commands that come with dbt which can be found in the [dbt documentation](https://docs.getdbt.com/reference/dbt-commands#available-commands) +* stellar-dbt-public is designed to use the `dbt build` command which will `run` the model and `test` the model table output +* (Optional) run with the `--full-refresh` option + +``` +dbt build --full-refresh +``` + +* Subsequent runs can be run with incremental mode (only inserts the newest of data instead of rebuilding all of history every time) + +``` +dbt build +``` + +* You can also specify just a single model if you don't want to run all stellar-dbt-public models + +``` +dbt build --select +``` + +Please see the [stellar-dbt-public/modles/marts](https://github.com/stellar/stellar-dbt-public/tree/master/models/marts) directory to see a full list of the available models that dbt can run \ No newline at end of file diff --git a/network/hubble/admin-guide/data-curation/overview.mdx b/network/hubble/admin-guide/data-curation/overview.mdx new file mode 100644 index 000000000..115561803 --- /dev/null +++ b/network/hubble/admin-guide/data-curation/overview.mdx @@ -0,0 +1,15 @@ +--- +title: "Overview" +sidebar_position: 0 +--- + +Data curation in Hubble is done through [stellar-dbt-public](https://github.com/stellar/stellar-dbt-public). stellar-dbt-public transforms raw Stellar network data from BigQuery datasets and tables into aggregates for more user friendly analytics. + +It is worth noting that most users will not need to standup and run their own stellar-dbt-public instance. The Stellar Development Foundation provides public access to fully transformed Stellar network data through the public datasets and tables in GCP BigQuery. Instructions on how to access this data can be found in the [Connecting](https://developers.stellar.org/network/hubble/analyst-guide/connecting) section. + +## Why Run stellar-dbt-public? + +Running stellar-dbt-public within your own infrastructure provides a number of benefits. You can: + +- Have full operational control without dependency on the Stellar Development Foundation for network data +- Run modified ETL/ELT pipelines that fit your individual business needs \ No newline at end of file diff --git a/network/hubble/admin-guide/scheduling-and-orchestration/README.mdx b/network/hubble/admin-guide/scheduling-and-orchestration/README.mdx new file mode 100644 index 000000000..d66b63e8a --- /dev/null +++ b/network/hubble/admin-guide/scheduling-and-orchestration/README.mdx @@ -0,0 +1,10 @@ +--- +title: Scheduling and Orchestration +sidebar_position: 100 +--- + +import DocCardList from "@theme/DocCardList"; + +Stitching all the components together. + + \ No newline at end of file diff --git a/network/hubble/admin-guide/scheduling-and-orchestration/architecture.mdx b/network/hubble/admin-guide/scheduling-and-orchestration/architecture.mdx new file mode 100644 index 000000000..e37dfff36 --- /dev/null +++ b/network/hubble/admin-guide/scheduling-and-orchestration/architecture.mdx @@ -0,0 +1,18 @@ +--- +title: Architecture +sidebar_position: 10 +--- + +import stellar_etl_airflow_arch from '/img/hubble/stellar_etl_airflow_architecture.png'; + +## Architecture Overview + + + +In general stellar-etl-airflow runs by: + +* Scheduling DAGs to run `stellar-etl` and upload the data outputted to BigQuery +* Scheduling DAGs to run `stellar-dbt-public` using the data in BigQuery + * We try to adhere to the best practices set by the [dbt docs](https://docs.getdbt.com/docs/build/projects) + +More detailed information about stellar-etl-airflow can be found in the [stellar-etl-airflow](https://github.com/stellar/stellar-etl-airflow/tree/master) repo. \ No newline at end of file diff --git a/network/hubble/admin-guide/scheduling-and-orchestration/getting-started.mdx b/network/hubble/admin-guide/scheduling-and-orchestration/getting-started.mdx new file mode 100644 index 000000000..ae461cf4d --- /dev/null +++ b/network/hubble/admin-guide/scheduling-and-orchestration/getting-started.mdx @@ -0,0 +1,87 @@ +--- +title: Getting Started +sidebar_position: 20 +--- + +import history_table_export from '/img/hubble/history_table_export.png'; +import state_table_export from '/img/hubble/state_table_export.png'; +import dbt_enriched_base_tables from '/img/hubble/dbt_enriched_base_tables.png'; + +[stellar-etl-airflow GitHub repository](https://github.com/stellar/stellar-etl-airflow/tree/master) + +## GCP Account Setup + +The Stellar Development Foundation runs Hubble in GCP using Composer and BigQuery. To follow the same deployment you will need to have access to GCP project. Instructions can be found in the [Get Started](https://cloud.google.com/docs/get-started) documentation from Google. + +Note: BigQuery and Composer should be available by default. If they are not you can find instructions for enabling them in the [BigQuery](https://cloud.google.com/bigquery?hl=en) or [Composer](https://cloud.google.com/composer?hl=en) Google documentation. + +## Create GCP Composer Instance to Run Airflow + +Instructions on bringing up a GCP Composer instance to run Hubble can be found in the [Installation and Setup](https://github.com/stellar/stellar-etl-airflow?tab=readme-ov-file#installation-and-setup) section in the [stellar-etl-airflow](https://github.com/stellar/stellar-etl-airflow) repository. + +:::note + +Hardware requirements can be very different depending on the Stellar network data you require. The default GCP settings may be higher/lower than actually required. + +::: + +## Configuring GCP Composer Airflow + +There are two things required for the configuration and setup of GCP Composer Airflow: + +* Upload DAGs to the Composer Airflow Bucket +* Configure the Airflow variables for your GCP setup + +For more detailed instructions please see the [stellar-etl-airflow Installation and Setup](https://github.com/stellar/stellar-etl-airflow?tab=readme-ov-file#installation-and-setup) documentation. + +### Uploading DAGs + +Within the [stellar-etl-airflow](https://github.com/stellar/stellar-etl-airflow) repo there is an [upload_static_to_gcs.sh](https://github.com/stellar/stellar-etl-airflow/blob/master/upload_static_to_gcs.sh) shell script that will upload all the DAGs and schemas into your Composer Airflow bucket. + +This can also be done using the [gcloud CLI or console](https://cloud.google.com/storage/docs/uploading-objects) and manually selecting the dags and schemas you wish to upload. + +### Configuring Airflow Variables + +Please see the [Airflow Variables Explanation](https://github.com/stellar/stellar-etl-airflow?tab=readme-ov-file#airflow-variables-explanation) documentation for more information about what should and needs to be configured. + +## Running the DAGs + +To run a DAG all you have to do is toggle the DAG on/off as seen below + +![Toggle DAGs](/img/hubble/airflow_dag_toggle.png) + +More information about each DAG can be found in the [DAG Diagrams](https://github.com/stellar/stellar-etl-airflow?tab=readme-ov-file#dag-diagrams) documentation. + +## Available DAGs + +More information can be found [here](https://github.com/stellar/stellar-etl-airflow/blob/master/README.md#public-dags) + +### History Table Export DAG + +[This DAG](https://github.com/stellar/stellar-etl-airflow/blob/master/dags/history_tables_dag.py): + +- Exports part of sources: ledgers, operations, transactions, trades, effects and assets from Stellar using the data lake of LedgerCloseMeta files + - Optionally this can ingest data using captive-core but that is not ideal nor recommended for usage with Airflow +- Inserts into BigQuery + + + +### State Table Export DAG + +[This DAG](https://github.com/stellar/stellar-etl-airflow/blob/master/dags/state_table_dag.py) + +- Exports accounts, account_signers, offers, claimable_balances, liquidity pools, trustlines, contract_data, contract_code, config_settings and ttl. +- Inserts into BigQuery + + + +### DBT Enriched Base Tables DAG + +[This DAG](https://github.com/stellar/stellar-etl-airflow/blob/master/dags/dbt_enriched_base_tables_dag.py) + +- Creates the DBT staging views for models +- Updates the enriched_history_operations table +- Updates the current state tables +- (Optional) warnings and errors are sent to slack. + + \ No newline at end of file diff --git a/network/hubble/admin-guide/scheduling-and-orchestration/overview.mdx b/network/hubble/admin-guide/scheduling-and-orchestration/overview.mdx new file mode 100644 index 000000000..9075c8331 --- /dev/null +++ b/network/hubble/admin-guide/scheduling-and-orchestration/overview.mdx @@ -0,0 +1,15 @@ +--- +title: "Overview" +sidebar_position: 0 +--- + +Hubble uses [stellar-etl-airflow](https://github.com/stellar/stellar-etl-airflow) to schedule and orchestrate all its workflows. This includes the scheduling and running of stellar-etl and stellar-dbt. + +It is worth noting that most users will not need to standup and run their own Hubble. The Stellar Development Foundation provides public access to the data through the public datasets and tables in GCP BigQuery. Instructions on how to access this data can be found in the [Connecting](https://developers.stellar.org/network/hubble/connecting) section. + +## Why Run stellar-etl-ariflow? + +Running stellar-etl-airflow within your own infrastructure provides a number of benefits. You can: + +- Have full operational control without dependency on the Stellar Development Foundation for network data +- Run modified ETL/ELT pipelines that fit your individual business needs \ No newline at end of file diff --git a/network/hubble/admin-guide/source-system-ingestion/README.mdx b/network/hubble/admin-guide/source-system-ingestion/README.mdx new file mode 100644 index 000000000..2043b6250 --- /dev/null +++ b/network/hubble/admin-guide/source-system-ingestion/README.mdx @@ -0,0 +1,10 @@ +--- +title: Source System Ingestion +sidebar_position: 10 +--- + +import DocCardList from "@theme/DocCardList"; + +Running stellar-etl for Stellar network data ingestion. + + \ No newline at end of file diff --git a/network/hubble/admin-guide/source-system-ingestion/architecture.mdx b/network/hubble/admin-guide/source-system-ingestion/architecture.mdx new file mode 100644 index 000000000..94b9dd3ae --- /dev/null +++ b/network/hubble/admin-guide/source-system-ingestion/architecture.mdx @@ -0,0 +1,25 @@ +--- +title: Architecture +sidebar_position: 10 +--- + +import stellar_arch from '/img/hubble/stellar_overall_architecture.png'; +import stellar_etl_arch from '/img/hubble/stellar_etl_architecture.png'; + +## Architecture Overview + + + + + +In general stellar-etl runs by: + +* Read raw data from the Stellar network +* This can be done by running a stellar-etl export command to export data between a start and end ledger + * stellar-etl has the ability to read from two different sources: + * Captive-core directly to get LedgerCloseMeta + * A data lake of compressed LedgerCloseMeta files from Ledger Exporter +* Tranforms the LedgerCloseMeta XDR into an easy to parse JSON format +* Optionally uploads the JSON files to GCS or any other cloud storage service + +More detailed information about stellar-etl and examples can be found in the [stellar-etl](https://github.com/stellar/stellar-etl/tree/master) repo. \ No newline at end of file diff --git a/network/hubble/admin-guide/source-system-ingestion/getting-started.mdx b/network/hubble/admin-guide/source-system-ingestion/getting-started.mdx new file mode 100644 index 000000000..31fd6b378 --- /dev/null +++ b/network/hubble/admin-guide/source-system-ingestion/getting-started.mdx @@ -0,0 +1,124 @@ +--- +title: Getting Started +sidebar_position: 20 +--- + +[stellar-etl GitHub repository](https://github.com/stellar/stellar-etl/tree/master) + +[stellar/stellar-etl docker images](https://hub.docker.com/r/stellar/stellar-etl) + +## Recommended Usage + +Generally if you do not need to modify any of the stellar-etl code, it is recommended that you use the [stellar/stellar-etl docker images](https://hub.docker.com/r/stellar/stellar-etl). + +Example to run locally with docker: + +``` +docker run --platform linux/amd64 -ti stellar/stellar-etl:latest +``` + +## Building and Running Locally + +### Install Golang + +* Make sure your golang version >= `1.22.1` + * Instructions to install golang can be found at [go.dev/doc/install](https://go.dev/doc/install) + +### Clone the repo + +``` +git clone https://github.com/stellar/stellar-etl +``` + +### Build stellar-etl + +* Run `go build` in the cloned stellar-etl repo + +``` +go build +``` + +### Run stellar-etl + +* A `stellar-etl` executable should have been created in your stellar-etl repo +* Example stellar-etl command: + +``` +./stellar-etl export_ledgers -s 10 -e 11 +``` + +This should create a `exported_ledgers.txt` file with the output of ledgers from ledgers 10 to 11: + +``` +{"base_fee":100,"base_reserve":100000000,"closed_at":"2024-02-06T17:34:12Z","failed_transaction_count":0,"fee_pool":0,"id":42949672960,"ledger_hash":"f7c89b35c50f74dc69eacd9dda8e9ec9f1af36b6a2928b77619c1beb5f5ca8d4","ledger_header":"AAAAAIGFrRh+oCo2QcAjG6IzWTlil89DNwYIwx6PrrmehujNf44MwMJZxPz3DJYHciV9ligoKwbmeiue4eM29CRWBJgAAAAAZcJtlAAAAAAAAAABAAAAANVyadliUPdJbQeb4ug1Ejbv/+jTnC4Gv6uxQh8X/GccAAAAQBW0ICM/1C7CML6ngZijKycAOIhzwGN6yUsznHznfJunIDyLLVF9/oqvLzP1vaGOhBf3Rmtm5WgGVgeLjlyJSAHfP2GYBKkv20BXGS3EPddI6neK3FK8SYzoBSTAFLgRGfBr+YHFQTIEJ0Y81WEOYClgyjOER8vd4qMQb3gM9nRvAAAACg3gtrOnZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABkBfXhAAAAAGQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=","max_tx_set_size":100,"operation_count":0,"previous_ledger_hash":"8185ad187ea02a3641c0231ba23359396297cf43370608c31e8faeb99e86e8cd","protocol_version":0,"sequence":10,"soroban_fee_write_1kb":0,"successful_transaction_count":0,"total_coins":1000000000000000000,"transaction_count":0,"tx_set_operation_count":"0"} +{"base_fee":100,"base_reserve":100000000,"closed_at":"2024-02-06T17:34:17Z","failed_transaction_count":0,"fee_pool":0,"id":47244640256,"ledger_hash":"5b9ac11c6040f4e2fa6a120b3dee9a4b338b7a25bcb8437dab0c0a5c557a41f5","ledger_header":"AAAAAPfImzXFD3TcaerNndqOnsnxrza2opKLd2GcG+tfXKjUK858NP5gM0pneHF0nRowsJBAzMwWDx0+tmbYIZkIT+8AAAAAZcJtmQAAAAAAAAABAAAAANVyadliUPdJbQeb4ug1Ejbv/+jTnC4Gv6uxQh8X/GccAAAAQDhZKPKBdeD4Sthcu+EsuzEtSyiXzXkHboOsgYT1tuV/juZyKqgrsVmg+RmMoRun+NKCdcB8LV9gaehiFm+XDgnfP2GYBKkv20BXGS3EPddI6neK3FK8SYzoBSTAFLgRGfBr+YHFQTIEJ0Y81WEOYClgyjOER8vd4qMQb3gM9nRvAAAACw3gtrOnZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABkBfXhAAAAAGQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=","max_tx_set_size":100,"operation_count":0,"previous_ledger_hash":"f7c89b35c50f74dc69eacd9dda8e9ec9f1af36b6a2928b77619c1beb5f5ca8d4","protocol_version":0,"sequence":11,"soroban_fee_write_1kb":0,"successful_transaction_count":0,"total_coins":1000000000000000000,"transaction_count":0,"tx_set_operation_count":"0"} +``` + +## stellar-etl Commands + +### export_ledgers + +``` +stellar-etl export_ledgers --start-ledger 1000 --end-ledger 500000 --output exported_ledgers.txt +``` + +This command exports ledgers within the provided range. + +### export_transactions + +``` +stellar-etl export_transactions --start-ledger 1000 --end-ledger 500000 --output exported_transactions.txt +``` + +This command exports transactions within the provided range. + +### export_operations + +``` +stellar-etl export_operations --start-ledger 1000 --end-ledger 500000 --output exported_operations.txt +``` + +This command exports operations within the provided range. + +### export_effects + +``` +stellar-etl export_effects --start-ledger 1000 --end-ledger 500000 --output exported_effects.txt +``` + +This command exports effects within the provided range. + +### export_assets + +``` +stellar-etl export_assets --start-ledger 1000 --end-ledger 500000 --output exported_assets.txt +``` + +Exports the assets that are created from payment operations over a specified ledger range. + + +### export_trades + +``` +stellar-etl export_trades --start-ledger 1000 --end-ledger 500000 --output exported_trades.txt +``` + +Exports trade data within the specified range to an output file + +### export_diagnostic_events + +``` +stellar-etl export_diagnostic_events --start-ledger 1000 --end-ledger 500000 --output export_diagnostic_events.txt +``` + +Exports diagnostic events data within the specified range to an output file + +### export_ledger_entry_changes + +``` +stellar-etl export_ledger_entry_changes --start-ledger 1000 --end-ledger 500000 --output exported_changes_folder/ +``` + +This command exports ledger changes within the provided ledger range. + +Note that this command will also exports every state change for each ledger entry type. [Information](https://github.com/stellar/stellar-etl?tab=readme-ov-file#export_ledger_entry_changes) on options to only output specifc ledger entry types. \ No newline at end of file diff --git a/network/hubble/admin-guide/source-system-ingestion/overview.mdx b/network/hubble/admin-guide/source-system-ingestion/overview.mdx new file mode 100644 index 000000000..891a88129 --- /dev/null +++ b/network/hubble/admin-guide/source-system-ingestion/overview.mdx @@ -0,0 +1,15 @@ +--- +title: "Overview" +sidebar_position: 0 +--- + +Stellar network data ingestion in Hubble is done through [stellar-etl](https://github.com/stellar/stellar-etl/tree/master). stellar-etl reads and transforms Stellar network data into OLAP friendly JSON files. + +It is worth noting that most users will not need to standup and run their own stellar-etl instance. The Stellar Development Foundation provides public access to fully transformed Stellar network data through the public datasets and tables in GCP BigQuery. Instructions on how to access this data can be found in the [Connecting](https://developers.stellar.org/network/hubble/analyst-guide/connecting) section. + +## Why Run stellar-etl? + +Running stellar-etl within your own infrastructure provides a number of benefits. You can: + +- Have full operational control without dependency on the Stellar Development Foundation for network data +- Run modified ETL/ELT pipelines that fit your individual business needs \ No newline at end of file diff --git a/network/hubble/admin-guide/visualization/README.mdx b/network/hubble/admin-guide/visualization/README.mdx new file mode 100644 index 000000000..8fe1522b0 --- /dev/null +++ b/network/hubble/admin-guide/visualization/README.mdx @@ -0,0 +1,10 @@ +--- +title: Visualization +sidebar_position: 30 +--- + +import DocCardList from "@theme/DocCardList"; + +Visualizing Stellar network data. + + \ No newline at end of file diff --git a/network/hubble/admin-guide/visualization/getting-started.mdx b/network/hubble/admin-guide/visualization/getting-started.mdx new file mode 100644 index 000000000..aa54a52f7 --- /dev/null +++ b/network/hubble/admin-guide/visualization/getting-started.mdx @@ -0,0 +1,41 @@ +--- +title: Getting Started +sidebar_position: 20 +--- + +This sections goes through using [Google's Looker Studio](https://lookerstudio.google.com/u/0/navigation/reporting) as a free and easy to use visualization tool that you can hook up your BigQuery Stellar network data to. + +There are many other free/paid visualization tools available. Hubble is compatible with any visualization tool with a BigQuery connector. + +## Creating your first visualization + +* Follow [Google's Quick Start Guide](https://support.google.com/looker-studio/answer/9171315?hl=en) + +## Hooking Up Data Sources + +The following will use the Stellar Development Foundations public datasets and tables as an example to hook up data sources to Looker Studio + +* Click `Create` in [Google's Looker Studio](https://lookerstudio.google.com/u/0/navigation/reporting) +* Click `Data Source` +* Find the `BigQuery` connector +* Use the project `crypto-stellar` +* Use the dataset `crypto_stellar` +* Select the table of interest +* Click `CONNECT` + +When you create a new report you should be able to now access data from `crypto-stellar.crypto_stellar.` + +## Making your first pie chart + +* Click `Create` in [Google's Looker Studio](https://lookerstudio.google.com/u/0/navigation/reporting) +* Click `Report` +* Click `My data sources` +* Click the data source you added above +* A table of the data should appear in a new report +* Click on the table +* Click on `Chart` on the right sidebar +* Click on the `Pie chart` image + +You have now created a new report with a pie chart. + +Looker Studio has many resources to help visualize and explore data. Learn more [here](https://support.google.com/looker-studio?sjid=9035399711189270749-NA#topic=6267740) \ No newline at end of file diff --git a/network/hubble/admin-guide/visualization/overview.mdx b/network/hubble/admin-guide/visualization/overview.mdx new file mode 100644 index 000000000..c2e6bda0e --- /dev/null +++ b/network/hubble/admin-guide/visualization/overview.mdx @@ -0,0 +1,6 @@ +--- +title: "Overview" +sidebar_position: 0 +--- + +There are various ways to visulize data from Hubble. The following section will go through the steps to use [Google's Looker Studio](https://cloud.google.com/looker-studio?hl=en) to help visualize Stellar network data. \ No newline at end of file diff --git a/network/hubble/analyst-guide/README.mdx b/network/hubble/analyst-guide/README.mdx new file mode 100644 index 000000000..49f4348cc --- /dev/null +++ b/network/hubble/analyst-guide/README.mdx @@ -0,0 +1,10 @@ +--- +title: Analyst Guide +sidebar_position: 15 +--- + +import DocCardList from "@theme/DocCardList"; + +All you need to know to use Hubble data for analysis. + + \ No newline at end of file diff --git a/network/hubble/connecting.mdx b/network/hubble/analyst-guide/connecting.mdx similarity index 100% rename from network/hubble/connecting.mdx rename to network/hubble/analyst-guide/connecting.mdx diff --git a/network/hubble/optimizing-queries.mdx b/network/hubble/analyst-guide/optimizing-queries.mdx similarity index 98% rename from network/hubble/optimizing-queries.mdx rename to network/hubble/analyst-guide/optimizing-queries.mdx index eb946d832..d94aa34f2 100644 --- a/network/hubble/optimizing-queries.mdx +++ b/network/hubble/analyst-guide/optimizing-queries.mdx @@ -19,7 +19,7 @@ Read the docs on [Viewing Metadata](./viewing-metadata.mdx) to learn more about #### Example - Profiling Operation Types -Let’s say you wanted to profile the [types of operations](../../docs/learn/fundamentals/transactions/list-of-operations) submitted to the Stellar Network monthly. +Let’s say you wanted to profile the [types of operations](../../../docs/learn/fundamentals/transactions/list-of-operations) submitted to the Stellar Network monthly. diff --git a/network/hubble/viewing-metadata.mdx b/network/hubble/analyst-guide/viewing-metadata.mdx similarity index 100% rename from network/hubble/viewing-metadata.mdx rename to network/hubble/analyst-guide/viewing-metadata.mdx diff --git a/static/img/hubble/airflow_dag_toggle.png b/static/img/hubble/airflow_dag_toggle.png new file mode 100644 index 000000000..5632dce35 Binary files /dev/null and b/static/img/hubble/airflow_dag_toggle.png differ diff --git a/static/img/hubble/dbt_enriched_base_tables.png b/static/img/hubble/dbt_enriched_base_tables.png new file mode 100644 index 000000000..d1c3b8551 Binary files /dev/null and b/static/img/hubble/dbt_enriched_base_tables.png differ diff --git a/static/img/hubble/history_table_export.png b/static/img/hubble/history_table_export.png new file mode 100644 index 000000000..e02ce0ca6 Binary files /dev/null and b/static/img/hubble/history_table_export.png differ diff --git a/static/img/hubble/state_table_export.png b/static/img/hubble/state_table_export.png new file mode 100644 index 000000000..55683987f Binary files /dev/null and b/static/img/hubble/state_table_export.png differ diff --git a/static/img/hubble/stellar_dbt_architecture.png b/static/img/hubble/stellar_dbt_architecture.png new file mode 100644 index 000000000..ba0f0b8c4 Binary files /dev/null and b/static/img/hubble/stellar_dbt_architecture.png differ diff --git a/static/img/hubble/stellar_etl_airflow_architecture.png b/static/img/hubble/stellar_etl_airflow_architecture.png new file mode 100644 index 000000000..892c0fae3 Binary files /dev/null and b/static/img/hubble/stellar_etl_airflow_architecture.png differ diff --git a/static/img/hubble/stellar_etl_architecture.png b/static/img/hubble/stellar_etl_architecture.png new file mode 100644 index 000000000..75e8493a3 Binary files /dev/null and b/static/img/hubble/stellar_etl_architecture.png differ diff --git a/static/img/hubble/stellar_overall_architecture.png b/static/img/hubble/stellar_overall_architecture.png new file mode 100644 index 000000000..77ed1470e Binary files /dev/null and b/static/img/hubble/stellar_overall_architecture.png differ