From bdfe85d70e9ee1a6b4fd4fc1b8f16142acbe16c7 Mon Sep 17 00:00:00 2001 From: rob Date: Fri, 14 Feb 2020 00:33:39 -0800 Subject: [PATCH 1/6] initial proposal --- grants/speculative/load_balanced_endpoints.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 grants/speculative/load_balanced_endpoints.md diff --git a/grants/speculative/load_balanced_endpoints.md b/grants/speculative/load_balanced_endpoints.md new file mode 100644 index 000000000..e69de29bb From 8aab49342df6e55a019717a929536647ecb8d48d Mon Sep 17 00:00:00 2001 From: rob Date: Fri, 14 Feb 2020 00:41:58 -0800 Subject: [PATCH 2/6] adding formating --- grants/speculative/load_balanced_endpoints.md | 83 +++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/grants/speculative/load_balanced_endpoints.md b/grants/speculative/load_balanced_endpoints.md index e69de29bb..c0bdb9e98 100644 --- a/grants/speculative/load_balanced_endpoints.md +++ b/grants/speculative/load_balanced_endpoints.md @@ -0,0 +1,83 @@ +## **Project Description** + +This project focuses on building a common set of infrastructure as code modules used to deploy autoscaling sentry layers and public nodes for DoS protection and load balanced endpoints. The project aims to integrate into the existing polkadot-secure-validator architecture and serve as drop in replacements for the single node / multi-provider public node deployments with secure tunnels to the validator node on a private network. Code will use the existing DevOps patterns already adopted by the Web3 Foundation and extend them with additional e2e testing, automation, and benchmarking. The proposal will lay the ground work for more advanced load balanced API endpoints and caching solutions by proving out simpler patterns for serving archival requests. + +## **Team members** + +- Lead Engineer: Rob Cannon +- Principal Investigator: Mitchell Krawiec-Thayer, Ph.D. + +## **Team Website** + +- [www.insightdatascience.com](http://www.insightdatascience.com/) +- [www.insightconsensus.com](http://www.insightconsensus.com/) + +## **Legal Structure** + +Rob and Mitchell are employees of Insight, an existing US-based company that has been supporting specialized engineers since 2012. + +## **Team's experience** + +**Rob Cannon** + +DevOps developer in residence at Insight who focuses primarily on blockchain infrastructure. Focussing primarily on building automated deployments of validator nodes for the ICON blockchain, he is now looking to extend these patterns into other blockchains. Prior to Insight, he worked in the oil and gas industry founding a data science startup and working for a major operator. + +[Github](https://github.com/robc-io), [Medium](https://medium.com/@robcannonxyz) + +**Mitchell Krawiec-Thayer** + +Decentralized Consensus Lead at Insight, mentor for Insight Residents. Have been doing privacy research and protocol SecEng for Monero Research Lab since 2017, and founded Noncesense Research Lab. Specializes in private protocol design, anti-heuristic engineering, empirical analysis, on & off-chain anomaly detection, metadata archive design & analysis. + +[GitHub](https://github.com/mitchellpkt/), [Twitter](https://twitter.com/Mitchellpkt0), [LinkedIn](https://www.linkedin.com/in/mitchellpkt/), [Medium](https://medium.com/@mitchellpkt) + +## Team Code Repos + +- [https://github.com/insight-infrastructure/terragrunt-polkadot](https://github.com/insight-infrastructure/terragrunt-polkadot) +- [https://github.com/insight-infrastructure/terraform-polkadot-aws-sentry-node](https://github.com/insight-infrastructure/polkadot-terragrunt) + +## Team LinkedIn Profiles + +- [https://www.linkedin.com/in/mitchellpkt/](https://www.linkedin.com/in/mitchellpkt/) +- [https://www.linkedin.com/in/rob-cannon-21571317/](https://www.linkedin.com/in/rob-cannon-21571317/) + +## Development Roadmap + +The basics for building the necessary infrastructure with a packer, ansible, and terraform based workflow are in place. Minimal templating is performed over configuration files with actions triggered off a single Makefile to deploy the stack. + +The real challenge with this proposal is to minimize the sync time needed to scale out public facing nodes which requires either pre-synced images or downloading the DB directly from a peer. Pre-syncing is done from taking images off a "source of truth" node and keeping a CDN updated with snapshots that can be pulled down with the latest blocks. Downloading the DB from a peer is the existing W3F approach and is as simple as scaling out the application from scratch. While API vendors like Infura favor the pre-syncing approach, both options have their own benefits and will be explored through a series of benchmarks. + +**Milestone 1: IaC Modules** + +[1-2 months] + +--- + +- Build source of truth nodes and autoscaling groups on AWS, Azure, and GCP with terraform +- Build integration testing with terratest over all providers +- Build agent to trigger periodic snapshots being taken to object store +- Build logging and monitoring exporters and service discovery agents + +**Milestone 2: Benchmarking** + +[1 Month] + +--- + +- Build load balancer and reverse proxy layer +- Integrate with existing secure validator CLI and/or build own CLI tool to support multiple node configurations +- Build testbench for various types of benchmarking to optimize scaling policies and source of truth node snapshot increments +- Build helm chart distribution + +### Long Term Plans + +--- + +- Comprehensive library of IaC modules that can be used and repurposed in a variety of contexts +- CLI tooling and API endpoints for deploying modular sets of self-healing infrastructure +- Caching layers and ETL pipelines to build indexed data endpoints to mimic Infura functionality + +## Additional Information + +While it is important for decentralized network operations to avoid homogeneity, certain components that have a low impact on the diversity of the network are prime targets for standardization. We feel that as the ecosystem grows, having a battle-tested set of modules will prove to be very valuable for best practice to be shared around the community. + +The methodology of pre-syncing images off a "source of truth" node was inspired by Infura's architectural pattern as within the context of ethereum, it is unreasonable to scale nodes with autoscaling groups when the nodes themselves take many hours to sync from scratch. As the polkadot ecosystem grows and hence the size of the chain data, it will be critical for operators to easily adopt best practices so that they are ready to scale out their infrastructure as the increase in API requests grows. \ No newline at end of file From ddb93a41d6faad69f8499a9e8526c54ee3e47be4 Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 25 Feb 2020 13:55:32 -0800 Subject: [PATCH 3/6] adding richard's bio and updating timelines --- grants/speculative/load_balanced_endpoints.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/grants/speculative/load_balanced_endpoints.md b/grants/speculative/load_balanced_endpoints.md index c0bdb9e98..ee4f0f704 100644 --- a/grants/speculative/load_balanced_endpoints.md +++ b/grants/speculative/load_balanced_endpoints.md @@ -22,7 +22,14 @@ Rob and Mitchell are employees of Insight, an existing US-based company that has DevOps developer in residence at Insight who focuses primarily on blockchain infrastructure. Focussing primarily on building automated deployments of validator nodes for the ICON blockchain, he is now looking to extend these patterns into other blockchains. Prior to Insight, he worked in the oil and gas industry founding a data science startup and working for a major operator. -[Github](https://github.com/robc-io), [Medium](https://medium.com/@robcannonxyz) +[Github](https://github.com/robc-io), [Medium](https://medium.com/@robcannonxyz), [Linkedin](https://www.linkedin.com/in/rob-cannon-21571317/) + +**Richard Mah** + +Holding a PhD in neuroscience, Richard has deep experience in the intersection between advanced data analysis and high performance computing. During his PhD, he employed a wide variety of machine learning techniques to automate the analysis of clinical data with a microservices-based architecture on Kubernetes. Since then, he became an Insight fellow working on immutable deployments of validator node infrastructure focusing specifically on monitoring and alarms. + +[Github](https://github.com/shinyfoil), [Website](https://www.richardmah.com/), [Linkedin](https://www.linkedin.com/in/richardmah/) + **Mitchell Krawiec-Thayer** @@ -46,9 +53,7 @@ The basics for building the necessary infrastructure with a packer, ansible, and The real challenge with this proposal is to minimize the sync time needed to scale out public facing nodes which requires either pre-synced images or downloading the DB directly from a peer. Pre-syncing is done from taking images off a "source of truth" node and keeping a CDN updated with snapshots that can be pulled down with the latest blocks. Downloading the DB from a peer is the existing W3F approach and is as simple as scaling out the application from scratch. While API vendors like Infura favor the pre-syncing approach, both options have their own benefits and will be explored through a series of benchmarks. -**Milestone 1: IaC Modules** - -[1-2 months] +**Milestone 1: IaC Modules** - 2 months --- @@ -57,9 +62,7 @@ The real challenge with this proposal is to minimize the sync time needed to sca - Build agent to trigger periodic snapshots being taken to object store - Build logging and monitoring exporters and service discovery agents -**Milestone 2: Benchmarking** - -[1 Month] +**Milestone 2: Benchmarking** - 1 month --- From 539f76c28b38557a04c4e62bcd3d24409f1cf931 Mon Sep 17 00:00:00 2001 From: rob Date: Mon, 9 Mar 2020 15:40:30 -0700 Subject: [PATCH 4/6] updating to refelct focus on endpoints with development --- grants/speculative/load_balanced_endpoints.md | 95 ++++++++++++++----- 1 file changed, 73 insertions(+), 22 deletions(-) diff --git a/grants/speculative/load_balanced_endpoints.md b/grants/speculative/load_balanced_endpoints.md index ee4f0f704..838f6fd4e 100644 --- a/grants/speculative/load_balanced_endpoints.md +++ b/grants/speculative/load_balanced_endpoints.md @@ -1,10 +1,10 @@ ## **Project Description** -This project focuses on building a common set of infrastructure as code modules used to deploy autoscaling sentry layers and public nodes for DoS protection and load balanced endpoints. The project aims to integrate into the existing polkadot-secure-validator architecture and serve as drop in replacements for the single node / multi-provider public node deployments with secure tunnels to the validator node on a private network. Code will use the existing DevOps patterns already adopted by the Web3 Foundation and extend them with additional e2e testing, automation, and benchmarking. The proposal will lay the ground work for more advanced load balanced API endpoints and caching solutions by proving out simpler patterns for serving archival requests. +This project focuses on building a set of load balanced endpoints for the Polkadot and Substrate chains. Specifically, we will be building a common set of reusable modules that can be deployed on each cloud provider. Development will initially focus on a VM based architecture for building autoscaling archival nodes but will transition to a cloud native Kubernetes based deployment with intelligent routing and caching layers. We will then build templates whereby microservices can be developed by Insight data engineering fellows in a repeatable manner. The project will give users easy to use deployments for hosting their own full nodes and evolve into a managed API service. Code will use existing DevOps patterns already adopted by the Web3 Foundation and extend them with additional e2e testing, automation, and benchmarking. ## **Team members** -- Lead Engineer: Rob Cannon +- Engineers: Richard Mah, Rob Cannon - Principal Investigator: Mitchell Krawiec-Thayer, Ph.D. ## **Team Website** @@ -14,7 +14,7 @@ This project focuses on building a common set of infrastructure as code modules ## **Legal Structure** -Rob and Mitchell are employees of Insight, an existing US-based company that has been supporting specialized engineers since 2012. +Rob and Mitchell are employees of Insight, an existing US-based company that has been supporting specialized engineers since 2012. Richard will be brought on as a contractor focusing on this grant for the duration. ## **Team's experience** @@ -39,8 +39,11 @@ Decentralized Consensus Lead at Insight, mentor for Insight Residents. Have been ## Team Code Repos +**All repos WIP / examples of deployment strategy** - [https://github.com/insight-infrastructure/terragrunt-polkadot](https://github.com/insight-infrastructure/terragrunt-polkadot) -- [https://github.com/insight-infrastructure/terraform-polkadot-aws-sentry-node](https://github.com/insight-infrastructure/polkadot-terragrunt) +- [https://github.com/insight-infrastructure/terraform-polkadot-aws-network](https://github.com/insight-infrastructure/terraform-polkadot-aws-network) +- [https://github.com/insight-infrastructure/terraform-polkadot-aws-sentry-node](https://github.com/insight-infrastructure/terraform-polkadot-aws-sentry-node) +- [https://github.com/insight-infrastructure/terraform-polkadot-gcp-network](https://github.com/insight-infrastructure/terraform-polkadot-gcp-network) ## Team LinkedIn Profiles @@ -49,38 +52,86 @@ Decentralized Consensus Lead at Insight, mentor for Insight Residents. Have been ## Development Roadmap -The basics for building the necessary infrastructure with a packer, ansible, and terraform based workflow are in place. Minimal templating is performed over configuration files with actions triggered off a single Makefile to deploy the stack. +There are two basic types of repos. -The real challenge with this proposal is to minimize the sync time needed to scale out public facing nodes which requires either pre-synced images or downloading the DB directly from a peer. Pre-syncing is done from taking images off a "source of truth" node and keeping a CDN updated with snapshots that can be pulled down with the latest blocks. Downloading the DB from a peer is the existing W3F approach and is as simple as scaling out the application from scratch. While API vendors like Infura favor the pre-syncing approach, both options have their own benefits and will be explored through a series of benchmarks. +1. Terraform modules to deploy the infrastructure +2. Terragrunt scaffolding to house the deployment of the individual terraform modules -**Milestone 1: IaC Modules** - 2 months +All deployment steps are wrapped with terraform including calling packer builds, ansible configuration, and deployment of helm charts. These terraform modules are then wrapped with terragrunt which have all the parameters exposed through configuration files that are rendered with Jinja. This way, we can go from a high level configuration file that then informs the critical parameters we want to adjust across the whole stack. It also decouples the templating side of the configuration management from the terraform module itself. Other terraform projects taken up by W3F perform rendering within the terraform modules themselves which inhibits their ability to be reused in other contexts. By contrast, the modules we build can be reused directly in a variety of contexts without rendering. Thus while the goal of this project is to build load balanced endpoints, the modules can easily be repurposed for validator node sentry layers. + +The first major design decision was to decide on whether to build the endpoints with VMs or on Kubernetes. While the ultimate goal of this project is to build a microservices based architecture with intelligent routing and a caching layer for pre-indexed queries, we felt that initially we should focus on building a viable version 1 endpoint for archival queries. Since archival nodes will be run on optimized instances with directly attached nvme volumes, the VM based nodes will likely stay in place for when intelligent routing and caching layers are added on. + +One of the real challenges with this proposal is to minimize the sync time needed to scale out public facing nodes which requires either pre-synced images or downloading the DB directly from a peer. Pre-syncing is done from taking images off a "source of truth" node and keeping a CDN updated with snapshots that can be pulled down with the latest blocks. Downloading the DB from a peer is the existing W3F approach and is as simple as scaling out the application from scratch. While API vendors like Infura favor the pre-syncing approach, both options have their own benefits and will be explored through a series of benchmarks. + +**Milestone 1: IaC Modules** + +**1 months - 12.5k USD** --- -- Build source of truth nodes and autoscaling groups on AWS, Azure, and GCP with terraform -- Build integration testing with terratest over all providers -- Build agent to trigger periodic snapshots being taken to object store -- Build logging and monitoring exporters and service discovery agents +**Goals:** + +- Build autoscaling groups, load balancers, and networking modules on AWS, Azure, GCP, and Digital Ocean with terraform, ansible, and packer packaged as reusable modules. +- Build module and scaffolding level CI tooling and testing with terratest +- Integrate logging and monitoring exporters and service discovery agents +- Expose v1 endpoint for full archival queries -**Milestone 2: Benchmarking** - 1 month +**Deliverables:** + +- Network modules for each provider - 4 repos +- Autoscaling groups and load balancers for each provider - 4 repos +- Documentation to support deployment processes + +**Milestone 2: Benchmarking** + +**1/2 Month - 6.25k USD** --- -- Build load balancer and reverse proxy layer -- Integrate with existing secure validator CLI and/or build own CLI tool to support multiple node configurations -- Build testbench for various types of benchmarking to optimize scaling policies and source of truth node snapshot increments -- Build helm chart distribution +- Conduct brief study to determine optimal scaling policies for autoscaling groups +- Determine metrics and thresholds to inform what chain size will require syncing optimizations (pre-synced images + CDN) + +**Deliverables:** + +- Optimization of scaling policies for autoscaling groups +- Report to W3F with findings and discuss next steps to handle syncing optimizations -### Long Term Plans +**Milestone 3: Kubernetes Modules** + +**1.5 months - 18.75k USD** + +--- + +- Build kubernetes clusters and associated provider specific manifests to bootstrap the clusters in line with the existing [polkadot-deployer](https://github.com/w3f/polkadot-deployer) tool. +- Build API gateway and service proxy for intelligent routing and caching layers +- Build microservice templates from which Insight data engineering fellows can build pre-indexed endpoints off of +- ETL pipelines for supporting pre-indexed queries + +**Deliverables:** + +- Kubernetes cluster modules and manifests for each provider - 4 infrastructure repos +- API gateway (likely Ambassador) and service proxy (likely Envoy) along with associated helm charts to build service mesh and supporting observability services - 1 repo or contributions to existing [polkadot-charts](https://github.com/w3f/polkadot-charts) repo +- Microservice cookiecutter templates + - Flask (Python) - 1 repo + - FastAPI (Python) - 1 repo +- Airflow ETL tools for batch processing - Contribute to [github.com/blockchain-etl/polkadot-etl](https://github.com/blockchain-etl/polkadot-etl) +- Documentation to support deployment process + +### **Long Term Plans** --- - Comprehensive library of IaC modules that can be used and repurposed in a variety of contexts -- CLI tooling and API endpoints for deploying modular sets of self-healing infrastructure -- Caching layers and ETL pipelines to build indexed data endpoints to mimic Infura functionality +- CLI tooling for deploying modular sets of self-healing infrastructure +- Microservices library to expose pre-indexed queries +- Additional versions of the API to reference different parachains + +## **Additional Information** + +While it is important for decentralized network operations to avoid homogeneity, certain components that have a low impact on the diversity of the network are prime targets for standardization. We feel that as the ecosystem grows, having a battle-tested set of reusable modules will prove to be very valuable for best practice to be shared around the community. -## Additional Information +CLI's are useful for making the deployments easily reconfigurable but can become quite opinionated when building in how optionality is expressed. Right now we leverage [cookiecutter](https://github.com/cookiecutter/cookiecutter) for both exposing a simple CLI for rendering scaffolding configs but also in how we plan on giving our Insight fellows starting points for their projects. Because we want to make sure our Insight fellows have the necessary boilerplate code to begin their projects, we are building libraries of starter packages to facilitate the productionization of their projects. -While it is important for decentralized network operations to avoid homogeneity, certain components that have a low impact on the diversity of the network are prime targets for standardization. We feel that as the ecosystem grows, having a battle-tested set of modules will prove to be very valuable for best practice to be shared around the community. +This project will get us to the doorstep of being building out various microservice endpoints that will make use of different types of caching and persistence layers. These types of endpoints make for fantastic Insight data engineering projects which we hope to recruit a number of fellows to take on during the session starting in June 2020 and beyond. Time permitting, we will be building boilerplate architectures to support both real-time and batch queries against a number of backends for different types of microservices. A month before this particular grant comes to an end, we will be soliciting feedback to inform what endpoints are most interesting to the foundation. -The methodology of pre-syncing images off a "source of truth" node was inspired by Infura's architectural pattern as within the context of ethereum, it is unreasonable to scale nodes with autoscaling groups when the nodes themselves take many hours to sync from scratch. As the polkadot ecosystem grows and hence the size of the chain data, it will be critical for operators to easily adopt best practices so that they are ready to scale out their infrastructure as the increase in API requests grows. \ No newline at end of file +What we are mainly focused on for now is creating a open framework from which additional APIs can be built. As new parachains come online, we hope to create additional endpoints to support their specialized requests. \ No newline at end of file From 9081704129d33db8fc2df51cce26722e35292f5c Mon Sep 17 00:00:00 2001 From: rob Date: Tue, 10 Mar 2020 14:19:09 -0700 Subject: [PATCH 5/6] updating a few langauge tweeks --- grants/speculative/load_balanced_endpoints.md | 215 +++++++++++++----- 1 file changed, 163 insertions(+), 52 deletions(-) diff --git a/grants/speculative/load_balanced_endpoints.md b/grants/speculative/load_balanced_endpoints.md index 838f6fd4e..3d61dfb06 100644 --- a/grants/speculative/load_balanced_endpoints.md +++ b/grants/speculative/load_balanced_endpoints.md @@ -1,6 +1,18 @@ ## **Project Description** -This project focuses on building a set of load balanced endpoints for the Polkadot and Substrate chains. Specifically, we will be building a common set of reusable modules that can be deployed on each cloud provider. Development will initially focus on a VM based architecture for building autoscaling archival nodes but will transition to a cloud native Kubernetes based deployment with intelligent routing and caching layers. We will then build templates whereby microservices can be developed by Insight data engineering fellows in a repeatable manner. The project will give users easy to use deployments for hosting their own full nodes and evolve into a managed API service. Code will use existing DevOps patterns already adopted by the Web3 Foundation and extend them with additional e2e testing, automation, and benchmarking. +This project focuses on building a set of load balanced endpoints for +the Polkadot and Substrate chains. Specifically, we will be building a +common set of reusable modules that can be deployed on each cloud +provider. Development will initially focus on a VM-based architecture +for building autoscaling archival nodes but will transition to a cloud +native Kubernetes-based deployment with intelligent routing and +caching layers. We will then build templates whereby microservices can +be developed by Insight data engineering fellows in a repeatable +manner. The project will give users easy to use deployments for +hosting their own full nodes and evolve into a managed API service. +Code will use existing DevOps patterns already adopted by the Web3 +Foundation and extend them with additional e2e testing, automation, +and benchmarking. ## **Team members** @@ -14,32 +26,59 @@ This project focuses on building a set of load balanced endpoints for the Polkad ## **Legal Structure** -Rob and Mitchell are employees of Insight, an existing US-based company that has been supporting specialized engineers since 2012. Richard will be brought on as a contractor focusing on this grant for the duration. +Rob and Mitchell are employees of Insight, an existing US-based +company that has been supporting specialized engineers since 2012. +Richard will be brought on as a contractor focusing on this grant for +the duration. ## **Team's experience** -**Rob Cannon** - -DevOps developer in residence at Insight who focuses primarily on blockchain infrastructure. Focussing primarily on building automated deployments of validator nodes for the ICON blockchain, he is now looking to extend these patterns into other blockchains. Prior to Insight, he worked in the oil and gas industry founding a data science startup and working for a major operator. - -[Github](https://github.com/robc-io), [Medium](https://medium.com/@robcannonxyz), [Linkedin](https://www.linkedin.com/in/rob-cannon-21571317/) - -**Richard Mah** - -Holding a PhD in neuroscience, Richard has deep experience in the intersection between advanced data analysis and high performance computing. During his PhD, he employed a wide variety of machine learning techniques to automate the analysis of clinical data with a microservices-based architecture on Kubernetes. Since then, he became an Insight fellow working on immutable deployments of validator node infrastructure focusing specifically on monitoring and alarms. - -[Github](https://github.com/shinyfoil), [Website](https://www.richardmah.com/), [Linkedin](https://www.linkedin.com/in/richardmah/) - - -**Mitchell Krawiec-Thayer** - -Decentralized Consensus Lead at Insight, mentor for Insight Residents. Have been doing privacy research and protocol SecEng for Monero Research Lab since 2017, and founded Noncesense Research Lab. Specializes in private protocol design, anti-heuristic engineering, empirical analysis, on & off-chain anomaly detection, metadata archive design & analysis. - -[GitHub](https://github.com/mitchellpkt/), [Twitter](https://twitter.com/Mitchellpkt0), [LinkedIn](https://www.linkedin.com/in/mitchellpkt/), [Medium](https://medium.com/@mitchellpkt) +### **Rob Cannon** + +DevOps developer in residence at Insight who focuses primarily on +blockchain infrastructure. Focussing primarily on building automated +deployments of validator nodes for the ICON blockchain, he is now +looking to extend these patterns into other blockchains. Prior to +Insight, he worked in the oil and gas industry founding a data science +startup and working for a major operator. + +[Github](https://github.com/robc-io), +[Medium](https://medium.com/@robcannonxyz), +[Linkedin](https://www.linkedin.com/in/rob-cannon-21571317/) + +### **Richard Mah** + +While attaining his PhD in neuroscience, Richard employed a wide +variety of machine learning techniques to automate the analysis of +clinical data with a microservices-based architecture on Kubernetes. +Since then, he became an Insight fellow working on immutable +deployments of validator node infrastructure focusing specifically on +monitoring and alarms. Richard has deep experience in the intersection +between advanced data analysis and high performance computing. + +[Github](https://github.com/shinyfoil), +[Website](https://www.richardmah.com/), +[Linkedin](https://www.linkedin.com/in/richardmah/) + +### **Mitchell Krawiec-Thayer** + +Mitchell started the blockchain engineering program at Insight in 2018 +and now leads the R & D Residents (while continuing privacy research +and protocol SecEng for Monero Research Lab). Specializations include +decentralized network security, anti-heuristic engineering, empirical +analysis, on & off-chain anomaly detection, metadata archive design & +analysis. Mitchell works closely with the Fellows and Residents to +ensure that they have access to the resources and mentorship that +empower them to excel at their engineering initiatives. + +[GitHub](https://github.com/mitchellpkt/), +[Twitter](https://twitter.com/Mitchellpkt0), +[LinkedIn](https://www.linkedin.com/in/mitchellpkt/), +[Medium](https://medium.com/@mitchellpkt) ## Team Code Repos -**All repos WIP / examples of deployment strategy** +**All repos WIP / examples of deployment strategy** - [https://github.com/insight-infrastructure/terragrunt-polkadot](https://github.com/insight-infrastructure/terragrunt-polkadot) - [https://github.com/insight-infrastructure/terraform-polkadot-aws-network](https://github.com/insight-infrastructure/terraform-polkadot-aws-network) - [https://github.com/insight-infrastructure/terraform-polkadot-aws-sentry-node](https://github.com/insight-infrastructure/terraform-polkadot-aws-sentry-node) @@ -52,18 +91,49 @@ Decentralized Consensus Lead at Insight, mentor for Insight Residents. Have been ## Development Roadmap -There are two basic types of repos. - -1. Terraform modules to deploy the infrastructure -2. Terragrunt scaffolding to house the deployment of the individual terraform modules - -All deployment steps are wrapped with terraform including calling packer builds, ansible configuration, and deployment of helm charts. These terraform modules are then wrapped with terragrunt which have all the parameters exposed through configuration files that are rendered with Jinja. This way, we can go from a high level configuration file that then informs the critical parameters we want to adjust across the whole stack. It also decouples the templating side of the configuration management from the terraform module itself. Other terraform projects taken up by W3F perform rendering within the terraform modules themselves which inhibits their ability to be reused in other contexts. By contrast, the modules we build can be reused directly in a variety of contexts without rendering. Thus while the goal of this project is to build load balanced endpoints, the modules can easily be repurposed for validator node sentry layers. - -The first major design decision was to decide on whether to build the endpoints with VMs or on Kubernetes. While the ultimate goal of this project is to build a microservices based architecture with intelligent routing and a caching layer for pre-indexed queries, we felt that initially we should focus on building a viable version 1 endpoint for archival queries. Since archival nodes will be run on optimized instances with directly attached nvme volumes, the VM based nodes will likely stay in place for when intelligent routing and caching layers are added on. - -One of the real challenges with this proposal is to minimize the sync time needed to scale out public facing nodes which requires either pre-synced images or downloading the DB directly from a peer. Pre-syncing is done from taking images off a "source of truth" node and keeping a CDN updated with snapshots that can be pulled down with the latest blocks. Downloading the DB from a peer is the existing W3F approach and is as simple as scaling out the application from scratch. While API vendors like Infura favor the pre-syncing approach, both options have their own benefits and will be explored through a series of benchmarks. - -**Milestone 1: IaC Modules** +There are two basic types of repos. + +1. Terraform modules to deploy the infrastructure +2. Terragrunt scaffolding to house the deployment of the individual +terraform modules + +All deployment steps are wrapped with terraform including calling +packer builds, ansible configuration, and deployment of helm charts. +These terraform modules are then wrapped with terragrunt which have +all the parameters exposed through configuration files that are +rendered with Jinja. This way, we can go from a high level +configuration file that then informs the critical parameters we want +to adjust across the whole stack. It also decouples the templating +side of the configuration management from the terraform module itself. +Other terraform projects taken up by W3F perform rendering within the +terraform modules themselves which inhibits their ability to be reused +in other contexts. By contrast, the modules we build can be reused +directly in a variety of contexts without rendering. Thus while the +goal of this project is to build load balanced endpoints, the modules +can easily be repurposed for validator node sentry layers. + +The first major design decision was to decide on whether to build the +endpoints with VMs or on Kubernetes. While the ultimate goal of this +project is to build a microservices based architecture with +intelligent routing and a caching layer for pre-indexed queries, we +felt that initially we should focus on building a viable version 1 +endpoint for archival queries. Since archival nodes will be run on +optimized instances with directly attached nvme volumes, the VM based +nodes will likely stay in place for when intelligent routing and +caching layers are added on. + +One of the real challenges with this proposal is to minimize the sync +time needed to scale out public facing nodes which requires either +pre-synced images or downloading the DB directly from a peer. +Pre-syncing is done from taking images off a "source of truth" node +and keeping a CDN updated with snapshots that can be pulled down with +the latest blocks. Downloading the DB from a peer is the existing W3F +approach and is as simple as scaling out the application from scratch. +While API vendors like Infura favor the pre-syncing approach, both +options have their own benefits and will be explored through a series +of benchmarks. + +### **Milestone 1: IaC Modules** **1 months - 12.5k USD** @@ -71,7 +141,9 @@ One of the real challenges with this proposal is to minimize the sync time neede **Goals:** -- Build autoscaling groups, load balancers, and networking modules on AWS, Azure, GCP, and Digital Ocean with terraform, ansible, and packer packaged as reusable modules. +- Build autoscaling groups, load balancers, and networking modules on +AWS, Azure, GCP, and Digital Ocean with terraform, ansible, and packer +packaged as reusable modules. - Build module and scaffolding level CI tooling and testing with terratest - Integrate logging and monitoring exporters and service discovery agents - Expose v1 endpoint for full archival queries @@ -82,56 +154,95 @@ One of the real challenges with this proposal is to minimize the sync time neede - Autoscaling groups and load balancers for each provider - 4 repos - Documentation to support deployment processes -**Milestone 2: Benchmarking** +### **Milestone 2: Benchmarking** **1/2 Month - 6.25k USD** --- -- Conduct brief study to determine optimal scaling policies for autoscaling groups -- Determine metrics and thresholds to inform what chain size will require syncing optimizations (pre-synced images + CDN) +- Conduct brief study to determine optimal scaling policies for +autoscaling groups +- Determine metrics and thresholds to inform what chain size will +require syncing optimizations (pre-synced images + CDN) **Deliverables:** - Optimization of scaling policies for autoscaling groups -- Report to W3F with findings and discuss next steps to handle syncing optimizations +- Report to W3F with findings and discuss next steps to handle syncing +optimizations -**Milestone 3: Kubernetes Modules** +### **Milestone 3: Kubernetes Modules** **1.5 months - 18.75k USD** --- -- Build kubernetes clusters and associated provider specific manifests to bootstrap the clusters in line with the existing [polkadot-deployer](https://github.com/w3f/polkadot-deployer) tool. +- Build kubernetes clusters and associated provider specific manifests +to bootstrap the clusters in line with the existing +[polkadot-deployer](https://github.com/w3f/polkadot-deployer) tool. - Build API gateway and service proxy for intelligent routing and caching layers -- Build microservice templates from which Insight data engineering fellows can build pre-indexed endpoints off of +- Build microservice templates from which Insight data engineering +fellows can build pre-indexed endpoints off of - ETL pipelines for supporting pre-indexed queries **Deliverables:** -- Kubernetes cluster modules and manifests for each provider - 4 infrastructure repos -- API gateway (likely Ambassador) and service proxy (likely Envoy) along with associated helm charts to build service mesh and supporting observability services - 1 repo or contributions to existing [polkadot-charts](https://github.com/w3f/polkadot-charts) repo +- Kubernetes cluster modules and manifests for each provider - 4 +infrastructure repos +- API gateway (likely Ambassador) and service proxy (likely Envoy) +along with associated helm charts to build service mesh and supporting +observability services - 1 repo or contributions to existing +[polkadot-charts](https://github.com/w3f/polkadot-charts) repo - Microservice cookiecutter templates - Flask (Python) - 1 repo - FastAPI (Python) - 1 repo -- Airflow ETL tools for batch processing - Contribute to [github.com/blockchain-etl/polkadot-etl](https://github.com/blockchain-etl/polkadot-etl) +- Airflow ETL tools for batch processing - Contribute to +[github.com/blockchain-etl/polkadot-etl](https://github.com/blockchain-etl/polkadot-etl) - Documentation to support deployment process ### **Long Term Plans** --- -- Comprehensive library of IaC modules that can be used and repurposed in a variety of contexts +- Comprehensive library of IaC modules that can be used and repurposed +in a variety of contexts - CLI tooling for deploying modular sets of self-healing infrastructure - Microservices library to expose pre-indexed queries - Additional versions of the API to reference different parachains ## **Additional Information** -While it is important for decentralized network operations to avoid homogeneity, certain components that have a low impact on the diversity of the network are prime targets for standardization. We feel that as the ecosystem grows, having a battle-tested set of reusable modules will prove to be very valuable for best practice to be shared around the community. - -CLI's are useful for making the deployments easily reconfigurable but can become quite opinionated when building in how optionality is expressed. Right now we leverage [cookiecutter](https://github.com/cookiecutter/cookiecutter) for both exposing a simple CLI for rendering scaffolding configs but also in how we plan on giving our Insight fellows starting points for their projects. Because we want to make sure our Insight fellows have the necessary boilerplate code to begin their projects, we are building libraries of starter packages to facilitate the productionization of their projects. - -This project will get us to the doorstep of being building out various microservice endpoints that will make use of different types of caching and persistence layers. These types of endpoints make for fantastic Insight data engineering projects which we hope to recruit a number of fellows to take on during the session starting in June 2020 and beyond. Time permitting, we will be building boilerplate architectures to support both real-time and batch queries against a number of backends for different types of microservices. A month before this particular grant comes to an end, we will be soliciting feedback to inform what endpoints are most interesting to the foundation. - -What we are mainly focused on for now is creating a open framework from which additional APIs can be built. As new parachains come online, we hope to create additional endpoints to support their specialized requests. \ No newline at end of file +While it is important for decentralized network operations to avoid +homogeneity, certain components that have a low impact on the +diversity of the network are prime targets for standardization. We +feel that as the ecosystem grows, having a battle-tested set of +reusable modules will prove to be very valuable for best practice to +be shared around the community. + +CLI's are useful for making the deployments easily reconfigurable but +can become quite opinionated when building in how optionality is +expressed. Right now we leverage +[cookiecutter](https://github.com/cookiecutter/cookiecutter) for both +exposing a simple CLI for rendering scaffolding configs but also in +how we plan on giving our Insight fellows starting points for their +projects. Because we want to make sure our Insight fellows have the +necessary boilerplate code to begin their projects, we are building +libraries of starter packages to facilitate the productionization of +their projects. + +This project will get us to the doorstep of being building out various +microservice endpoints that will make use of different types of +caching and persistence layers. These types of endpoints make for +fantastic Insight data engineering projects which we hope to recruit a +number of fellows to take on during the session starting in June 2020 +and beyond. Time permitting, we will be building boilerplate +architectures to support both real-time and batch queries against a +number of backends for different types of microservices. A month +before this particular grant comes to an end, we will be soliciting +feedback to inform what endpoints are most interesting to the +foundation. + +What we are mainly focused on for now is creating a open framework +from which additional APIs can be built. As new parachains come +online, we hope to create additional endpoints to support their +specialized requests. \ No newline at end of file From 511e9346bee9846c3f52896dd336b94b29fc4357 Mon Sep 17 00:00:00 2001 From: rob Date: Mon, 16 Mar 2020 12:48:50 -0700 Subject: [PATCH 6/6] reducing scope and ask --- grants/speculative/load_balanced_endpoints.md | 37 ++++++++++++------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/grants/speculative/load_balanced_endpoints.md b/grants/speculative/load_balanced_endpoints.md index 3d61dfb06..a68ad2a75 100644 --- a/grants/speculative/load_balanced_endpoints.md +++ b/grants/speculative/load_balanced_endpoints.md @@ -78,16 +78,21 @@ empower them to excel at their engineering initiatives. ## Team Code Repos -**All repos WIP / examples of deployment strategy** -- [https://github.com/insight-infrastructure/terragrunt-polkadot](https://github.com/insight-infrastructure/terragrunt-polkadot) -- [https://github.com/insight-infrastructure/terraform-polkadot-aws-network](https://github.com/insight-infrastructure/terraform-polkadot-aws-network) -- [https://github.com/insight-infrastructure/terraform-polkadot-aws-sentry-node](https://github.com/insight-infrastructure/terraform-polkadot-aws-sentry-node) -- [https://github.com/insight-infrastructure/terraform-polkadot-gcp-network](https://github.com/insight-infrastructure/terraform-polkadot-gcp-network) +**Current list of project related repos - WIP** +- [terragrunt-polkadot](https://github.com/insight-infrastructure/terragrunt-polkadot) +- [terraform-polkadot-aws-network](https://github.com/insight-infrastructure/terraform-polkadot-aws-network) +- [terraform-polkadot-aws-sentry-node](https://github.com/insight-infrastructure/terraform-polkadot-aws-sentry-node) +- [terraform-polkadot-gcp-network](https://github.com/insight-infrastructure/terraform-polkadot-gcp-network) +- [terraform-polkadot-gcp-asg-node](https://github.com/insight-infrastructure/terraform-polkadot-gcp-asg-node) +- [terraform-polkadot-do-network](https://github.com/insight-infrastructure/terraform-polkadot-do-network) +- [terraform-polkadot-azure-network](https://github.com/insight-infrastructure/terraform-polkadot-azure-network) +- [terraform-polkadot-azure-asg-node](https://github.com/insight-infrastructure/terraform-polkadot-azure-asg-node) ## Team LinkedIn Profiles - [https://www.linkedin.com/in/mitchellpkt/](https://www.linkedin.com/in/mitchellpkt/) - [https://www.linkedin.com/in/rob-cannon-21571317/](https://www.linkedin.com/in/rob-cannon-21571317/) +- [https://www.linkedin.com/in/richardmah/](https://www.linkedin.com/in/richardmah/) ## Development Roadmap @@ -135,7 +140,7 @@ of benchmarks. ### **Milestone 1: IaC Modules** -**1 months - 12.5k USD** +**1 Month - 10k USD + 24 b-dots** --- @@ -156,7 +161,7 @@ packaged as reusable modules. ### **Milestone 2: Benchmarking** -**1/2 Month - 6.25k USD** +**1/2 Month - 5k USD + 12 b-dots** --- @@ -173,7 +178,7 @@ optimizations ### **Milestone 3: Kubernetes Modules** -**1.5 months - 18.75k USD** +**1 Month - 9k USD + 24 b-dots** --- @@ -181,9 +186,6 @@ optimizations to bootstrap the clusters in line with the existing [polkadot-deployer](https://github.com/w3f/polkadot-deployer) tool. - Build API gateway and service proxy for intelligent routing and caching layers -- Build microservice templates from which Insight data engineering -fellows can build pre-indexed endpoints off of -- ETL pipelines for supporting pre-indexed queries **Deliverables:** @@ -193,12 +195,19 @@ infrastructure repos along with associated helm charts to build service mesh and supporting observability services - 1 repo or contributions to existing [polkadot-charts](https://github.com/w3f/polkadot-charts) repo +- Documentation to support deployment process + +### **Reach Goals: Microservice Templates** + +- Build microservice templates from which Insight data engineering fellows can build specialized endpoints from +- Recruit 10 Insight data engineering fellows to build additional endpoints and data pipelines and 3 dev ops fellows for productionizing and packaging of endpoints + +**Deliverables:** + +- Airflow ETL tools for batch processing - Contribute to [github.com/blockchain-etl/polkadot-etl](https://github.com/blockchain-etl/polkadot-etl) - Microservice cookiecutter templates - Flask (Python) - 1 repo - FastAPI (Python) - 1 repo -- Airflow ETL tools for batch processing - Contribute to -[github.com/blockchain-etl/polkadot-etl](https://github.com/blockchain-etl/polkadot-etl) -- Documentation to support deployment process ### **Long Term Plans**