Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sample App: Todolist #46

Merged
merged 103 commits into from
Mar 10, 2020
Merged

Sample App: Todolist #46

merged 103 commits into from
Mar 10, 2020

Conversation

sgammon
Copy link
Owner

@sgammon sgammon commented Feb 23, 2020

UPDATE: Stuff that depends on this sample is needed downstream, so this is getting merged early as alpha2 and kicked down the road.

This changeset adds a new sample app, which ties together all the features thus far and adds some new ones to facilitate easy app development. I'm using this sample to help define the underlying framework structure, when used with Micronaut/Soy.

The sample is built with unified modelling via Protobuf, and RPC dispatch via gRPC. The UI is built with Soy, with SSR support via Soy/Java, and re-hydrated CSR support via Soy/JS (and idom).

The app's data backend is powered by Firestore, with objects mediated for serialization by the framework. This process is also driven by the protos. Auth is also powered by Firebase, which enables easy data permissions.

Via the central API definition, the app can automatically generate an API console, docs, and Open API/Swagger configs. It is also dispatchable from nearly any platform (via either gRPC or REST). Endpoints served in this manner support either Protobuf on the wire, or JSON (when using REST).

Styles are defined in SASS but processed by both GSS and PostCSS, which enables both (1) tight optimization on a per-browser basis, and (2) style modularity when serving. The server can also rewrite styles using a Soy/GSS rewrite map.

The sample additionally demonstrates testing, with coverage (where supported by the framework, so far).

Related and blocking issues:

  • App Framework (#27)
    • BaseController / AppController
    • PageContext / PageContextManager
    • Some way to mount and catalog assets note: kicked
  • MVP Functionality Matrix (#43)
  • Source-aware Server Targets note: kicked
  • Modular JS serving, by entrypoint note: kicked
  • Modular CSS serving, by entrypoint note: kicked
  • Early page render context/frame template
  • Basic asset templates/common templates
  • Unable to natively compile complex Soy templates (filed as #35)

The following checklist tracks progress towards the above:

  • Data Adapters
    • Initial adapter: in-memory
    • First implementation: Firestore note: kicked
  • Basic gRPC Support
    • Ability to build/inject/run a gRPC server
    • Some way to smoothly serve/proxy this
    • Some plan for Envoy support
    • TLS vs. ALTS? note: kicked
    • Ability to generate a gRPC client note: kicked
  • Frontend serving logic (more coming soon) note: kicked
  • Basic Todolist app
    • Unified Model: services and objects
      • Object definitions
      • Service definition (Tasks)
      • Service configuration
        • Auth (API keys, Firebase auth)
        • Documentation config
        • Quotas and properties
    • Server-side
      • HomeController: serve anonymous app frame
        • Basic implementation
        • Login UI kickoff (via Firebase Auth UI / MDL) note: kicked
        • Anonymous app container note: kicked
      • TodolistController: serve logged-in app frame note: kicked
        • Basic implementation note: kicked
        • Logout UI kickoff and handler note: kicked
      • Auth/session lifecycle note: kicked
      • Rejoiner support note: kicked
      • GraphQL schema/support note: kicked
    • Web app note: kicked
      • gRPC invocation logic note: kicked
      • Initial style structure note: kicked
      • Java-based app implementation note: kicked
  • Application-quality tooling
    • Support for hermetic GCB builds
    • Support for deployment directly from GCB
    • Production routing (cluster-internal and ILB)
    • Coverage reporting from CI (Buildkite)
    • Source quality / linting in CI (Buildkite)
    • Test result reporting from CI (Buildkite)
  • Issues to file note: kicked
    • Future: Telemetry note: kicked
    • Future: iOS App note: kicked
    • Future: Android App note: kicked
    • Future: Chrome Extension note: kicked

@sgammon sgammon added enhancement New feature or request samples Issues dealing with sample apps. labels Feb 23, 2020
@sgammon sgammon added this to the 🚀 Beta milestone Feb 23, 2020
@sgammon sgammon self-assigned this Feb 23, 2020
@codecov
Copy link

codecov bot commented Feb 23, 2020

Codecov Report

Merging #46 into feature/testsuite will decrease coverage by 0.1%.
The diff coverage is 99.89%.

Impacted file tree graph

@@                  Coverage Diff                  @@
##           feature/testsuite      #46      +/-   ##
=====================================================
- Coverage                100%   99.89%   -0.11%     
=====================================================
  Files                      1       33      +32     
  Lines                      9      957     +948     
=====================================================
+ Hits                       9      956     +947     
- Misses                     0        1       +1
Flag Coverage Δ
#backend 99.89% <99.89%> (-0.11%) ⬇️
#jvm 99.89% <99.89%> (-0.11%) ⬇️
Impacted Files Coverage Δ
java/gust/backend/model/CollapsedMessage.java 0% <0%> (ø)
java/gust/backend/ApplicationBoot.java 100% <100%> (ø)
java/gust/backend/model/FetchOptions.java 100% <100%> (ø)
...a/gust/backend/driver/inmemory/InMemoryDriver.java 100% <100%> (ø)
java/gust/backend/runtime/ReactiveFuture.java 100% <100%> (ø)
java/gust/backend/model/ModelDeflateException.java 100% <100%> (ø)
java/gust/backend/model/ProtoModelCodec.java 100% <100%> (ø)
java/gust/backend/model/CacheDriver.java 100% <100%> (ø)
java/gust/backend/model/ModelWriteConflict.java 100% <100%> (ø)
java/gust/backend/model/MissingAnnotatedField.java 100% <100%> (ø)
... and 55 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ba341b...133d216. Read the comment docs.

@sgammon sgammon force-pushed the sample/todolist branch 9 times, most recently from 0666e27 to a8337a3 Compare February 23, 2020 09:24
This changeset adds a new sample app, which ties together all the
features thus far and adds some new ones to facilitate easy app
development. I'm using this sample to help define the underlying
framework structure, when used with Micronaut/Soy.

The sample is built with unified modelling via Protobuf, and RPC
dispatch via gRPC. The UI is built with Soy, with SSR support via
Soy/Java, and re-hydrated CSR support via Soy/JS (and `idom`).

The app's data backend is powered by Firestore, with objects
mediated for serialization by the framework. This process is also
driven by the protos. Auth is also powered by Firebase, which
enables easy data permissions.

Via the central API definition, the app can automatically generate
an API console, docs, and Open API/Swagger configs. It is also
dispatchable from nearly any platform (via either gRPC or REST).
Endpoints served in this manner support either Protobuf on the wire,
or JSON (when using REST).

Styles are defined in SASS but processed by both GSS and PostCSS,
which enables both (1) tight optimization on a per-browser basis,
and (2) style modularity when serving. The server can also rewrite
styles using a Soy/GSS rewrite map.

The sample additionally demonstrates testing, with coverage (where
supported by the framework, so far).

Related and blocking issues:
- [ ] Sample Apps (#26)
- [ ] App Framework (#27)
- [ ] MVP Functionality Matrix (#43)
- [ ] Source-aware Server Targets
- [ ] Modular JS serving, by entrypoint
- [ ] Modular CSS serving, by entrypoint
- [x] Early page render context/frame template
- [x] Basic asset templates/common templates
- [ ] Unable to natively compile complex Soy templates

The following checklist tracks progress towards the above:
- [ ] Data Adapters
  - [ ] Initial adapter: in-memory
  - [ ] First implementation: Firestore
- [ ] RBE/RCE Fixes
  - [x] Remove Python (for now)
  - [ ] Restore Python support w/fix for CI
- [ ] Basic gRPC Support
  - [x] Ability to build/inject/run a gRPC server
  - [ ] Some way to smoothly serve/proxy this
  - [ ] Some plan for Envoy support
  - [ ] TLS vs. ATLS?
  - [ ] Ability to generate a gRPC client
- [ ] Frontend serving logic (more coming soon)
- [ ] Basic *Todolist* app
  - [x] Unified Model: services and objects
    - [x] Object definitions
    - [x] Service definition (`Tasks`)
    - [x] Service configuration
      - [x] Auth (API keys, Firebase auth)
      - [x] Documentation config
      - [x] Quotas and properties
  - [ ] Server-side
    - [ ] `HomeController`: serve anonymous app frame
      - [ ] Basic implementation
      - [ ] Login UI kickoff (via Firebase Auth UI / MDL)
      - [ ] Anonymous app container
    - [ ] `AppController`: serve logged-in app frame
      - [ ] Basic implementation
      - [ ] Logout UI kickoff and handler
    - [ ] Auth/session lifecycle
    - [ ] Rejoiner support
    - [ ] GraphQL schema/support
  - [ ] Web app
    - [ ] gRPC invocation logic
    - [ ] Initial style structure
    - [ ] Java-based app implementation
- [ ] Issues to file
  - [ ] Future: Telemetry
  - [ ] Future: iOS App
  - [ ] Future: Android App
  - [ ] Future: Chrome Extension
- Gather proto and other page context into `SoyContextMediator`
- Override Micronaut Views/Views Soy deps
  (see micronaut-projects/micronaut-views#34)
- Update Soy lib and jssrc deps
- Fix CSP error specified by default in `application.yml`
- Serve HTML from controller
- Add better exclusions for Google coordinates
- Add source dependencies for Protobuf and gRPC
- Upgrade/conform dependency tree to Protobuf/gRPC latest
- Add support for reflection service
- Make service / interceptor simple passthroughs
- Run app on :8081
- Run API on :8082
sgammon and others added 3 commits March 1, 2020 00:23
* Feature: Model Adapters

This changeset introduces backend logic to deal with business data
(i.e. *schema-driven models*). As part of this work, I'm also
building out underlying stuff needed for that, including managed
transport, generic model adapters/codecs, and so on.

Related issues:
- App Framework (#27)
- MVP Functionality Matrix (#43)

In this PR:
- [x] **Backend logic: Models**
  - [x] Interfaces for persistence managers, drivers, codecs
  - [x] Serializer and deserializer interfaces
  - [x] Cache-related manager interfaces
  - [x] Container for encoded models
  - [x] Codec for encoding models in proto formats
  - [x] Encoded (generic) model
    - [x] Serialization to `CollapsedModel`/`OutputStream`/`bytes`
    - [x] Deserialization from `InputStream`/`bytes`
  - [x] Interface support for simple CRUD operations
    - [x] `retrieve`: Fetch a structured record from storage.
    - [x] `persist`: Write a structured record to storage.
    - [x] `delete`: Erase a structured record from storage.
  - [x] Cache driver support for basic operations
    - [x] `put`: Write a model to a cache
    - [x] `fetch`: Retrieve a model from the cache
    - [x] `evict`: Evict one or more cached models, by key
    - [x] `flush`: Drop all keys from the cache
- [x] **Backend logic: Transport**
  - [x] Simple `TransportManager` interface
  - [x] Managed gRPC channels for Google APIs
    - [x] `GoogleTransportManager`: channel management
    - [x] Ability to refresh channels?
  - [x] Service support list
    - [x] Firestore
- [x] **Backend logic: Built-in Drivers**
  - [x] In-memory Adapter (reference)
    - [x] Initial `InmemoryManager` implementation
    - [x] Initial `InmemoryDriver` implementation
    - [x] Initial `InmemoryCache` implementation
- [x] Test suite
  - [x] Simple model interactions
  - [x] Update/create methods
  - [x] Delete methods
  - [x] Cache operations
@sgammon sgammon merged commit 131f3a8 into feature/testsuite Mar 10, 2020
sgammon pushed a commit that referenced this pull request Mar 10, 2020
* Feature: Testsuite

This changeset adds a substantive and useful testsuite, with tests
split based on coverage support and backend/frontend disposition.
Coverage support is included in this changeset (at least for tests
in languages that support it, which, for now, is Java).

* Additional coverage reporting tools

- Report Java coverage (properly tagged) to Codecov
- Run coverage reporter in CI
- Small tweaks to VERBOSE and QUIET in Makefile

* Update rules_graal with support for GraalVM 20.0.0

* Apply new GraalVM version

* Initial work/structure on backend Node support

* Yarn dependency upgrades

* Fix frontend tests

* Update J2CL, rules_closure

* Fixes for upcoming dep updates

* Update all deps

* Remove H5BP

* Update Maven deps

* Fix import paths in tests/style and test/soy

* Standardize on io_bazel_rules_sass for rules_sass

* Fix path to karma_config.js

* Fix import path for dom test

* Fix sample app target build

* Remove superfluous wait in CI

* Basic ability to build and run a NodeJS app

* Refactor build of samples

* Replace superfluous sample image build

* Change TS sample URL to root

* Working NodeJS testsuite (using Jasmine)

- Ability to execute backend JS tests
- No support for coverage yet
- Support for test reporting via ReportCI

* Add logic to merge and report test results

- Merge test results via custom Python script
- Report via ReportCI

* Add report generator (HTML) to tool requirements

* Generate HTML test report during test transmit

* Enable ReportCI

* Fix reporting path for CI

* Switch to latest upload script

* Simple JS (frontend) coverage reporter

* Fix platform-agnostic outpath

* Additional outpath fix

* Disable JS coverage report (for now)

* Fix Makefile syntax error

* Add coverage/test reports as build artifacts

* Serve just the index

* Export coverage tarball

* Update protobuf -> 3.11.4

* Upgrade Bazel > 2.1.1

* Add api-common submodule and dependency

* Enable ability to declare internal protos

* Refactor/cleanup datamodel core

* Small nit fixes

* Add BZL overlay for proto_common

* Sample App: Todolist (#46)

* Sample App: Todolist

This changeset adds a new sample app, which ties together all the
features thus far and adds some new ones to facilitate easy app
development. I'm using this sample to help define the underlying
framework structure, when used with Micronaut/Soy.

The sample is built with unified modelling via Protobuf, and RPC
dispatch via gRPC. The UI is built with Soy, with SSR support via
Soy/Java, and re-hydrated CSR support via Soy/JS (and `idom`).

The app's data backend is powered by Firestore, with objects
mediated for serialization by the framework. This process is also
driven by the protos. Auth is also powered by Firebase, which
enables easy data permissions.

Via the central API definition, the app can automatically generate
an API console, docs, and Open API/Swagger configs. It is also
dispatchable from nearly any platform (via either gRPC or REST).
Endpoints served in this manner support either Protobuf on the wire,
or JSON (when using REST).

Styles are defined in SASS but processed by both GSS and PostCSS,
which enables both (1) tight optimization on a per-browser basis,
and (2) style modularity when serving. The server can also rewrite
styles using a Soy/GSS rewrite map.

The sample additionally demonstrates testing, with coverage (where
supported by the framework, so far).

* Add simple tests for Soy renaming maps

* Cleanup cloudbuild targets

* Switch tests off in GCB

* Cleanup and testing for existing core.backend logic

* Fix definitions for JS interop

* Early `PageContext` logic and example

- Gather proto and other page context into `SoyContextMediator`
- Override Micronaut Views/Views Soy deps
  (see micronaut-projects/micronaut-views#34)
- Update Soy lib and jssrc deps
- Fix CSP error specified by default in `application.yml`
- Serve HTML from controller

* Refactor Java dependencies and rules

- Add better exclusions for Google coordinates
- Add source dependencies for Protobuf and gRPC
- Upgrade/conform dependency tree to Protobuf/gRPC latest

* Seal Java dependencies

* Further gRPC support in Todolist

- Add support for reflection service
- Make service / interceptor simple passthroughs

* New todolist port arrangement

- Run app on :8081
- Run API on :8082

* Add Todolist sample to cloudbuild config

* Eliminate remote build in GCB

* Adjust build stamp to fallback to COMMIT_SHA variable

* Apply fix for image tag in GCB

* Add K8S rules dependency

* Further adjustments to GCB config

* Fix misplaced deploy jar target

* Pull and tag latest todolist image on build

* Add initial ambassador config

* Remove failing build artifact upload in GCB

* Add initial test for PageContext

* Improvements to page context integration

- Add extended `SoyProtoContextMediator` interface
- Add efficient empty context use
- Cover `PageContext` with tests

* Adjust coverage reporting

* Refactor `HomeController` to use new `AppController`

- Add `PageContextManager` and factory
- Init `AppController` and `BaseController`
- Adjust Bazel flags to be normalized

* Cleanup and simplify PageContext/PageContextManager

- Remove embedded PageContextManager.Factory
- Clean up DI flow
- Fix issue injecting HTTP request

* Initial Todolist infrastructure configs

- Add full suite of K8S configs, scoped to `todolist` namespace
- Add customized Envoy configuration and image

* Toolchain support for Docker, Kubernetes and packaging

- Add macros for generic Docker images
- Add macros for Kubernetes configs
- Add macros for packaging

* Pull in Envoy base image, setup K9 defaults

* Add rules_pkg dependency

* Add yarn run alias for gRPC CLI

* Converge bazelrc and WORKSPACE for Todolist

* Major work on Envoy proxy

* Use manually-built Envoy image, for now

* Fix spacing issue in bazelrc

* Add external LB for testing

* Use fully-qualified service DNS

* Setup devops flow for Envoy and Todolist (including deploy)

- Add deploy routine to GCB (CI will likely fail)
- Enable reflection service (+add logging)
- Tweaks to Envoy config, including healthcheck logging

* Fixes for ingress configuration

* Transition to independent NEGs

* [skip ci] Change to Bazel 2.1.0 in GCB

* Prep version bump -> 1.0.0-alpha2

* Fix incorrect Bazel version in GCB

* Fix prefix for Bazel images in GCB (late steps)

* Pin and build K8S toolchain from source

* Attempt fix for K8S transitive load

* Upgrade K8S > 1.15.9

* Update K8S hash, add dep on infrastructure tools

* Switchup alias for K8S repo infra

* Apply incompatible trip flags to fix K8S in GCB

* Alias K8S repo

* Clean up K8S tools reference

* Add custom builder image for deploy, use it

* Switch up GCB image for 2nd-to-last build step

* Upgrade GCB image -> v1b

* Shift build steps to GCB image

* Add Alpine version of builder image

* Prep CI image for use

* Back to Ubuntu builder, install OpenJDK

* Add git to GCB environment

* Auto-remove packages in builder image

* Upgrade to builder v1d

* Add unzip to build image :eyeroll:

* Version bump for build image -> v1e

* Add regular Python package to GCB

* Version bump for GCB -> v1f

* Upgrade GCB image -> v1g (add symlink to gcloud)

* Fix Python rules in CI

* Add todolist internal gateway

* Fix pip issues in CI

* Remove external LB for Todolist

* Always use latest GCB builder

* Cleanup Python deps, add pypandoc to build env

* Support for ibazel-based development flow

- Call ibazel from Makefile when `m dev` is used
- Tweak flags and invocation to work
- Flush out issue with Soy render context

* Update Micronaut/Soy packages with context fix

* Fix access to ADC

* Add ibazel_live_reload to Java app targets

* Support Java-side live reload

* Re-enable test reporting, add Travis env vars

* Stop upgrading deps in CI image

* Add Makefile task to build builder image

* Major README and licensing cleanup/improvements

* Refactor app boot logic

* Cleanup after refactor, fix various issues reported by Codacy

* Cleanup startup exception flow

* Add GraphQL to readme

* Root folder cleanup

* Add new badges to README

* Tweaks to README

* Remove .develop aliases

* Adjust coverage config to be lenient until it settles

* Remove Makefile aliases

* Fix Makefile command list in README

* Small nits

* Fix spaces-in-tags in Soy templates

* Add interface to set page title

* Remove old page root

* Install logback as the underlying log implementation

* Add sugar for page context, title, assets

- Add ability to change page title, inject stylesheets/scripts
- Add ability to easily generate trusted URI protos
- Rename SoyProtoContextMediator > PageRender
- Make PageContextManager usable as a PageRender
- Use new fluent interface in HomeController

* 🚀 Fix Soy render with injected context

- Working injected context. Yay!
- Logback is working too
- Cleaned up page context manager with improved log messages

* Fix render bugs on homepage

* Remove empty table in README

* Feature: Model Layer (#65)

* Feature: Model Adapters

This changeset introduces backend logic to deal with business data
(i.e. *schema-driven models*). As part of this work, I'm also
building out underlying stuff needed for that, including managed
transport, generic model adapters/codecs, and so on.

Related issues:
- App Framework (#27)
- MVP Functionality Matrix (#43)

In this PR:
- [x] **Backend logic: Models**
  - [x] Interfaces for persistence managers, drivers, codecs
  - [x] Serializer and deserializer interfaces
  - [x] Cache-related manager interfaces
  - [x] Container for encoded models
  - [x] Codec for encoding models in proto formats
  - [x] Encoded (generic) model
    - [x] Serialization to `CollapsedModel`/`OutputStream`/`bytes`
    - [x] Deserialization from `InputStream`/`bytes`
  - [x] Interface support for simple CRUD operations
    - [x] `retrieve`: Fetch a structured record from storage.
    - [x] `persist`: Write a structured record to storage.
    - [x] `delete`: Erase a structured record from storage.
  - [x] Cache driver support for basic operations
    - [x] `put`: Write a model to a cache
    - [x] `fetch`: Retrieve a model from the cache
    - [x] `evict`: Evict one or more cached models, by key
    - [x] `flush`: Drop all keys from the cache
- [x] **Backend logic: Transport**
  - [x] Simple `TransportManager` interface
  - [x] Managed gRPC channels for Google APIs
    - [x] `GoogleTransportManager`: channel management
    - [x] Ability to refresh channels?
  - [x] Service support list
    - [x] Firestore
- [x] **Backend logic: Built-in Drivers**
  - [x] In-memory Adapter (reference)
    - [x] Initial `InmemoryManager` implementation
    - [x] Initial `InmemoryDriver` implementation
    - [x] Initial `InmemoryCache` implementation
- [x] Test suite
  - [x] Simple model interactions
  - [x] Update/create methods
  - [x] Delete methods
  - [x] Cache operations

* Warning fixes all around
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request samples Issues dealing with sample apps.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant