Skip to content

Latest commit

 

History

History
233 lines (199 loc) · 11.6 KB

TODO.md

File metadata and controls

233 lines (199 loc) · 11.6 KB

TODOS

Release 1.0.0 - Basic match endpoint, error handling, testing, logging and documentation

Additions

Error Handling

  • WIP Implement good error handling
    • use crates error types for better Error handling e.g. reqwest::Error
    • use thiserror in library part
    • use eyre consistently for results with reports in binary part
    • Q use .map_err (e.g. return HTTP status codes, other errors)
      • A We should return more warnings/errors in the error message field for frontend error handling
    • Handle errors that got bubbled up to the MatchInfoProcessor gracefully and return a maximum of valuable information on the MatchInfo and the errors to the client
      • Send log entry to Client for better error handling on client-side
      • On hard error, no match_info but instead error status code (HTTP)
    • handle serde_json::Value::Null errors better when parsing data from aoe2.net
  • implement todo!()s
  • don't overwrite aoc_ref_data if not able to parse it in thread, so we have at least one working version
  • collect all SoftFail errors within each request to our API, collect them at the end in MatchInfoProcessor and write them back to error_message in MatchInfoResult
  • Add assertions to http requests: https://github.com/beltram/asserhttp

Testing

  • implement functionality to download a specific match via /api/match for usage in test cases
    • create a command-line parameter for export-test-data example to run it with a specific matchid/match-uuid (UUID is probably preferable because of validation with uuid crate)
    • then run all other functions to download the resources need to answer with that specific request
    • rebuild and export the MatchInfoRequest for a random player from it for test case usage of our normal functionality
  • implement useful tests/raise test-coverage to a (valuable) maximum
    • put unit tests into the same file of the type they refer to
    • use claim for tests
  • Special cases done right? (talk through them together)
    • Implement (integration) test cases for these
    • Data structure does not match with data from aoe2net
      • Q: take a look for a serde attribute to mark fields in structs that are not as important for our processing, so we don't throw a parsing error if non-essential fields don't match/exist
      • A: We only parse Players of last_match into some losely-typed datastructure for easier handling, the rest is serde_json::Value and parsing on the run
    • last_match == 404
    • New players without ranking (as deranked, but less than 10 games played)
    • Deranked players (have an empty leaderboard entry)
      • then we should use ratinghistory for the last data and rank=DR
    • Coop games
    • Game Type except RM (0) and DM (2)
    • FFA with teams set to ’-1’
  • Write functionality to save a set of JSON responses (also our own) to a file to use them inside the integration tests and be able to update frequently
    • Parse requests and use wiremock for HTTP-mocking and test requests made by the api_handler
    • Compare our parsed initial response (manually checked) with the one in memory from the offline data
  • Use https://crates.io/crates/walkdir for walking test directory tree
  • Use https://crates.io/crates/automod for adding (regression) tests dynamically

Fixes

  • Fix character escaping in e.g. "name": "\"[RUS-F]GriN\""

  • Make error message more understandable for frontend:

    "GenericResponderError":
"Other ApiRequestError: HTTP-Client experienced an error: error decoding response
body: EOF while parsing a value at line 1 column 0."

    probably from

    http: error: ConnectionError: ('Connection aborted.', RemoteDisconnected('
Remote end closed connection without response')) while doing a GET request 
to URL: http://127.0.0.1:8000/matchinfo?id_type=profile_id&id_number=224786&language=en&game=aoe2de

  • Investigate HTTPie errors for more edge cases

  • in case team == -1 start setting from Team 1/2 not from the back (7/8)

Refactoring

  • Migrate to clap v3 from structopt
  • Parse MatchInfoRequest for export-sample-data and full-integration test from ron file for ease of testing/exporting
    • create struct that contains a MatchInfoRequest and a folder-layout and for other useful/needed information so we can create different test cases easier for integration testing
  • Refactor both, parsing and mock binding logic in full integration test
  • create only new clients for each new api-root not for each request to us
  • Q: how can we make creating requests easier and less boilerplate? (trait objects, etc.)
    • Create API client struct that wraps ApiRequests and ApiResponses
    • Also think about the openAPI parsing and request generating logics for the future
    • parse_into::<T> method for ApiRequest and FileRequest
    • ParsableRequest trait
  • async stuff done right?
  • use https://docs.rs/reqwest/0.11.0/reqwest/struct.Url.html#method.join for base_path and joining files for DS: reqwest::Url
  • structured logging: use tracing crate instead of log and refactor accordingly
  • Use a concurrent hashmap instead of a HashMap: https://crates.io/crates/dashmap
  • (?) Migrate to Axum from warp
  • Use RwLock instead of Mutex for non-blocking aoc-ref-data lookup with multiple readers
  • Check value of https://crates.io/crates/indexmap for the player alias indexing
  • Use cow for less cloning to satisfy the borrow checker
  • check where enums in parameters are more applicable (no stringly typed apis)
  • no self on with_ alternative constructors
  • having a struct for exporting/mocking/maintenance to spare parameters and get get rid of unnecessary boilerplate
  • make use of conversion traits: fn foo<P: Into<PathBuf>>(p: P)
  • use case for enum with str representation?
  • implement FromStr for types? https://doc.rust-lang.org/std/str/trait.FromStr.html
  • no stringly typed function parameters: https://hermanradtke.com/2015/05/06/creating-a-rust-function-that-accepts-string-or-str.html
  • crossbeam-deque use case?
  • parking_lot use for Mutex (faster)
  • impl Into<String> for stringly-typed parameters, because we may not need a String but something that can be turned into one
    • if a conversion can fail we can do it with impl TryInto<String>
  • Use bail! and ensure! macros from anynhow where applicable

Documentation

  • Create good base documentation
  • Add more documentation

Performance

Rust Performance Book

Benchmarking

  • Q: how is our backend reacting to 100+ concurrent API requests?
    • implement benchmark getting all ’MatchInfoResult’s for the To100
    • what architectural changes need to be made to support many clients on our api
  • smoke test with https://github.com/tarekziade/salvo
  • Use bencher for benchmarking features on stable
  • Maybe criterion which is a more sophisticated alternative
  • Can dig deeper after profiling with counts

Flamegraph

  • Use flamegraph. A very simple and portable tool to understand where the time is spent in the application. For more detailed info, try: perf record -g --call-graph=dwarf /path/to/your/application, then load it into Firefox Profiler.

Release 1.1.0 - SUBSCRIPTION requests

Additions

  • GET last_match for aoc-reference-data profile ids in another thread and save the content to a HashMap -> for later subscriptions
    • if a profile id from this list asks for a MatchInfo we can shorten the path/do less requests
    • new subscriptions that are not on aoc-reference-data can be made persistent within a ron file that gets parsed on startup
    • active subscribed profile ids get copied from this HashMap into an ActiveSubs-HashMap where requests are made more frequently to check for changes and send out a `delta
  • Add automatic translation of user facing strings with https://crates.io/crates/libretranslate

Intended Procedure

  • Client: SUBSCRIBE lastmatch(player_id=197751)
  • Server: CREATE Observable
  • Server: SEND initial state to CLIENT
  • Server: POLL AoE2.net / Caching
  • Server: UPDATE Observable
  • Server: ON UPDATE of Observable send PATCH with DELTA to CLIENT
  • Client: UPDATE internal data structure with PATCH

Performance optimisation

Release 1.2.0 - User management and persistence

  • User database (PostgreSQL/MariaDB), user log-in and dashboard
  • Transparency management in user dashboard

Notes