Designing Airframe 2025.1 for the Scala 3 Era

[xerial]

With most modules in Airframe 2024 now stable, it's time to consider transitioning to Scala 3-only releases. This shift will improve maintainability and allow us to deprecate or redesign certain features.

Goal and Design

• Tailor Airframe specifically for Scala 3 and later versions, discontinuing Scala 2 support.
• Introduce airframe-core, which consolidates wvlet.airframe.xxx packages as a Scala 3-only module. For minimizing the pain of migration, other than wvlet.log.LogSupport, the same packages will be used, which means airframe-core cannot be used with legacy modules.
• Migration strategy:
  • Maintain the current Airframe for Scala 2 until at least 2026.
  • Gradually migrate and redesign modules into airframe-core.
    Once the airframe-core is production-ready, relocate the Scala 2 cross-build code to a separate repository or branch and transition it to maintenance mode.

Background

• Airframe's value lies in enhancing Scala's usability by offering essential building blocks for applications. Its primary goal is to provide a convenient extension for Scala developers.
• Scala 3 is ready for production use.
• SBT 2 (beta) supports Scala 3, eliminating the need to maintain Scala 2 releases once SBT 2 is officially released
  • Except Spark applications, which still need to use Scala 2 for a while.
• Scala 2/Scala 3 cross-buids are partially blocking adding Scala 3 specific features
  • surface surface (fix): Fix #3409  #3412
  • airframe-finagle will not support Scala 3 as Finagle in OSS is maintenance mode.
  • Can’t use Scala 3’s new syntaxes in the library
• Airframe also have several legacy design that should be refined

Sunsetting Less Used Modules and Features

• airframe-config, and its DI integration
• airframe-di’s in-trait binding
• airframe-http-recorder ← at least using SQLite for recording request was not a good idea for maintainability. Should we use JSON for recording?
• airframe-http-okhttp
• airframe-jdbc ← reading JDBC result is managed by airframe-codec
• airframe-sql, while this SQL parser is useful, once it’s used outside, it becomes hard to change the interface design of model classes.
• airframe-grpc

New Features

• New versioning scheme: YYYY.(milestone month).xx so that we can tell what are major changes
• Universal CLI builder (airframe-launcher → airframe-cli)
• Unified data pack/unpack interface
• Common Scanner/Parser base implementation

Unifying Core Modules

As the most of Airframe modules have almost no dependency, we can merge several packages into a single module. For example, log, di, control, codec, json, msgpack, ulid, jmx, launcher, metrics, rx, surface can be packaged in a single module. Airspec, a testing framework, also depends of part of the module is embedding them.

The module dependency will be simplified like:

airframe-core (← airframe-core-macros as macro code needs to be in a different compilation unit) 
  → airframe-http 
    → sbt-airframe-http
    → airframe-http-netty
  → airspec

Airframe DI’s True Value is not DI

Airframe-di began as a Scala redesign of Google's Guice, but we've since discovered its primary benefits lie elsewhere. The framework excels at automatically constructing classes from a session and providing integrated resource management. While dependency injection in Java often evokes memories of overly complex syntax, Airframe-di has resolved many of these issues. However, explaining its advantages remains challenging, as only experts fully grasp the nuances of various DI frameworks. https://wvlet.org/airframe/docs/comparison.html

https://blog.3qe.us/entry/2024/04/16/221847

Minor improvement:

• The new bindInstance[X] and bindProvider[D, X] syntax eliminates the need for global "import wvlet.airframe.*". Use the new syntax only in the new version.

Airframe Codec can focus on encoding small data

Airframe-codec was initially designed for handling large volumes of data. However, real-world use cases have shown a greater need for small-footprint encoding (e.g., HTTP RPC request/response encoding, JSON conversion). In practice, convenient methods for error handling, validation, and format conversions (JSON, MessagePack, JDBC result) have proven more valuable. We've also recognized the necessity for a standard approach to packing and unpacking binary data (e.g., packInt, packFloat, unpackXXX) beyond primitive classes.

Our original goal was to design a format switch (row-oriented/columnar) and provide a unified interface for querying universal format data. However, this idea hasn't come to fruition. So some redesign would be necessary to address these needs, especially for managing tabular data.

Human Readable Logging

airframe-log (wvlet.log) was renamed from wvlet-log for the ease of maintenance, but the package is still under wvlet.log and depends on java.util.logging. While java.util.logging helps organizing hierarchical logging, but other libraries (e.g., slf4j-log4j, Trino) often touches java.util.logging configurations and disturbs logging. Having an isolated logging would be more easy to use.

Logging for Historical Record Logging

Collecting small logs (e.g., JSON) and creating their columnar archives (e.g., Parquet) needs to be more convenient as querying columnar data is getting easier via DuckDB.

Airframe HTTP

• @rpc annotation is unnecessary as we find paths from RxRouterProvider in Scala 3
• Some legacy design e.g., HttpRequest[F, Req, Resp] found to be unnecessary