Marten 6 / JasperFx "Grand Unified Theory"

[jeremydmiller]

Blame @oskardudycz for this, he set my head spinning with all this late last week.

Problems & Goals

• Marten has become the dominant project across the entire JasperFx ecosystem
• Jasper itself is languishing, but there's valuable code in there wasting away
• One of the next big features to build out in Marten is outbox integration to external messaging infrastructure. Jasper has a working outbox integration with Marten, but that
  requires using Jasper, which also forces users to choose Lamar as the IoC tool
• Really, one of the next goals of Marten period is to support a plethora of different strategies to stream events captured in Marten to messaging transports

1. Using the async daemon to forward events to some sort of outgoing mechanism with a strong, event ordering guarantee
2. Use an "outbox" from Marten to whatever outgoing mechanism
3. WAL integration from Postgresql to Kafka/Pulsar

To utilize the existing outbox mechanism in Jasper with Marten, we could either:

1. Make Jasper lightweight enough that folks won't mind pulling in Jasper just to get the outbox with Marten
2. Cannibalize Jasper and remove the outbox functionality to a potentially shared library that could be used by a new Marten.Outbox add on or by Jasper itself

At this moment, I think we've got a better shot at making the 2nd approach work sooner rather than later

Potential, Supported Tech Usage Stacks

I think these are the potential combinations of JasperFx and major tooling that other folks would try to use:

1. Marten with any and all other .Net application frameworks
2. Marten + feeds to Rabbit MQ / Pulsar / Kafka / other transports. Could include outbox, WAL integration, or daemon used
   to publish events with a strict ordering guarantee
3. Jasper as a mediator within ASP.Net Core
4. Jasper as an in-memory command bus within .Net applications
5. Jasper + Marten in server side applications
6. Jasper + EF Core with Sql Server or Postgresql
7. Jasper + Rabbit MQ / Pulsar / Other transports

Library Naming Convention Rules (?)

Here's what I want to suggest for library naming conventions going forward.

• Top level, "folks would purposely choose to install this directly" libraries should get a short name either using the existing "close to Jasper landmark" or by picking yet another type of Mustelidae. I think Oskar and I both vote for "Wolverine" for the next new project that's worthy of the name ;-)
• Building block libraries that are probably just used transitively should be named "JasperFx.[name]". For example, I think I'd vote to rename BaselineTypeDiscovery to JasperFx.TypeDiscovery
• Adapter libraries (and I hate these, but you have to), go [ProjectName].[Category].[Other Tool]? Or [ProjectionName].[OtherTool]. Examples are Jasper.RabbitMQ or Marten.NodaTime

JasperFx -- Current Library Structure

This is our current library structure in JasperFx

classDiagram
class Baseline 
class BaselineTypeDiscovery

class Oakton
Oakton ..> Baseline
Oakton ..> BaselineTypeDiscovery

class Lamar

Lamar ..> BaselineTypeDiscovery

class LamarCodeGeneration
class LamarCompiler
class Lamar_Microsoft_DependencyInjection

Lamar ..> LamarCodeGeneration
Lamar_Microsoft_DependencyInjection ..> Lamar

class LamarCodeGeneration_Commands
LamarCodeGeneration_Commands ..> LamarCodeGeneration
LamarCodeGeneration_Commands ..> Oakton

Lamar_Diagnostics ..> Lamar
Lamar_Diagnostics ..> Oakton

class Alba
Alba ..> Baseline

class Weasel_Core
class Weasel_Postgresql
class Weasel_SqlServer
class Weasel_CommandLine

Weasel_Postgresql ..> Weasel_Core
Weasel_SqlServer ..> Weasel_Core
Weasel_CommandLine ..> Weasel_Core



class Marten

Marten ..> Baseline
Marten ..> BaselineTypeDiscovery
Marten ..> Weasel_Postgresql
Marten ..> LamarCodeGeneration
Marten ..> LamarCompiler

class Marten_CommandLine
Marten_CommandLine ..> Weasel_CommandLine
Marten_CommandLine ..> Oakton
Marten_CommandLine ..> Marten

class Jasper

Jasper ..> Lamar
Jasper ..> LamarCodeGeneration
Jasper ..> LamarCompiler
Jasper ..> Baseline
Jasper ..> BaselineTypeDiscovery
Jasper ..> Oakton
Jasper ..> LamarCodeGeneration_Commands

class Jasper_Persistence_Database
class Jasper_Persistence_Postgresql
class Jasper_Persistence_SqlServer
class Jasper_Persistence_Marten
class Jasper_Persistence_EntityFrameworkCore

Jasper_Persistence_Database ..> Jasper
Jasper_Persistence_Database ..> Weasel_Core
Jasper_Persistence_Postgresql ..> Jasper_Persistence_Database
Jasper_Persistence_Postgresql ..> Weasel_Postgresql
Jasper_Persistence_SqlServer ..> Jasper_Persistence_Database
Jasper_Persistence_SqlServer ..> Weasel_SqlServer

Jasper_Persistence_EntityFrameworkCore ..> Jasper_Persistence_Database

Jasper_Persistence_Marten ..> Jasper_Persistence_Postgresql
Jasper_Persistence_Marten ..> Marten

Loading

Grand Unified Theory

• BaselineTypeDiscovery --> JasperFx.TypeDiscovery in its own repository

• LamarCodeGeneration --> JasperFx.CodeGeneration it its own repository, along with LamarCompiler --> JasperFx.RuntimeCompiler

• Move some of the code generation and middleware related code from Jasper to JasperFx.CodeGeneration. [Transactional] would move over. Some code around the HandlerChain model in Jasper could move over. If we did that, then we could eliminate
  Jasper.Persistence.Marten and actually have the Marten middleware inside of Marten itself.

• Eliminate Jasper.Marten ??? If the raw IEnvelopeTransaction goes to the new messaging primitive lib, and the codegen middleware to Marten itself,
  there's no need for this library

• Rip the environment checks and new IStatefulResource model in Oakton to something small and common like JasperFx.Resources or JasperFx.Core. Take
  a direct reference from both Marten & Jasper to add more environment checks. Maybe even Lamar itself, so Lamar container checks are all wired in -- I like this idea (Jeremy)

• Could consider moving the remaining useful bits of Baseline to JasperFx.Common. There's a lot of overlap between Lamar and Baseline today. Think I'd vote to thin down Baseline even further if we did that. There's some things in Baseline that are now in the BCL that weren't 12-15 years ago when FubuCore/Baseline started

• A new Marten.Outbox add on library would add support for the jasper messaging primitives and the durability agent that's today in Jasper. That gives you effective outbox support within Marten SaveChangesAsync() calls. The abstractions in JasperFx.MessagingPrimitives should be broad enough to allow for efficient usage of MassTransit or NServiceBus to be plugged in as the senders as well as raw adapters to the various messaging transports (Rabbit MQ, Kafka, etc.).

• Rip the Envelope, some of the core Jasper abstractions like ISender and ISendingAgent, the durability agent, and maybe even some of the basic TPL Dataflow code into a new common JasperFx.MessagingPrimitives library that would be consumed by both Marten.Outbox (new Marten addon) and Jasper itself. This is going to take some serious refactoring in Jasper itself to thin down the "special" parts of the Envelope structure, but that's been halfway planned anyway

• Jasper gets a lot smaller, and mostly gets focused on command execution, the handler pipeline, open telemetry diagnostics. The current database persistence libraries probably end up in other places

classDiagram

class JasperFx_MessagingPrimitives
class JasperFx_Messaging_Database
class JasperFx_Messaging_Postgresql
class JasperFx_Messaging_SqlServer

class JasperFx_Messaging_RabbitMq
class JasperFx_Messaging_Pulsar
class JasperFx_Messaging_Kafka

JasperFx_Messaging_RabbitMq ..> JasperFx_MessagingPrimitives
JasperFx_Messaging_Pulsar ..> JasperFx_MessagingPrimitives
JasperFx_Messaging_Kafka ..> JasperFx_MessagingPrimitives

JasperFx_Messaging_Database ..> JasperFx_MessagingPrimitives
JasperFx_Messaging_Database ..> Weasel_Core

JasperFx_Messaging_Postgresql ..> JasperFx_Messaging_Database
JasperFx_Messaging_Postgresql ..> Weasel_Postgresql

class Marten
class Jasper
class Marten_Outbox

Marten_Outbox ..> Marten
Marten_Outbox ..> JasperFx_Messaging_Postgresql

Jasper ..> JasperFx_MessagingPrimitives
Jasper ..> JasperFx_CodeGeneration
Jasper ..> JasperFx_TypeDiscovery
Jasper ..> JasperFx_RuntimeCompiler

Marten ..> JasperFx_CodeGeneration
Marten ..> JasperFx_TypeDiscovery
Marten ..> JasperFx_RuntimeCompiler

class JasperFx_Common
Oakton ..> JasperFx_Common
Marten ..> JasperFx_Common
Lamar ..> JasperFx_Common

Loading

Re-birth of Storyteller

Storyteller is dead in the water, but I still want to resurrect it some day, maybe under a different name (Bobcat? Wolverine?). If Storyteller
lives again with a new UI, I'd like to dogfood quite a bit of Jasper in it. An idea I've had for a bit was to rip the Envelope model and the
TCP-based socket WireProtocol in Jasper to use in the Storyteller library for communication between the running Storyteller process and the new
Storyteller UI. These changes outlined above would further that goal

[jeremydmiller]

I'm going to start a new discussion on this and just link to this.