Nevermore 12.0

[PaulStovell]

Nevermore 12.0 is a major refactor, rewrite and design change to Nevermore. This issue summarises some of the key changes and what I was trying to do when working on it.

Highlights

A lot has changed! Here are the main reasons for the changes and why you should care about upgrading:

• ⏩ Faster! Nevermore 12.0 is faster for most queries, and when reading larger documents, it's much faster and uses less memory (up to 75% less). Many queries are significantly optimized.
• 💀 Nevermore.Contracts is dead. You no longer need IId or IDocument. This library is gone.
• 🕐 Async support. You have been awaiting a long time for this 😉
• 🌲 Extensible. You can add new types, handle inherited documents, and more, without changing existing code.
• 🌮 New queries. Documents are boring. Stream plain classes, tuples, and more.
• ✂️ Read/write. Split IReadTransaction and IWriteTransaction.
• 🤐 Compression. Use compression for large documents.
• 🔍 Roslyn analyzer. Detects common mistakes at compile time.
• 🔢 Source link. We now publish symbols and source link information so you can step directly into Nevermore code as you debug.
• 📖 Documentation. The wiki now contains a heap of information about how to use Nevermore.

There are some breaking changes which will be outlined below, but the highlights are:

• Some method signatures have changed
• IId, IDocument etc. are gone - projects that need them will need to copy the code
• AmazingConverter and some other internal classes are gone
• ReferenceCollection is gone, if you need this there's a shim you can copy

⚠️ Because so much changed, make sure you test your application thoroughly upon upgrading

Performance improvements

A benchmarking project has been added. Here's how old Nevermore 12.0 compares to (on master) on my computer running Windows (bare metal) against a local SQL Express instance:

These improvements come about thanks to:

• Caching of the various plans we make around how we read and map documents
• Using CommandBehavior.SequentialAccess when querying, and changing all reading code to support it
• When reading documents > 1KB, instead of reading them as a giant string (which for 85KB documents sends them to the LOH), we read them directly in a buffer from the data reader. This should improve load and query performance for at least 20% of the documents in a large Octopus customer database, or 25% of the documents in Octofront.

Load(string[]) got improved quite a bit as the new Nevermore pulls in #92.

Loaded 50000 products in 7347ms (old Nevermore)
Loaded 50000 products in 322ms (new Nevermore)

Comparing Nevermore to hand-coding SQL is difficult as it's an 🍎 to 🦧 comparison, as deserializing JSON will add some overhead. A fairer comparison is between hand-coded SqlCommand and Nevermore's new support for tuples/plain classes, which is very close to hand-coded performance:

.NET Core 3.1

Nevermore has been upgraded to .NET Core 3.1. This lets it take advantage of things like IAsyncEnumerable. You'll need to be on .NET Core 3.1 or above to use this version of Nevermore. I figure this should be OK for us as Octofront can be upgraded and Octopus is already on 3.1.

Simplified setup

Nevermore 12.0 is designed to be much easier to set up. All you actually need is a connection string:

var config = new RelationalStoreConfiguration(ConnectionString);
config.Mappings.Register(new PersonMap());

store = new RelationalStore(config);

You no longer need to call TransientFaultHandling.InitializeRetryManager(); as this happens automatically.

New query types

You aren't limited to just querying documents that have been mapped. Using Stream, you can run arbitrary queries against POCO classes:

class Result
{
    public string FullName { get; set; }
    public string Email { get; set; }
}

// This pattern can be used for just about any quick query. For this to work, property names on the type
// must match the name of columns from the result set.
var result = transaction.Stream<Result>(
    "select FirstName + ' ' + LastName as FullName, Email from dbo.Person order by FirstName"
).First();

You can also query tuples:

var result = transaction.Stream<(string LastName, int Count)>(
    "select LastName, count(*) from dbo.Person group by LastName order by count(*) desc, len(LastName) desc"
).ToList();

And lists of primitive types:

var names = transaction.Stream<string>(
    "select FirstName from dbo.Person order by FirstName"
).ToList();

Async support

There are now async versions of insert, update, and all the queries.

using var transaction = await Store.BeginWriteTransactionAsync();

await transaction.InsertAsync(
    new Product { Name = "First product", Price = 100.00M, Type = ProductType.Dodgy}, 
    new InsertOptions { CustomAssignedId = "Product-First"});

var first = await transaction.LoadAsync<Product>("Product-First");

Extensible

Nevermore 12.0 is extensible, and the extensions have been designed to be easy to use.

You can provide a custom ITypeHandler when you want to control how values from database columns are read or written to CLR objects:

public class UriTypeHandler : ITypeHandler
{
    public bool CanConvert(Type objectType)
    {
        return objectType == typeof(Uri);
    }

    public object ReadDatabase(DbDataReader reader, int columnIndex)
    {
        if (reader.IsDBNull(columnIndex))
            return default(Uri);
        var text = reader.GetString(columnIndex);
        return new Uri(text);
    }

    public void WriteDatabase(DbParameter parameter, object value)
    {
        parameter.Value = ((Uri) value)?.ToString();
    }
}

These are then registered against the store:

store.Configuration.TypeHandlerRegistry.Register(new UriTypeHandler());

(You can also make your ITypeHandler a JsonConverter if you want to centralize the logic for either column or JSON storage)

They will then be used:

• When querying documents with a JSON column
• When querying to plain generic CLR objects (without a JSON column)
• When querying tuples
• When querying simply for that type
• For inserts, updates, deletes, and so on

You won't need to create an IReaderWriter or set it on any column anymore.

Handling inherited classes (e.g., Account/AzureAccount) has also been simplified a lot. The rules are:

• You have to have a column named Type, and it has to be before the JSON type
• You register an IInstanceTypeResolver

Instance type resolvers are easy:

public class ProductTypeResolver : IInstanceTypeResolver
{
    public Type Resolve(Type baseType, object typeColumnValue)
    {
        if (typeof(Product).IsAssignableFrom(baseType) && typeColumnValue is ProductType productType)
        {
            // Note that these could easily be split into three different IInstanceTypeResolver classes.
            if (productType == ProductType.Dodgy) return typeof(DodgyProduct);
            if (productType == ProductType.Special) return typeof(SpecialProduct);
            if (productType == ProductType.Normal) return typeof(Product);
        }

        return null;
    }
}

There are also ways to provide fallback behavior for when a type can't be matched.

Nevermore.Contracts is dead

We (Shannon and I) decided to remove this. It means that you no longer need to implement IId or IDocument, because they don't exist anymore.

You can use Stream to query any class without a document type defined.

If you use TableQuery or any of the other query builders, or call the Insert, Update, Delete methods, you'll need to provide a DocumentMap. And, you will need an ID.

The convention is to have a string property called "Id". But you can:

1. Use a different name - just set it using IdCoumn.ColumnName("MyId") on the DocumentMap
2. Change the type. It doesn't have to be a string, but it has to be able to be cast to a string in a few situations.

ReferenceCollection is also gone. If you use this, you'll need to create your own type. You'll also need to provide a ITypeHandler that can read/write it to columns. It's no longer included in Nevermore as a better practice is to use OPENJSON.

[PaulStovell]

This has been merged to master and published to NuGet as Nevermore 12.1.0.