TypeDB 3.0.0

Download from TypeDB Package Repository:

Distributions for 3.0.0

Pull the Docker image:

docker pull typedb/typedb:3.0.0

Announcement

This last year of rewriting TypeDB 2.x into TypeDB 3.0 has been an adventure of learning, iterating, and lots of fun for our whole team.

We're thrilled to say we've achieved our first tranche of goals:

• Initial testing shows TypeDB 3.0's performance is comparable-to or surpasses MongoDB in transactional workloads (a 3-5x performance gain over TypeDB 2.x!)
• TypeDB 3.0 leverages its new Rust codebase to greatly increase correctness, reduce memory footprint, and improve performance
• It features a simplified architecture that opens the door for adding new features and optimisations quickly

Not only that, but we've upgraded TypeQL to version 3.0 to directly address long-standing requests:

• Rules have been replaced with Functions: functions are more flexible, easier to reason about, and much more familiar to programmers. Functions are like subqueries you can re-use and invoke whenever you want. We think you'll love them!
• Query pipelining: combined with more powerful expressions and functions, you can now build read, write, or transformation pipelines that do everything you need without coming back to the client. This is a triple threat: more readable, more maintainable, and more performant.
• Data constraints: we welcome the arrival of Cardiniality, value Range, and value Enumeration restrictions, among others. These have been our top requested features for over a year and now they're at your fingertips! TypeDB will now automatically validate that your data has the exact connectivity and shape you require.

There's so much more than this, and we hope you'll dig into TypeDB 3.0 to try it out.

Top 10 TypeDB 3.0 Features

1. Goodbye sessions, hello transactions: TypeDB 3.0 eliminates sessions and simplifies your interactions to use 3 types of transactions. Use read, write, or schema transactions when reading data, modifying data, or exclusively modifying schema & data, respectively. Transactions are still ACID up to snapshot isolation, and support concurrent reads and writes.

2. Standard return types - Rows or Documents: TypeDB 3.0 simplifies the answers to be either Rows or Documents. Rows will feel familiar to SQL users, though still contain our traditional Concept data types. Documents can be structured and build using the enhanced fetch clause.

3. Enhanced schema language: We've streamlined define, undefine, and introduce redefine to modify schemas - while restructuring the definition language to be shorter, more consistent, and more understandible:

TypeDB 2.x:

define
  person sub entity,
    owns age,
    plays friendship:friend;
  friendship sub relation,
    relates friend;
  age sub attribute, value string;

  abstract-friendship sub relation,
    relates friend, abstract;


undefine
  abstract-friendship relates friend, abstract; # does this undefine the `abstract`, the `friend` role, or the `abstract-friendship`?
  

TypeDB 3.x:

define
  entity person,
    owns age,
    plays friendship:friend;
  relation friendship,
    relates friend;
  attribute age, value string;
  relation abstract-friendship,
    relates friend @abstract;
  entity child, owns age;
  entity dog;


undefine
  @abstract from abstract-friendship relates friend; # clearly removes @abstract
  owns age from child; # clearly removes only the age ownership from the child type
  dog; # clearly undefines the dog type

4. Query pipelines: Pipelines are best illustrated. Let's say we want to assign a tax credit for a person called Bill, corresponding to how many children he has.

In TypeDB 2.x, we have to split this into multiple queries that do redundant work, and perform multiple network round trips. In addition, we have to split some of our logic between the database and the application, which damages maintainability!

# count how many children Bill has
children = tx.query().get_aggregate('match $p isa person, has email "[email protected]"; (child: $child, parent: $p) isa parentship; get $child; sum;').resolve()
# compute tax credit
tax_credit = 1000*children;
# assign tax credit
tx.query().insert(f'match $p isa person, has email "[email protected]"; insert $p has tax_credit {tax_credit};')

In TypeDB 3.0, this is streamlined into one query pipeline:

tx.query("""
  match $p isa person, has email "[email protected]"; parentship (child: $child, parent: $p);
  reduce $count = count($child) within $p;
  match let $tax_credit = 1000 * $count;
  insert $p has tax_credit == $tax_credit;
""")

This is a simple pipeline: we could continue to chain operations, such as inserts and deletes, to build complex transformations.

5. Reduce and aggregation operations: As the previous example illustrates, what used to be a get; sum; operation is now a reduce clause! This is a much more expressive way of aggregating values and allows using them in future operations.

6. Enhanced Fetch: fetch clauses now are structured exactly how they will return JSON objects:

match
    $p isa person, has email $email, has age $age; $email == "[email protected]";
fetch {
    "email": $email, 
    "tax-identifier": $p.tax_id,
    "names": [ $p.name ],
    "age_next_year": $age + 1,
    "total_salary": (
        match
        $p has salary $salary;
        return sum($salary);
    ),
    "children_ages": [
        match
        parentship (child: $child, parent: $p);
        fetch {
            "age": $child.age,
        };
    ],
    "all_attributes": { $p.* }
};

This will return a stream of JSON documents that look like this:

{
    "email": "[email protected]", 
    "tax-identifier": "123-45-6789",
    "names": [ "bill", "billstone" ],
    "age_next_year": 51,
    "total_salary": 50000,
    "children_ages": [
      { "age": 10 },
      { "age": 13 },
    ],
    "all_attributes": { 
      "email": "[email protected]",
      "age": 50,
      "tax_id": "123-45-6789",
      "names" [ "bill", "billstone" ],
      "salary": [ 10000, 40000 ],
    }
};

{
    "email": "[email protected]",
    "tax-identifier": "123-45-6789",
    "names": [
        "bill",
        "billstone"
    ],
    "age_next_year": 51,
    "total_salary": 50000
    "children_ages": [
        { "age": 10 },
        { "age": 12 }
    ],
    "all_attributes": {
        "age": 50,
        "email": "[email protected]",
        "name": [
            "bill",
            "billstone"
        ],
        "salary": [
            10000,
            40000
        ],
        "tax_id": "123-45-6789"
    },

}

7. Functions: functions take a set of arguments, and return either a stream or a tuple of concepts. Functions can contain any read pipeline!

define
fun mean_salary($p: person) -> double:
  match $p has salary $salary;
  return mean($salary);

8. Let-bindings and inline expressions: expressions are now bound to a new variable (using the same $ syntax - no more ? variables!) using the 'let' keyword. If you don't need to name your result, you can inline expressions:

match
  $p has salary $annual_salary;
  let $monthly_salary = $annual_salary / 12;
  parentship (parent: $p, child: $child);
  $child has age > (10 + 8);

9. Data constraints: TypeDB 3.0 ships with built-in controls for cardinality, values, ranges, and abstractness, best shown by example:

define
  entity person, 
    owns email, # NEW: unless specified, ownerships have a default cardinality of @card(0..1) (exactly 0 or 1 attributes)
    owns name @card(0..),  # specifically relaxed cardinality of 0 to infinity
    owns tax_id @card(1, 1), # require exactly 1 tax_id to be owned 
    owns age,
    owns gender;

    plays parentship:parent, # NEW: unless specified, played roles have an implicit @card(0..) (any number of connections allowed)
    plays parentship:child;

  relation parentship,
    relates child, # NEW: unless specified, relation roles have a default cardinality of @card(0..1) (exactly 0 or 1 of the role)
    relates parent @card(1..2);  # specialised relation roles to require 1 or 2 parents for each parentship relation

  attribute age, value integer @range(0..); # require all ages to be of values equal to or greater than 0
  attribute gender, value string @values("male", "female", "other"); # restrict the domain of values to an exact set

10. New built-in value types: TypeDB 3.0 ships with a larger set of value types:

• integer (renamed from long in 2.x)
• double
• boolean
• string
• NEW: date, representing a date, without a time
• datetime
• NEW: datetime-tz, representing a date time with timezone
• NEW: duration
• NEW: Fixed-Decimal numbers

Aggregated release note ---------------

New Features

• User authentication and network encryption

  User authentication

  Bring user authentication and user management functionalities into TypeDB Core through the following changes:

  • Implement user management functionality, handling user creation, retrieval, update, and deletion.
  • Implement authentication protocol - TypeDB Core drivers are now required to supply credentials when making a connection

  System database

  Implement a system database, a special database that the server will use to store various systems-related information. The primary motivation is to store database server users for authentication purpose, but in the future can be extended to store other system-related information.

  Network encryption

  Bring n...

TypeDB 3.0.0-alpha-10

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-10

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-10

New Features

• IID constraint

  We add support for IID constraints in match clauses.

  Example:

  match $x iid 0x1E001A0000000012345678;
  

Bugs Fixed

• Fix greedy planner to sort on correct vars

  Currently, the greedy planner picks a plan in form of an ordering of constraints and variables. An added constraint produces variables if it relates those variables and they haven't been added to the plan at an earlier stage.

  The greedy planner picks constraints based on their "minimal" cost for a directional lookup, that may sort on either of one or more variables that the constraint relates. However, it does not record which variable the lookup should be sorted on. This means, as it stands, produced variables are added in random order after their constraint to the ordered plan.

  This PR aims to ensure that we sort on the variable for the appropriate direction, by adding them first.

• Guarantee transaction causality

  Under concurrent write workloads, causality can appear to be violated. For example:

  - Open tx1, tx2
  - start tx1 commit (validation begins)
  - start tx2 commit (validation begins and ends, no conflict with tx1!)
  - open tx3
  - end tx1 commit (validation finishes)
  

  When we open tx3, we end up with a snapshot that is actually from before tx1, even though tx2 has committed - because we don't know the status of tx1 yet, and in our current simplified model, transaction are assigned a linear commit order decided at WAL write time, the read watermark remains before tx1 until it finishes validating.

  In this scenario, a client that commits a transaction can actually end up opening the next transaction snapshot before the last commit that successfully returned.

  After this change, when opening a transaction, TypeDB wait until the currently pending transactions all finish, guaranteeing we see the latest possible data version. The assumption is that 1) validation in general is a small amount of time and 2) is fully concurrent, so the wait time should be very small, and only occur under large concurrently committing transactions.

Code Refactors

• Optimisations from typeql

  Updates typeql for parsing optimisations.

Other Improvements

• Add BDD tests for functions

  Add basic BDD tests for functions & implement validation.

• Frugal type seeder

  Optimises parts of the type seeder to avoid unnecessary effort.

• Improve Bytes ergonomics by allowing automatic dereference to byte slices

TypeDB 3.0.0-alpha-9

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-9

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-9

New Features

• Exit on panic

  In Rust, whenever a panic is encountered in a background thread, the main thread is normally unaffected until it either joins the background thread, or attempts to lock a mutex that was held by the panicking thread. If that communication never occurs (e.g. no locks were held by a worker), the server may end up in an invalid state.

  We override the default behaviour to exit from the process with code 1.

• Stabilise fetch and introduce fetch functions
  We introduce function invocation in fetch queries. Fetch can call already existing functions:

  match
    $p isa person;
  fetch {
    "names": [ get_names($p) ],
    "age": get_age($p)
  };
  

  and also use local function blocks:

  match
    $p isa person;
  fetch {
    "names": [
      match
        $p has name $n;
        return { $n };
    ],
    "age": (
      match
        $p has age $a;
        return first $a;
    )
  };
  

  In the examples above, results collected in concept document lists ("names": [ ... ]) represent streams (purposely multiple answers), while single (optional) results ("age": ...) represent a single concept document leaf and do not require (although allow) any wrapping.

Moreover, we stabilize attribute fetching, allowing you to expect a specific concept document structure based on your schema. This way, if the attribute type is owned with default or key cardinality (@card(0..1) or @card(1..1)), meaning that there can be at most one attribute of this type, it can be fetched as a single leaf, while other cardinalities force list representation and make your system safe and consistent. For example, a query

match
  $p isa person;
fetch {
  $p.*
};

can automatically produce a document like

{
  "names": [ "Linus", "Torvalds" ],
  "age": 54
}

with

define
  entity person 
    owns name @card(1..), 
    owns age @card(1..1);

Additionally, fetch now returns attributes of all subtypes x sub name; y sub name; for $p.name, just like regular match queries like match $p owns name $n.

With this, feel free to construct your fetch statements the way you want:

match
  $p isa person, has name $name;
  fetch {
    "info": {
      "name": {
        "from entity": [ $p.person-name ],
        "from var": $name,
      },
      "optional age": $p.age,
      "rating": calculate_rating($p)
    }
  };

to expect consistent structure of the results:

[
  {
    "info": {
      "name": {
        "from entity": [ "Name1", "Surname1" ],
        "from var": "Name1"
      },
      "optional age": 25,
      "rating": 19.5
    }
  },
  {
    "info": {
      "name": {
        "from entity": [ "Name1", "Surname1" ],
        "from var": "Surname1"
      },
      "optional age": 25,
      "rating": 19.5
    }
  },
  {
    "info": {
      "name": {
        "from entity": [ "Bob" ],
        "from var": "Bob"
      },
      "optional age": null
      "rating": 28.783
    }
  }
]

• Implement query executable cache

  We implement the cache (somewhat arbitrarily limited to 100 entries) for compiled executable queries, along with cache maintanance when statistics change significantly or the schema updates.

  Query execution without any cache hits still looks like this:

  Parsing -> Translation (to intermediate representation) -> Annotation -> Compilation -> Execution
  

  However, with a cache hit, we now have:

  Parsing -> Translation ---Cache--> Execution
  

  skipping the annotation and compilation/planning phases, which take significant time.

  Note that schema transactions don't have a query executable cache, since keeping the cache in-sync when schema operations run can be error prone.

  The query cache is a structural cache, which means it will ignore all Parameters in the query: variable names, constants and values, and fetch document keys. Most production systems run a limited set of query structures, only varying values and terms, making a structural cache like this highly effective!

• Introduce naive retries for suspended function calls
  Introduces retries for suspended function calls.
  Functions may suspend to break cycles of recursion. Restoring suspend points is essential to completeness of recursive functions, and thus correctness of negations which call them.

• Query execution analyser

  We implement a useful debugging feature - a query analyzer. This is similar to Postgres's Explain Analyze, which produces both the query plan plus some details about the data that has flowed through the query plan and the time at each step within it.

  Example output:

  Query profile[measurements_enabled=true]
    -----
    Stage or Pattern [id=0] - Match
      0. Sorted Iterator Intersection [bound_vars=[], output_size=1, sort_by=p0]
        [p0 isa ITEM] filter [] with (outputs=p0, )
      ==> batches: 158, rows: 10000, micros: 6407
  
      1. Sorted Iterator Intersection [bound_vars=[p0], output_size=2, sort_by=p1]
        Reverse[p1 rp p0 (role: __$2__)] filter [] with (inputs=p0, outputs=p1, checks=__$2__, )
      ==> batches: 854, rows: 39967, micros: 75716
  
    -----
    Stage or Pattern [id=1] - Reduce
      0. Reduction
      ==> batches: 1, rows: 10000, micros: 116035
  
    -----
    Stage or Pattern [id=2] - Insert
      0. Put attribute
      ==> batches: 10000, rows: 10000, micros: 5890
  
      1. Put has
      ==> batches: 10000, rows: 10000, micros: 54264
  

  When disabled, profiling is a no-op (no strings are created, locks taken, or times measured), though there is still some cost associated with cloning Arcs containing the profiling data structures around.

  To enable query profiling, the easiest way (for now) is to enable TRACE logging level for the executor package, currrently configured in //common/logger/logger.rs:

  .add_directive(LevelFilter::INFO.into())
  // add:
  // .add_directive("executor=trace".parse().unwrap())
  

  Alternatively, just set the enable boolean to true in the QueryProfile::new() constructor.

Bugs Fixed

• Variable position fix

• Disable variables of category value being narrow-able to attribute
  Disable variables of category value being narrowed to attribute

  $name = "some name"; $person has name == $name; # Fine
  $name == "some name"; $person has name $name; # Also Fine
  $name = "some name"; $person has name $name; # Disallowed
  

• Fix disjunction inputs when lowering
  The disjunction compiler explicitly accepts a list of variables which are bound when the disjunction is evaluated. This also fixes a bug where input variables used in downstream steps would not be copied over.

• Fixes early termination of type-inference pruning when disjunctions change internally
  Fixes early termination of type-inference pruning when disjunctions change internally

Code Refactors

• Update FactoryCI images to typedb-ubuntu

• Rename org to 'typedb' in deployment.bzl

• Rename org to 'typedb' in CircleCI docker job

• Boxed large result Error variants

  We follow some best practices to box the large Error variants of Result types, which should optimise both the stack size and the amount of memory that has to be copied between function calls.

• Decimal to double cast

  We can now use decimals in double expressions.

• Optimise has comparators and implement Executable Display

  We inline comparators into HasReverse executors, as a continuation of #7233, which inlined comparators into Isa executors. Both optimisations reduce the search space by using the value ranges provided by constants in the query or previously executed query steps. We also improve the query planner's statistics and cost function.

Other Improvements

• Add cargo check to CI

  We add a job to Factory CI which verifies that the Cargo configuration is valid and buildable.

• Return query operations errors on commit
  We return errors from the awaited operations on commit as the commit error. It can be useful if you want to run a number of queries from your TypeDB Driver like:

  let queries = [........];
  let mut promises = vec![];
  for query in queries {
      promises.push(transaction.query(query));
  }
  
  let result = transaction.commit().await;
  println!("Commit result will contain the unresolved query's error: {}", result.unwrap_err());

• Update FactoryCI images to typedb-ubuntu

• Fix structural equality test

• Implement structural equality

  We implement a form of Structural Equality, which check the equality of two queries, in Intermediate Representation format, which allows us to compare to queries for equality while ignoring user-written variable names and function names and constants. This is a building block for the next step, which will introduce a executable cache, allowing TypeDB to skip type inference and planning for queries that already exist in the cache.

• Deleted BUILD_java

• Commit generated Cargo.tomls + Cargo.lock

  We commit the generated cargo manifests so that TypeDB can be built as a normal cargo project without using Bazel.

  Integration and unit tests can be run normally, and rust-analyzer behaves correctly. Behaviour tests currentl...

TypeDB 3.0.0-alpha-8

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-8

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-8

New Features

• Is constraint

  We implement the is constraint support in queries:

  match
      $x isa company;
      $y isa person;
      $z isa person;
      not { $y is $z; };  # <---
      $r1 isa employment, links ($x, $y);
      $r2 isa employment, links ($x, $z);
      select $x, $y, $z;

• Introduce 3.0 undefine queries
  We introduce undefine queries with the new "targeted" syntax to enable safe and transparent schema concept undefinitions.

If you want to undefine the whole type, you can just say:

undefine 
  person;

If you want to undefine a capability from somewhere, use undefine **capability** from **somewhere**.
It consistently works with owns (person is preserved, only the owns is undefined):

undefine
  owns name from person;

annotations of owns (person owns name is preserved, only @regex(...) (you don't need to specify the regex's argument) is undefined):

undefine
  @regex from person owns name;

and any other capability, even specialisation:

undefine
  as parent from fathership relates father;

Want to undefine multiple concepts in one go? Worry not!

undefine
  person;
  relates employee from employment;
  @regex from name;
  @values from email;

The error messages in all definition queries (define, redefine, and undefine) were enhanced, and the respective query/language BDD tests were introduced in the CI with concept unit tests.

Additional fixes:

• answers of match capability queries like match $x owns $y, match $x relates $y, and $match $x plays $y now include transitive capabilities;
• define lets you write multiple declarations of an undefined type, specifying its kind anywhere, even in the last mention of this type;
• failures in schema queries for schema transactions no longer lead to freezes of the following opened transactions;
• no more crashes on long string attributes deletion with existing has edges;

Bugs Fixed

• Fix outputs from earlier disjunction branches affecting later branches
  Fixes outputs from earlier disjunction branches affecting later branches

• Ensure sorted join executor respects selected outputs

  Before this change, an intersection executor (which is used for any number of sorted iterators, not necessarily a join) would write into the output row all items in the constituent executors, so long as they have a row position assigned. That would happen even if those values were not selected for future steps, which prevented deduplication of answers from being handled correctly.

Code Refactors

• Simplify attribute encoding

  We remove separate prefixes for attribute instances and instead introduce a value type prefix after the attribute type encoding.

  Encoding before:

  [String-Attribute-Instance][<type-id>][<value>][<length>]
  

  After:

  [Attribute-Instance][<type-id>][String][<value>][<length>]
  

  Note: this change breaks backwards compatibility

• Refactor after basic functions
  Refactors the executor to be have more simple, flat instructions than less, complex ones.

Other Improvements

• Carry correct types into type list executor

  In cases where the list of types is dependent on the context (e.g. {$t label T;} or {$t label U;};, the type list executor (role name, explicit label, or kind) should only be producing the types in the list defined by the constraint. That was handled correctly when the type constraint was used as a post-check, but incorrectly during direct execution.

• Update spec.md

• Introduce 3.0 value and as match constraints
  We introduce value match constraint for querying for attribute types of specific value types:

  match
    $lo value long;
  

  We introduce as match constraint for querying for role types specialisation:

  match
    $relation relates $role as parentship:child;
  

  which is equivalent to:

  match
    $relation relates $role;
    $role sub parentship:child;
  

  but is useful for shortening your queries, making it similar to define definitions, and in case of anonymous variables:

  match
    $relation relates $_ as parentship:child;
  

• Optimise comparators

  We optimise the execution of queries containing comparators (>/</==/...) where these can be used to restrict the data search space. This PR sets up general infrastructure for using restrictions when searching the data, but uses it only for optimising ISA lookups - follow up work will optimise HAS operations as well.

• Add type annotation to fix build in circleci

• Update spec: fix list insert, add value pattern, add as! pattern

• Bugfix: do not reuse cartesian product iterators between inputs

  We can't reuse the iterators created from the previous input row, as they would iterate over different tuples. Now, when we take a new input, we reset the cartesian iterators.

• Update spec.md (fix sssnake expresssions)

• Green match BDD redux

  We resolve the remaining flaky failures in match BDDs.

• Update banner.png

• Update many unit & integration tests & enable in CI
  Updates many unit & integration tests & enables them in CI

TypeDB 3.0.0-alpha-7

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-7

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-7

New Features

• Expression support

  We add expression execution to match queries:

  match
      $person_1 isa person, has age $age_1;
      $person_2 isa person, has age $age_2;
      $age_2 == $age_1 + 2;

• Basic streaming functions

  Implement function execution for non-recursive stream functions which return streams only.
  Recursive functions & Non-stream functions will throw unimplemented.

  These can currently only be used from the preamble. Sample:

             with
              fun get_ages($p_arg: person) -> { age }:
              match
                  $p_arg has age $age_return;
              return {$age_return};
  
              match
                  $p isa person;
                  $z in get_ages($p);

• Implement tabling machinery for cyclic functions

  Introduces machinery needed to support finding the fixed-point of cyclic function calls. Cyclic functions now run, and return an incomplete set of results followed by an error. It is possible that the planner chooses a plan.

Bugs Fixed

• Minor query planner bug fixes

  1. Fix the issue where an iterator would attempt to use a variable before assigning to it.
  2. Let role name constraint behave as a post-check.

Code Refactors

• Resolve warnings

Other Improvements

• Update protocol to 3.0.0-alpha-7 release

• 3.0 Rename Ok.Empty to Ok.Done. Add query_type to the Done response

  We add a query_type field to all the Query.InitialRes.Ok protobuf messages to support retrieval of this information for any QueryAnswer on the client side.
  Additionally, the Ok.Empty message was renamed to Ok.Done.

• Fix and add to CI concept BDD tests

  We fix data commit error collection and checks and add concept package BDD tests to CI.

TypeDB 3.0.0-alpha-6

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-6

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-6

New Features

• Partial disjunction support

  We introduce partial support for disjunctions in queries:

  match
      $person isa person;
      { $person has name $_; } or { $person has age $_; };

• Fetch execution

  We implement fetch execution, given an executable pipeline that may contain a Fetch terminal stage.

  Note: we have commented out match-return subqueries, fetching of expressions ({ "val" : $x + 1 }), and fetching of function outputs ("val": mean_salary($person)), as these require function-body evaluation under the hood - this is not yet implemented.

Bugs Fixed

• Fix document answers streaming for fetch
  We fix document answers streaming for fetch in order to complete the first version of fetch queries execution.

Code Refactors

• Function compilation preparation & trivial plan implementation
  Prepares higher level packages for compiling functions.

• Fetch annotation, compilation, and executables

  We implement Fetch annotation, compilation and executable building, rearchitecting the rest of the compiler to allow for tree-shaped nesting of queries (eg. functions or fetch sub-pipelines).

• Fetch annotations framework

  We implement further refactoring, which pull Fetch into Annotations and Executables, without implementing any sub-methods yet.

Other Improvements

• Add database name validation for database creation
  We add validation of names for created databases. Now, all database names should be valid TypeQL identifiers.

• Enable connection BDDs in CI
  We update some of the steps for BDDs to match the updated definitions.
  Additionally, we add connection BDDs to the factory CI to make sure that this piece of the system is safe and stable.

• Refactor compiler

  We refactor the compiler to standardize naming, removing the usage of 'program', and introducing 'executable' as the second stage of compilation.

• Refactor pipeline annotations

  We implement the next step of Fetch implementation, which allows us to embed Annotated pipelines into Fetch sub-queries and into functions. We comment out the code paths related to type inference for functions, since functions are now enhanced to include pipelines.

TypeDB 2.29.1

Install & Run: https://typedb.com/docs/home/install

Download from TypeDB Package Repository:

Server only: Distributions for 2.29.1

Server + Console: Distributions for 2.29.1

New Features

• Update docker image to have both x86_64 and arm64 versions
  Publish arm64 docker images. Previously, docker would fall-back to running the x86_64 images, which sometimes resulted in a JVM crash loop.

TypeDB 2.29.0

Install & Run: https://typedb.com/docs/home/install

Download from TypeDB Package Repository:

Server only: Distributions for 2.29.0

Server + Console: Distributions for 2.29.0

New Features

• Update TypeDB Console version
  Update TypeDB Console to a version with an updated password-update command which allows specifying the password directly.

Code Refactors

• Cleaner logs when transaction close forcibly terminates reasoner
  We update the reasoner to explicitly check whether the cause of termination was a transaction close or an exception, and only log in the latter case.

Other Improvements

• Update APT maintainer to TypeDB Community

  The maintainer field of our APT package is now TypeDB Community [email protected].

• Fix large queries freezing server with new thread-pool for transaction requests
  We introduce a separate thread-pool for servicing transaction requests. This avoids the case where multiple long running queries can occupy all the threads in the pool, and causing the server to be unable to process other requests. Critically, this allows session close calls to terminate any such transactions.

TypeDB 3.0.0-alpha-5

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-5

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-5

New Features

• Negation

  We add support for negations in match queries:

  match $x isa person, has name $a; not { $a == "Jeff"; };

  Note: unlike in 2.x, TypeDB 3.x supports nesting negations, which leads to a form of "for all" operator:

  match
      $super isa set;
      $sub isa set;
      not { # all elements of $sub are also elements of $super:
          (item: $element, set: $sub) isa set-membership;  # there is no element in $sub...
          not { (item: $element, set: $super) isa set-membership; }; # ... that isn't also an element of $super
      };

• Require implementation

  Implement the 'require' clause:

  match
  ...
  require $x, $y, $z;
  

  Will filter the match output stream to ensure that the variable $x, $y, $z are all non-empty variables (if they are optional).

• Let match stages accept inputs

  We now support match stages in the middle of a pipeline, which enables queries like this:

  insert
      $org isa organisation;
  match
      $p isa person, has age 10;
  insert
      (group: $org, member: $p) isa membership;
  

Other Improvements

• Add console script as assembly test

• Add BDD test for pipeline modifiers
  Add BDD test for pipeline modifiers; Fixes some bugs uncovered in the process.

• Fix failing type-inference tests

• Fetch part I

  We implement the initial architecture and refactoring required for the Fetch implementation, including new IR data structures and translation steps.

• TypeDB 3 - Specification

  Added specification of the TypeDB 3.0 database system behaviour.

TypeDB 3.0.0-alpha-4

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-4

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-4

New Features

• Include console in 3.0 distribution
  Includes console in 3.0 distribution. It can perform database management & run queries.

  Boot up server and console using the same binary, just like TypeDB 2.x:

  ./typedb server
  

  and

  ./typedb console
  

• Introduce reduce stages for pipelines
  Introduce reduce stages for pipelines to enable aggregates and group-aggregates.
  Aggregate operators available: count, sum max, min, mean, median, std

  Calculating mean duration of a site's visitors:

  match
    $page isa page, has url == "typedb.com";
    (page: $page) isa page-visit, has duration $duration;
  reduce $mean_visitor = mean($duration);
  
  // returns [ $mean_visitor ]
  

  Counting number of page visits grouped per website

  match 
    $page isa page;
    (page: $page, visitor: $visitor) isa page-visit;
  reduce $site_visitors = count($visitor) within $page; 
  
  // return [ $page, $site_visitors ] for each page
  

• Add support for ISO timezones; finalise datetime & duration support

Bugs Fixed

• Fix greedy planner: variables are produced immediately
• Fix QueryStreamTransmitter to not lose Continue signals and hang streaming operations
• Fix reboot bugs, update logging

Other Improvements

• Fix datetime-tz encode: send datetime as naive_utc instead of naive_local

• Fix datetimes: encode in seconds instead of millis

• Add query type to the query rows header. Implement connection/database bdd steps
  We update the server to server the updated protocol and return the type of the executed query as a part of the query row stream answer.
  We also fix the server's database manipulation code and implement additional bdd steps to pass the connection/database bdd tests.

• Fix within-transaction query concurrency

  We update the in-transaction query behaviour to result in a predictable and well-defined behaviour. In any applicable transaction, the following rules hold:

  1. Read queries are able to run concurrency and lazily, without limitation
  2. Write queries always execute eagerly, but answers are sent back lazily
  3. Schema queries and write queries interrupt all running read queries and finished write queries that are lazily sending answers

  As a user, we see the following - reads after writes or schema queries are just fine:

  let transaction = transaction(...);
  let writes_iter = transaction.query("insert $x isa person;").await.unwrap().into_rows();
  let reads_iter = transaction.query("match $x isa person;").await.unwrap().into_rows();
  
  // both are readable:
  writes_iter.next().await.unwrap();
  reads_iter.next().await.unwrap();
  

  In the other order, we will get an interrupt:

  let transaction = transaction(...);
  let reads_iter = transaction.query("match $x isa person;").await.unwrap().into_rows();
  let writes_iter = transaction.query("insert $x isa person;").await.unwrap().into_rows();
  
  // only writes are still available
  writes_iter.next().await.unwrap();
  assert!(reads_iter.next().await.is_err());