Add a writing API and implement the md_apply_delta API in terms of it (…

…#30)

* Create basic entrypoint API for creating a read-write handle on an existing metadata image.

* Split up headers and make sure dnmd.h can build if it is included as the first #include.

* Add in support for tear-offs and centralize refcounting in a base "controlling IUnknown" type to avoid having to have both ref-counting rules (composed and otherwise) in the same object.

* Provide tear-off for emit

* Move IUnknown implementation to be provided by a template base type instead of by a macro.

* Fix build with new headers

* Fix some simple refcounting errors and typos found with some basic smoke testing.

* Make dnmd.h not have any dependencies on any other headers being implicitly included before it is included.

* Ensure we fully clean-up all allocations by the editor when destroying the handle.

* Implement proof-of-concept API for creating an indirection table to validate basic flow of the editor API design.

* Implement some basic writing logic and fix query context creation to allocate memory if a writable query is created over a currently-readonly table in an editable context.

* Update based on rebase.

* Finish implementing table resizing.

* Add include lost in rebase

* Don't include dncp in dnmd.hpp. DNMD should not depend on dncp at all.

* Add set APIs for heap columns

* Use create_cursor directly for less overhead.

* Add row insertion and creation of indirection tables.

* Add support for creating new tables in the append method if a table does not exist initially.

* Add internal API for copying heaps over from another image. Logic ported from the mddump_delta_support branch in the main repo.

* Remove fixed TODO and add an assert (this was fixed in the caller)

* Fix calculating the writable data address.

* Add table sorting validation checks

Validate that calls to md_find_range_from_cursor only try to find a range for the primary key column. Calling md_find_range_from_cursor with a non-primary-key on a sorted table causes inconsistent behavior.

Validate that a sorted table is still sorted when key columns are written to. This design imposes a requirement on the user to write the columns for a new row in order of the sorting keys, but it also enables the user to not have to consider if the table is sorted (and more importantly, we don't need to expose that concept to the user).

* Remove completed TODOs

* Fix build breaks

* Refactor check for sorted blob columns

* Handle edge cases for heaps (no heap and too large a heap)

* Start putting real method bodies into the IMetaDataEmit API.

* Add string splitting and add more emit apis

* Make mdhandle_t images always editable, lazily mark as edited. Refactor editor and delta APIs to have a better split.

* Fix build failures in emit

* Implement a few more emit APIs.

* Fix comparison

* More APIs and fix build

* Hide some struct definitions in source files instead of headers. Make the editor object lazily initialized and make "editor != NULL" the mark for "is this image edited".

* Style cleanup.

* First pass implementing delta application API.

* Various fixes to get minimal test passing

* Various fixes

* Fix Linux build and fix one place where I missed a malloc call.

* Make sure we always initialize list columns, even in the "add a child element" case.

Re-create the query context when we're validating sort order.

* Fix setting up the indirection table (and pointing the parent table to it)
Add some comments about future required work around handling already-created indirection tables.

* Fix bugs relating to maintaining the indirection tables and shifting row references.

We now correctly apply deltas that require us to introduce and update indirection tables. As a result, our delta application is fully-featured (to my knowledge).

* Add comment about adding to the composite enc log

* Modify table allocation and heap resizing to ensure that flags and table layouts in memory are consistent with how they'd be represented on disk (simplifies saving).

* Add validation of Enc edit chains.

* Fix creating the indirection table now that we size it correctly based on the actual target table size.

Also, move insert_row_into_table to use a 1-based row index. It was the only place using a 0-based index, which was really confusing.

* Remove some unnecessary casts and add some casts to ensure that we widen before doing a possibly-overflowing operation.

* Remove changes in dnmd_interfaces. We'll make those changes in a follow-up PR

* Add some comments and move one API to be internal

* Clean up handling the high bit.

* Move the query context out to its own file and rename to "access" context as it now also has write support.

Move the writing functions out of query.c and in to write.c

* Fix appending for guid heap

* Refactor APIs that were used to implement delta application to be more self-contained in the API surface now that we can reach into cursors more easily.

* Disable binary search lookups when a row-add operation is in progress.
Reimplement sorting validation with cursors instead of query access contexts. It's significantly cleaner and easier to understand.

* Move list column initialization into the public row-add API. Now all columns will be initialized to traversable/readable values after all row-add calls.

* Don't validate sorting on column write when we're adding a row. Instead only validate at the end of the row-add operation.

* Rewrite how we add rows to tables so that we have a few simple operations that only do basic work like allocating, shifting, initializing, updating references (append, add before, add after) and one complex operation (add element to list) that handles the crazy logic with indirection tables.

With this design, it is not possible to insert rows into a table in such a way to break layout of the table and tables that need special handling only work with the special handling functions (and ones that don't are blocked from the special handling functions).

* Fix adding an item to a list in the following cases:

- The list is not the first empty list in a set of lists
- The list is not the only empty list at the end of the table.

* Ensure that the lists in the Param table are always sorted by their Sequence column.

* PR feedback.

* Fix Windows Release build failures.

* Clang/GCC warn (which we turn into error) for unused functions. static functions are included in this warning, but "inline" functions are not.

* Explicitly ignore composed value in release builds.

* Add missing const

* Change back to static and just suppress the warning

* Apply suggestions from code review

Co-authored-by: Aaron Robinson <[email protected]>

* More error checks.

* Add a qsort_s implementation adapted from CoreCLR's utilcode and use it to sort the parameter list.

We'll conditionally use our implementation when an implementation does not exist on the platform.

* Change signature of compare callback.

* MSVC's qsort_s is just ever so slightly incompatible with the standard definition, so just hard-code the sort that we need instead of fighting it and adding a ton of CMake goo.

* Revert changes to configure.cmake

* Change the quicksort out for a simple insertion-sort-like algorithm as we're likely do already be in order for most items.

* Build

* Apply suggestions from code review

Co-authored-by: Aaron Robinson <[email protected]>

---------

Co-authored-by: Aaron Robinson <[email protected]>

src/dnmd/CMakeLists.txt

@@ -1,11 +1,14 @@
 set(SOURCES
+  access.c
   bytes.c
+  deltas.c
+  editor.c
   entry.c
   query.c
   streams.c
   tables.c
   sig.c
-  deltas.c
+  write.c
 )
 
 set(HEADERS

src/dnmd/access.c

@@ -0,0 +1,90 @@
+#include "internal.h"
+
+bool create_access_context(mdcursor_t* cursor, col_index_t col_idx, uint32_t row_count, bool make_writable, access_cxt_t* acxt)
+{
+    assert(acxt != NULL);
+    mdtable_t* table = CursorTable(cursor);
+    if (table == NULL)
+        return false;
+
+    uint32_t row = CursorRow(cursor);
+    if (row == 0 || row > table->row_count)
+        return false;
+
+    uint8_t idx = col_to_index(col_idx, table);
+    assert(idx < table->column_count);
+
+    // Metadata row indexing is 1-based.
+    row--;
+    acxt->table = table;
+    acxt->col_details = table->column_details[idx];
+
+    // Compute the offset into the first row.
+    uint32_t offset = ExtractOffset(acxt->col_details);
+
+    if (make_writable)
+    {
+        acxt->writable_data = get_writable_table_data(table, make_writable);
+        acxt->writable_data = acxt->writable_data + (row * table->row_size_bytes) + offset;
+    }
+    else
+    {
+        acxt->writable_data = NULL;
+    }
+
+    acxt->start = acxt->data = table->data.ptr + (row * table->row_size_bytes) + offset;
+
+    // Compute the beginning of the row after the last valid row.
+    uint32_t last_row = row + row_count;
+    if (last_row > table->row_count)
+        last_row = table->row_count;
+    acxt->end = table->data.ptr + (last_row * table->row_size_bytes);
+
+    // Limit the data read to the width of the column
+    acxt->data_len_col = (acxt->col_details & mdtc_b2) ? 2 : 4;
+    acxt->data_len = acxt->data_len_col;
+
+    // Compute the next row stride. Take the total length and substract
+    // the data length for the column to get at the next row's column.
+    acxt->next_row_stride = table->row_size_bytes - acxt->data_len_col;
+    return true;
+}
+
+bool read_column_data(access_cxt_t* acxt, uint32_t* data)
+{
+    assert(acxt != NULL && data != NULL);
+    *data = 0;
+
+    if (acxt->writable_data != NULL)
+        acxt->writable_data += (acxt->col_details & mdtc_b2) ? 2 : 4;
+
+    return (acxt->col_details & mdtc_b2)
+        ? read_u16(&acxt->data, &acxt->data_len, (uint16_t*)data)
+        : read_u32(&acxt->data, &acxt->data_len, data);
+}
+
+bool write_column_data(access_cxt_t* acxt, uint32_t data)
+{
+    assert(acxt != NULL && acxt->writable_data != NULL);
+
+    acxt->data += (acxt->col_details & mdtc_b2) ? 2 : 4;
+
+    return (acxt->col_details & mdtc_b2)
+        ? write_u16(&acxt->writable_data, &acxt->data_len, (uint16_t)data)
+        : write_u32(&acxt->writable_data, &acxt->data_len, data);
+}
+
+bool next_row(access_cxt_t* acxt)
+{
+    assert(acxt != NULL);
+    // We will only traverse correctly if we've already read the column in this row.
+    assert(acxt->data_len == 0);
+    acxt->data += acxt->next_row_stride;
+
+    if (acxt->writable_data != NULL)
+        acxt->writable_data += acxt->next_row_stride;
+
+    // Restore the data length of the column data.
+    acxt->data_len = acxt->data_len_col;
+    return acxt->data < acxt->end;
+}

src/dnmd/bytes.c

@@ -73,6 +73,47 @@ static bool read_le(uint8_t const** data, size_t* data_len, void* o, size_t o_si
     return true;
 }
 
+// This is a little-endian format in the physical form.
+static bool write_le(uint8_t** data, size_t* data_len, uint64_t o, size_t o_size_to_write)
+{
+    assert(data != NULL && data_len != NULL);
+    if (*data_len < o_size_to_write)
+        return false;
+
+    uint8_t* d = *data;
+    switch(o_size_to_write)
+    {
+    case 8:
+        *d++ = (uint8_t)o;
+        *d++ = (uint8_t)(o >> 8);
+        *d++ = (uint8_t)(o >> 16);
+        *d++ = (uint8_t)(o >> 24);
+        *d++ = (uint8_t)(o >> 32);
+        *d++ = (uint8_t)(o >> 40);
+        *d++ = (uint8_t)(o >> 48);
+        *d++ = (uint8_t)(o >> 56);
+        break;
+    case 4:
+        *d++ = (uint8_t)o;
+        *d++ = (uint8_t)(o >> 8);
+        *d++ = (uint8_t)(o >> 16);
+        *d++ = (uint8_t)(o >> 24);
+        break;
+    case 2:
+        *d++ = (uint8_t)o;
+        *d++ = (uint8_t)(o >> 8);
+        break;
+    case 1:
+        *d++ = (uint8_t)o;
+        break;
+    default:
+        return false;
+    }
+    *data = d;
+    *data_len -= o_size_to_write;
+    return true;
+}
+
 bool read_u8(uint8_t const** data, size_t* data_len, uint8_t* o)
 {
     return read_le(data, data_len, o, sizeof(*o));
@@ -113,6 +154,46 @@ bool read_i64(uint8_t const** data, size_t* data_len, int64_t* o)
     return read_le(data, data_len, o, sizeof(*o));
 }
 
+bool write_u8(uint8_t** data, size_t* data_len, uint8_t o)
+{
+    return write_le(data, data_len, o, sizeof(o));
+}
+
+bool write_i8(uint8_t** data, size_t* data_len, int8_t o)
+{
+    return write_le(data, data_len, o, sizeof(o));
+}
+
+bool write_u16(uint8_t** data, size_t* data_len, uint16_t o)
+{
+    return write_le(data, data_len, o, sizeof(o));
+}
+
+bool write_i16(uint8_t** data, size_t* data_len, int16_t o)
+{
+    return write_le(data, data_len, o, sizeof(o));
+}
+
+bool write_u32(uint8_t** data, size_t* data_len, uint32_t o)
+{
+    return write_le(data, data_len, o, sizeof(o));
+}
+
+bool write_i32(uint8_t** data, size_t* data_len, int32_t o)
+{
+    return write_le(data, data_len, o, sizeof(o));
+}
+
+bool write_u64(uint8_t** data, size_t* data_len, uint64_t o)
+{
+    return write_le(data, data_len, o, sizeof(o));
+}
+
+bool write_i64(uint8_t** data, size_t* data_len, int64_t o)
+{
+    return write_le(data, data_len, o, sizeof(o));
+}
+
 // II.23.2
 // This is a big-endian format in the physical form.
 bool decompress_u32(uint8_t const** data, size_t* data_len, uint32_t* o)