Dynamic Create Drop ColumnFamily

[中文版 (Chinese)]

1. Background

All along, SidePlugin does not support the dynamic creation of ColumnFamily，which is mainly due to performance considerations.There are few application scenarios for the dynamic creation of ColumnFamily，but it will incur some performance overhead. For example, MyRocks deals with this issue by using object-level TLS to improve the performance issues it causes.

In MyTopling, we eliminated this problem by completely disabling the dynamic creation of ColumnFamily.

Notes：

ToplingDB disables the dynamic creation of ColumnFamily by default. To enable it:

make DEBUG_LEVEL=0 # must clean to delete old *.o files
make -j`nproc` EXTRA_CXXFLAGS='-DROCKSDB_DYNAMIC_CREATE_CF' rocksdbjava DEBUG_LEVEL=0

2. dynamically creating ColumnFamily

We added this feature in 2023-06-08, which was originally only declared interfaces, but is now fully implemented:

2.1. C++

struct DB_MultiCF { // These are the only members that are available when you use the dynamic CF creation feature, ignoring the irrelevant code
  virtual Status CreateColumnFamily(const std::string& cfname,
                                    const std::string& spec,
                                    ColumnFamilyHandle**) = 0;
  virtual Status DropColumnFamily(const std::string& cfname) = 0;
  virtual Status DropColumnFamily(ColumnFamilyHandle*) = 0;
  DB* db = nullptr;
  // When the dynamic CF creation function is used, cf_handles cannot be securely accessed. The user manages the cf handles returned by CreateColumnFamily
  std::vector<ColumnFamilyHandle*> cf_handles;
};

demo code：dyna_new_cf.cc

2.2. Java

// Skipping the other methods, there is no DB_MultiCF class in Java, and user code needs to manage all CF objects on its own
public class SidePluginRepo extends RocksObject {
  public ColumnFamilyHandle createCF(RocksDB db, String cfname, String spec) throws RocksDBException;
  public void dropCF(RocksDB db, String cfname) throws RocksDBException;
  public void dropCF(RocksDB db, ColumnFamilyHandle cfh) throws RocksDBException;
}

2.3. Parameter

The parameter spec is used to specify the CF option, which is typically the CFOptions object name defined in JSON/YAML.

2.4. Why can't I just use DB::CreateColumnFamily

ToplingDB uses SidePlugin for configuration management. If DB::CreateColumnFamily is used directly, the created CF will not be perceived by SidePlugin.

• You can't see it on the WebView
• Various meta-objects and configuration parameters are also out of SidePlugin's control

3. Open the undefined ColumnFamily

The json/yaml file may have only the most basic CF defined, or no CF defined at all, and then dynamically create a lot of CFS on the fly, but the next time you Open DB, you have to specify all of them, which is almost impossible to do.

To solve this problem, we added a new parameter to json/yaml dyna_cf_opt(Complete configuration)：

## omit... Omit... Omit... Omit... Omit...
databases:
  db_bench_enterprise:
    method: DB::Open
    params:
      db_options: "$dbo"
      column_families:  # for multi cf db
        default: "$default"
      dyna_cf_opt: "$default"
      path: /dev/shm/db_bench_enterprise
open: db_bench_enterprise

Thus, for CF that exist in DB but are not defined in json/yaml, we can open them by obtaining the cf name through ListColumnFamilies and using dyna_cf_opt as CFOption.

In the original RocksDB interface, different CFS can specify different options, and ToplingDB's json/yaml uses the column_families parameter to achieve this goal. However, with dyna_cf_opt, all undefined CF's use the same Option. Therefore, different CF's cannot use different options.