Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat isolation mode #358

Merged
merged 48 commits into from
Dec 8, 2023
Merged
Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
30a38c0
Reproduce Many to One issue
fogelito Nov 5, 2023
2d6a8c3
Add isolation mode constants and setters
abnegate Nov 7, 2023
ccff2ef
Add tenant insertion + querying
abnegate Nov 10, 2023
9da5b28
Delete isolation mode DBs
abnegate Nov 13, 2023
18a0485
Optional database for create/exists/delete
abnegate Nov 13, 2023
0cda563
Format
abnegate Nov 13, 2023
9be6cb0
Simply table sharing setters
abnegate Nov 14, 2023
f9da8fc
Cache fixes
abnegate Nov 15, 2023
bbeecde
Test fixes
abnegate Nov 16, 2023
6a35631
Merge remote-tracking branch 'origin/main' into feat-isolation-mode
abnegate Nov 17, 2023
f250d07
Format
abnegate Nov 17, 2023
2902e70
Default null tenant
abnegate Nov 17, 2023
b7bf0c5
Fix isolation modes + parallel workflow
abnegate Nov 17, 2023
becd2d5
Fix test workflow
abnegate Nov 17, 2023
4b825ec
Fix unit test workflow
abnegate Nov 17, 2023
025eb5e
Postgres isolation modes
abnegate Nov 20, 2023
3813cdd
WIP SQLite isolation modes
abnegate Nov 20, 2023
5550a7c
Fix SQLite isolation modes
abnegate Nov 22, 2023
c3f1d2e
Mongo isolation modes
abnegate Nov 22, 2023
b7bcc9b
Merge remote-tracking branch 'origin/main' into feat-isolation-mode
abnegate Nov 22, 2023
01443a3
Fix image tag
abnegate Nov 23, 2023
92acf5e
Conditional indexes for better query performance
abnegate Nov 24, 2023
c363b5a
Format
abnegate Nov 24, 2023
3183694
Use integer for tenant column
abnegate Nov 24, 2023
baea354
deleteCached* -> purgeCached*
abnegate Nov 24, 2023
0d65504
Tabs -> spaces
abnegate Nov 24, 2023
32b10a3
Automate default attributes/index counts
abnegate Nov 24, 2023
d057eaf
Add back size comments
abnegate Nov 24, 2023
c365f85
Format
abnegate Nov 24, 2023
52fc889
Fix attribute checks
abnegate Nov 24, 2023
3818118
Fix int type checks account for STRINGIFY_FETCHES
abnegate Nov 24, 2023
576461b
Treat _tenant as a string to allow structure validation account for s…
abnegate Nov 24, 2023
04e2333
Optimize unique index order for tenant uid
abnegate Nov 27, 2023
3c9da71
Assert fetched tenant matches inserted
abnegate Nov 27, 2023
f838c9d
Format
abnegate Nov 27, 2023
22f9009
Merge remote-tracking branch 'origin/main' into feat-isolation-mode
abnegate Nov 27, 2023
8756141
Allow null tenant
abnegate Nov 27, 2023
4b0f897
Add exception check
abnegate Nov 27, 2023
1dd550b
Optimize permission indexes
abnegate Nov 27, 2023
ba6d104
Optimize permission index order
abnegate Nov 27, 2023
989eaf6
Remove index
abnegate Nov 27, 2023
c1341e2
Fix index count test
abnegate Nov 27, 2023
fbdacca
Don't cover tenant on fulltext indexes
abnegate Nov 30, 2023
0361929
Format
abnegate Nov 30, 2023
1711f31
Fix sqlite test
abnegate Nov 30, 2023
7fd5070
Format
abnegate Nov 30, 2023
426476b
Add tenant + internal ID index for pagination query speed
abnegate Dec 8, 2023
5398922
Fix syntax
abnegate Dec 8, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 93 additions & 33 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -1,40 +1,100 @@
name: "Tests"

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

env:
IMAGE: databases-dev
CACHE_KEY: databases-dev-${{ github.event.pull_request.head.sha }}

on: [pull_request]

jobs:
tests:
name: Unit & E2E
setup:
name: Setup & Build Docker Image
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Build Docker Image
uses: docker/build-push-action@v3
with:
context: .
push: false
tags: ${{ env.IMAGE }}
load: true
cache-from: type=gha
cache-to: type=gha,mode=max
outputs: type=docker,dest=/tmp/${{ env.IMAGE }}.tar

- name: Cache Docker Image
uses: actions/cache@v3
with:
key: ${{ env.CACHE_KEY }}
path: /tmp/${{ env.IMAGE }}.tar

unit_test:
name: Unit Test
runs-on: ubuntu-latest
needs: setup

steps:
- name: Checkout repository
uses: actions/checkout@v3
with:
fetch-depth: 2
submodules: recursive

- run: git checkout HEAD^2

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Build image
uses: docker/build-push-action@v3
with:
context: .
push: false
tags: database-dev
load: true
cache-from: type=gha
cache-to: type=gha,mode=max

- name: Start Databases
run: |
docker compose up -d
sleep 10

- name: Run Tests
run: docker compose exec -T tests vendor/bin/phpunit --configuration phpunit.xml

- name: Check Coverage
run: docker compose exec -T tests vendor/bin/coverage-check tmp/clover.xml 90
- name: checkout
uses: actions/checkout@v2

- name: Load Cache
uses: actions/cache@v3
with:
key: ${{ env.CACHE_KEY }}
path: /tmp/${{ env.IMAGE }}.tar
fail-on-cache-miss: true

- name: Load and Start Services
run: |
docker load --input /tmp/${{ env.IMAGE }}.tar
docker compose up -d
sleep 10

- name: Run Unit Tests
run: docker compose exec tests vendor/bin/phpunit /usr/src/code/tests/unit

adapter_test:
name: Adapter Tests
runs-on: ubuntu-latest
needs: setup
strategy:
fail-fast: false
matrix:
adapter:
[
MariaDB,
MySQL,
Postgres,
SQLite,
MongoDB,
]

steps:
- name: checkout
uses: actions/checkout@v3

- name: Load Cache
uses: actions/cache@v3
with:
key: ${{ env.CACHE_KEY }}
path: /tmp/${{ env.IMAGE }}.tar
fail-on-cache-miss: true

- name: Load and Start Services
run: |
docker load --input /tmp/${{ env.IMAGE }}.tar
docker compose up -d
sleep 10

- name: Run ${{matrix.service}} Tests
run: docker compose exec -T tests vendor/bin/phpunit /usr/src/code/tests/e2e/Adapter/${{matrix.adapter}}Test.php --debug
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,10 +244,10 @@ $database->setNamespace(
);

// Get default database
$database->getDefaultDatabase();
$database->getDatabase();

// Sets default database
$database->setDefaultDatabase(
$database->setDatabase(
name: 'dbName'
);

Expand Down Expand Up @@ -402,7 +402,7 @@ $database->deleteCollection(
);

// Delete cached documents of a collection
$database->deleteCachedCollection(
$database->purgeCachedCollection(
collection: 'users'
);
```
Expand Down Expand Up @@ -836,7 +836,7 @@ $database->deleteDocument(

// Delete a cached document
Note: Cached Documents or Collections are automatically deleted when a document or collection is updated or deleted
$database->deleteCachedDocument(
$database->purgeCachedDocument(
collection: 'movies',
id: $document->getId()
);
Expand Down
2 changes: 1 addition & 1 deletion bin/tasks/index.php
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@
return;
}

$database->setDefaultDatabase($name);
$database->setDatabase($name);
$database->setNamespace($namespace);

Console::info("For query: greaterThan(created, 2010-01-01 05:00:00)', 'equal(genre,travel)");
Expand Down
16 changes: 8 additions & 8 deletions bin/tasks/load.php
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@
$pdo = new PDO("mysql:host={$dbHost};port={$dbPort};charset=utf8mb4", $dbUser, $dbPass, MariaDB::getPDOAttributes());

$database = new Database(new MariaDB($pdo), $cache);
$database->setDefaultDatabase($name);
$database->setDatabase($name);
$database->setNamespace($namespace);

// Outline collection schema
Expand Down Expand Up @@ -89,7 +89,7 @@
$pdo = $pool->get();

$database = new Database(new MariaDB($pdo), $cache);
$database->setDefaultDatabase($name);
$database->setDatabase($name);
$database->setNamespace($namespace);

// Each coroutine loads 1000 documents
Expand All @@ -116,7 +116,7 @@
$pdo = new PDO("mysql:host={$dbHost};port={$dbPort};charset=utf8mb4", $dbUser, $dbPass, MySQL::getPDOAttributes());

$database = new Database(new MySQL($pdo), $cache);
$database->setDefaultDatabase($name);
$database->setDatabase($name);
$database->setNamespace($namespace);

// Outline collection schema
Expand Down Expand Up @@ -150,7 +150,7 @@
$pdo = $pool->get();

$database = new Database(new MySQL($pdo), $cache);
$database->setDefaultDatabase($name);
$database->setDatabase($name);
$database->setNamespace($namespace);

// Each coroutine loads 1000 documents
Expand Down Expand Up @@ -178,7 +178,7 @@
);

$database = new Database(new Mongo($client), $cache);
$database->setDefaultDatabase($name);
$database->setDatabase($name);
$database->setNamespace($namespace);

// Outline collection schema
Expand All @@ -192,7 +192,7 @@
for ($i = 0; $i < $limit / 1000; $i++) {
go(function () use ($client, $faker, $name, $namespace, $cache) {
$database = new Database(new Mongo($client), $cache);
$database->setDefaultDatabase($name);
$database->setDatabase($name);
$database->setNamespace($namespace);

// Each coroutine loads 1000 documents
Expand Down Expand Up @@ -227,8 +227,8 @@

function createSchema(Database $database): void
{
if ($database->exists($database->getDefaultDatabase())) {
$database->delete($database->getDefaultDatabase());
if ($database->exists($database->getDatabase())) {
$database->delete($database->getDatabase());
}
$database->create();

Expand Down
6 changes: 3 additions & 3 deletions bin/tasks/query.php
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
);

$database = new Database(new Mongo($client), $cache);
$database->setDefaultDatabase($name);
$database->setDatabase($name);
$database->setNamespace($namespace);
break;

Expand All @@ -58,7 +58,7 @@
$pdo = new PDO("mysql:host={$dbHost};port={$dbPort};charset=utf8mb4", $dbUser, $dbPass, MariaDB::getPDOAttributes());

$database = new Database(new MariaDB($pdo), $cache);
$database->setDefaultDatabase($name);
$database->setDatabase($name);
$database->setNamespace($namespace);
break;

Expand All @@ -71,7 +71,7 @@
$pdo = new PDO("mysql:host={$dbHost};port={$dbPort};charset=utf8mb4", $dbUser, $dbPass, MySQL::getPDOAttributes());

$database = new Database(new MySQL($pdo), $cache);
$database->setDefaultDatabase($name);
$database->setDatabase($name);
$database->setNamespace($namespace);
break;

Expand Down
5 changes: 4 additions & 1 deletion composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,10 @@
"psr-4": {"Utopia\\Database\\": "src/Database"}
},
"autoload-dev": {
"psr-4": {"Utopia\\Tests\\": "tests/Database"}
"psr-4": {
"Tests\\E2E\\": "tests/e2e",
"Tests\\Unit\\": "tests/unit"
}
},
"scripts": {
"build": [
Expand Down
30 changes: 15 additions & 15 deletions composer.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ services:

tests:
container_name: tests
image: database-dev
image: databases-dev
build:
context: .
networks:
Expand Down
7 changes: 5 additions & 2 deletions phpunit.xml
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,11 @@
processIsolation="false"
stopOnFailure="false">
<testsuites>
<testsuite name="Application Test Suite">
<directory>./tests/</directory>
<testsuite name="unit">
<directory>./tests/unit</directory>
</testsuite>
<testsuite name="e2e">
<directory>./tests/e2e/Adapter</directory>
</testsuite>
</testsuites>
<coverage processUncoveredFiles="true">
Expand Down
Loading
Loading