Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat query gen 2.3 #163

Merged
merged 31 commits into from
Aug 14, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
4b0c0e2
WIP: Query syntax V2
Meldiron Jul 15, 2022
b66e052
Finish Queries V2 implementation
Meldiron Jul 15, 2022
6703819
WIP: Fix current tests
Meldiron Jul 15, 2022
928fb86
Tests fixed
Meldiron Jul 15, 2022
847fff1
Tests for new helpers
Meldiron Jul 15, 2022
10384f3
Refactor Queries parser
Meldiron Jul 22, 2022
091e10c
feat: torstens review
TorstenDittmann Jul 22, 2022
b868901
feat: use const for used characters
TorstenDittmann Jul 22, 2022
d3e287c
tests: add more query tests
TorstenDittmann Jul 22, 2022
4b12d17
Merge branch 'main' of https://github.com/utopia-php/database into fe…
TorstenDittmann Jul 22, 2022
8172d66
tests: fix tests for keywords
TorstenDittmann Jul 22, 2022
8abcc16
feat: add bench
TorstenDittmann Jul 22, 2022
7bdd754
add typing
TorstenDittmann Jul 22, 2022
c500c4e
Improve performance of parser
Meldiron Jul 23, 2022
029dad4
Improve performance of query parser again
Meldiron Jul 23, 2022
7824eea
Merge pull request #158 from utopia-php/feat-query-gen-2.1-bench
Meldiron Jul 24, 2022
1ff03f3
Rename from operator to method
Meldiron Jul 24, 2022
fd049f4
PR review changes
Meldiron Jul 24, 2022
2154dfa
Fix backshash tests
Meldiron Jul 24, 2022
b866d3d
Escaping bug fix
Meldiron Jul 25, 2022
1319d56
Separate string support from array support
Meldiron Jul 25, 2022
3330976
Removed benchmarks
Meldiron Jul 25, 2022
f01a300
Remove more benchmark stuff
Meldiron Jul 25, 2022
88c5772
Fix yet another edge case with backslashes
Meldiron Jul 25, 2022
74435b5
Commented out alias support.
Meldiron Jul 27, 2022
34e492b
Merge branch 'main' into feat-query-gen-2.2
stnguyen90 Aug 8, 2022
97754fb
Update Query class
stnguyen90 Aug 8, 2022
4348f07
Update Database find methods to only use queries
stnguyen90 Aug 9, 2022
3f7af16
Update Queries Validator to validate all queries
stnguyen90 Aug 11, 2022
edd8404
Rename QueryValidator to Query
stnguyen90 Aug 11, 2022
d5de1d6
Merge branch 'main' into feat-query-gen-2.3
stnguyen90 Aug 14, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ static::getDatabase()->createDocument('movies', new Document([

```php
$documents = static::getDatabase()->find('movies', [
new Query('year', Query::TYPE_EQUAL, [2019]),
Query::equal('year', [2019]),
]);
```

Expand Down
24 changes: 15 additions & 9 deletions bin/tasks/query.php
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
/**
* @var CLI
*/ global $cli;

use Faker\Factory;
use MongoDB\Client;
use Utopia\Cache\Cache;
Expand All @@ -17,19 +18,21 @@
use Utopia\Database\Validator\Authorization;
use Utopia\Validator\Numeric;
use Utopia\Validator\Text;

$cli
->task('query')
->desc('Query mock data')
->param('adapter', '', new Text(0), 'Database adapter', false)
->param('name', '', new Text(0), 'Name of created database.', false)
->param('limit', 25, new Numeric(), 'Limit on queried documents', true)
->action(function ($adapter, $name, $limit) {
->action(function (string $adapter, string $name, int $limit) {
$database = null;

switch ($adapter) {
case 'mongodb':
$options = ["typeMap" => ['root' => 'array', 'document' => 'array', 'array' => 'array']];
$client = new Client('mongodb://mongo/',
$client = new Client(
'mongodb://mongo/',
[
'username' => 'root',
'password' => 'example',
Expand Down Expand Up @@ -130,7 +133,8 @@
fclose($f);
});

function runQueries($database, $limit) {
function runQueries(Database $database, int $limit)
{
$results = [];
// Recent travel blogs
$query = ["created.greater(1262322000)", "genre.equal('travel')"];
Expand All @@ -151,21 +155,23 @@ function runQueries($database, $limit) {
return $results;
}

function addRoles($faker, $count) {
for ($i=0; $i < $count; $i++) {
function addRoles($faker, $count)
{
for ($i = 0; $i < $count; $i++) {
Authorization::setRole($faker->numerify('user####'));
}
return count(Authorization::getRoles());
}

function runQuery($query, $database, $limit) {
Console::log('Running query: ['.implode(', ', $query).']');
$query = array_map(function($q) {
function runQuery(array $query, Database $database, int $limit)
{
Console::log('Running query: [' . implode(', ', $query) . ']');
$query = array_map(function ($q) {
return Query::parse($q);
}, $query);

$start = microtime(true);
$documents = $database->find('articles', $query, $limit);
$database->find('articles', array_merge($query, [Query::limit($limit)]));
$time = microtime(true) - $start;
Console::success("{$time} s");
return $time;
Expand Down
14 changes: 7 additions & 7 deletions composer.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions src/Database/Adapter.php
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,7 @@ abstract public function deleteDocument(string $collection, string $id): bool;
* @param int $offset
* @param array $orderAttributes
* @param array $orderTypes
* @param array $cursor Array copy of document used for before/after pagination
* @param array $cursor
* @param string $cursorDirection
*
* @return Document[]
Expand Down Expand Up @@ -459,11 +459,11 @@ abstract public function getSupportForCasting(): bool;
* @throws Exception
* @return string
*/
public function filter(string $value):string
public function filter(string $value): string
{
$value = preg_replace("/[^A-Za-z0-9\_\-]/", '', $value);

if(\is_null($value)) {
if (\is_null($value)) {
throw new Exception('Failed to filter key');
}

Expand Down
63 changes: 32 additions & 31 deletions src/Database/Adapter/MariaDB.php
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ public function createCollection(string $name, array $attributes = [], array $in
$indexAttributes = $index->getAttribute('attributes');
foreach ($indexAttributes as $nested => $attribute) {
$indexLength = $index->getAttribute('lengths')[$key] ?? '';
$indexLength = (empty($indexLength)) ? '' : '('.(int)$indexLength.')';
$indexLength = (empty($indexLength)) ? '' : '(' . (int)$indexLength . ')';
$indexOrder = $index->getAttribute('orders')[$key] ?? '';
$indexAttribute = $this->filter($attribute);

Expand Down Expand Up @@ -776,6 +776,7 @@ public function deleteDocument(string $collection, string $id): bool
* @param array $orderTypes
* @param array $cursor
* @param string $cursorDirection
*
* @return Document[]
* @throws Exception
* @throws PDOException
Expand All @@ -795,8 +796,8 @@ public function find(string $collection, array $queries = [], int $limit = 25, i
}, $orderAttributes);

$hasIdAttribute = false;
foreach($orderAttributes as $i => $attribute) {
if($attribute === '_uid') {
foreach ($orderAttributes as $i => $attribute) {
if ($attribute === '_uid') {
$hasIdAttribute = true;
}

Expand All @@ -805,21 +806,21 @@ public function find(string $collection, array $queries = [], int $limit = 25, i

// Get most dominant/first order attribute
if ($i === 0 && !empty($cursor)) {
$orderOperatorInternalId = Query::TYPE_GREATER; // To preserve natural order
$orderOperator = $orderType === Database::ORDER_DESC ? Query::TYPE_LESSER : Query::TYPE_GREATER;
$orderMethodInternalId = Query::TYPE_GREATER; // To preserve natural order
$orderMethod = $orderType === Database::ORDER_DESC ? Query::TYPE_LESSER : Query::TYPE_GREATER;

if ($cursorDirection === Database::CURSOR_BEFORE) {
$orderType = $orderType === Database::ORDER_ASC ? Database::ORDER_DESC : Database::ORDER_ASC;
$orderOperatorInternalId = $orderType === Database::ORDER_ASC ? Query::TYPE_LESSER : Query::TYPE_GREATER;
$orderOperator = $orderType === Database::ORDER_DESC ? Query::TYPE_LESSER : Query::TYPE_GREATER;
$orderMethodInternalId = $orderType === Database::ORDER_ASC ? Query::TYPE_LESSER : Query::TYPE_GREATER;
$orderMethod = $orderType === Database::ORDER_DESC ? Query::TYPE_LESSER : Query::TYPE_GREATER;
}

$where[] = "(
table_main.{$attribute} {$this->getSQLOperator($orderOperator)} :cursor
table_main.{$attribute} {$this->getSQLOperator($orderMethod)} :cursor
OR (
table_main.{$attribute} = :cursor
AND
table_main._id {$this->getSQLOperator($orderOperatorInternalId)} {$cursor['$internalId']}
table_main._id {$this->getSQLOperator($orderMethodInternalId)} {$cursor['$internalId']}
)
)";
} else if ($cursorDirection === Database::CURSOR_BEFORE) {
Expand All @@ -832,20 +833,20 @@ public function find(string $collection, array $queries = [], int $limit = 25, i
// Allow after pagination without any order
if (empty($orderAttributes) && !empty($cursor)) {
$orderType = $orderTypes[0] ?? Database::ORDER_ASC;
$orderOperator = $cursorDirection === Database::CURSOR_AFTER ? ($orderType === Database::ORDER_DESC ? Query::TYPE_LESSER : Query::TYPE_GREATER
$orderMethod = $cursorDirection === Database::CURSOR_AFTER ? ($orderType === Database::ORDER_DESC ? Query::TYPE_LESSER : Query::TYPE_GREATER
) : ($orderType === Database::ORDER_DESC ? Query::TYPE_GREATER : Query::TYPE_LESSER
);
$where[] = "( table_main._id {$this->getSQLOperator($orderOperator)} {$cursor['$internalId']} )";
$where[] = "( table_main._id {$this->getSQLOperator($orderMethod)} {$cursor['$internalId']} )";
}

// Allow order type without any order attribute, fallback to the natural order (_id)
if(!$hasIdAttribute) {
if (!$hasIdAttribute) {
if (empty($orderAttributes) && !empty($orderTypes)) {
$order = $orderTypes[0] ?? Database::ORDER_ASC;
if ($cursorDirection === Database::CURSOR_BEFORE) {
$order = $order === Database::ORDER_ASC ? Database::ORDER_DESC : Database::ORDER_ASC;
}

$orders[] = 'table_main._id ' . $this->filter($order);
} else {
$orders[] = 'table_main._id ' . ($cursorDirection === Database::CURSOR_AFTER ? Database::ORDER_ASC : Database::ORDER_DESC); // Enforce last ORDER by '_id'
Expand All @@ -862,7 +863,7 @@ public function find(string $collection, array $queries = [], int $limit = 25, i

$conditions = [];
foreach ($query->getValues() as $key => $value) {
$conditions[] = $this->getSQLCondition('table_main.' . $query->getAttribute(), $query->getOperator(), ':attribute_' . $i . '_' . $key . '_' . $query->getAttribute(), $value);
$conditions[] = $this->getSQLCondition('table_main.' . $query->getAttribute(), $query->getMethod(), ':attribute_' . $i . '_' . $key . '_' . $query->getAttribute(), $value);
}
$condition = implode(' OR ', $conditions);
$where[] = empty($condition) ? '' : '(' . $condition . ')';
Expand All @@ -888,7 +889,7 @@ public function find(string $collection, array $queries = [], int $limit = 25, i
");

foreach ($queries as $i => $query) {
if ($query->getOperator() === Query::TYPE_SEARCH) continue;
if ($query->getMethod() === Query::TYPE_SEARCH) continue;
foreach ($query->getValues() as $key => $value) {
$stmt->bindValue(':attribute_' . $i . '_' . $key . '_' . $query->getAttribute(), $value, $this->getPDOType($value));
}
Expand Down Expand Up @@ -968,7 +969,7 @@ public function count(string $collection, array $queries = [], int $max = 0): in

$conditions = [];
foreach ($query->getValues() as $key => $value) {
$conditions[] = $this->getSQLCondition('table_main.' . $query->getAttribute(), $query->getOperator(), ':attribute_' . $i . '_' . $key . '_' . $query->getAttribute(), $value);
$conditions[] = $this->getSQLCondition('table_main.' . $query->getAttribute(), $query->getMethod(), ':attribute_' . $i . '_' . $key . '_' . $query->getAttribute(), $value);
}

$condition = implode(' OR ', $conditions);
Expand All @@ -991,7 +992,7 @@ public function count(string $collection, array $queries = [], int $max = 0): in
");

foreach ($queries as $i => $query) {
if ($query->getOperator() === Query::TYPE_SEARCH) continue;
if ($query->getMethod() === Query::TYPE_SEARCH) continue;
foreach ($query->getValues() as $key => $value) {
$stmt->bindValue(':attribute_' . $i . '_' . $key . '_' . $query->getAttribute(), $value, $this->getPDOType($value));
}
Expand Down Expand Up @@ -1037,7 +1038,7 @@ public function sum(string $collection, string $attribute, array $queries = [],

$conditions = [];
foreach ($query->getValues() as $key => $value) {
$conditions[] = $this->getSQLCondition('table_main.' . $query->getAttribute(), $query->getOperator(), ':attribute_' . $i . '_' . $key . '_' . $query->getAttribute(), $value);
$conditions[] = $this->getSQLCondition('table_main.' . $query->getAttribute(), $query->getMethod(), ':attribute_' . $i . '_' . $key . '_' . $query->getAttribute(), $value);
}

$where[] = implode(' OR ', $conditions);
Expand All @@ -1058,7 +1059,7 @@ public function sum(string $collection, string $attribute, array $queries = [],
");

foreach ($queries as $i => $query) {
if ($query->getOperator() === Query::TYPE_SEARCH) continue;
if ($query->getMethod() === Query::TYPE_SEARCH) continue;
foreach ($query->getValues() as $key => $value) {
$stmt->bindValue(':attribute_' . $i . '_' . $key . '_' . $query->getAttribute(), $value, $this->getPDOType($value));
}
Expand Down Expand Up @@ -1370,15 +1371,15 @@ protected function getSQLType(string $type, int $size, bool $signed = true): str
* Get SQL Conditions
*
* @param string $attribute
* @param string $operator
* @param string $method
* @param string $placeholder
* @param mixed $value
* @return string
* @throws Exception
*/
protected function getSQLCondition(string $attribute, string $operator, string $placeholder, $value): string
protected function getSQLCondition(string $attribute, string $method, string $placeholder, $value): string
{
switch ($operator) {
switch ($method) {
case Query::TYPE_SEARCH:
/**
* Replace reserved chars with space.
Expand All @@ -1393,20 +1394,21 @@ protected function getSQLCondition(string $attribute, string $operator, string $
return 'MATCH(' . $attribute . ') AGAINST(' . $this->getPDO()->quote($value) . ' IN BOOLEAN MODE)';

default:
return $attribute . ' ' . $this->getSQLOperator($operator) . ' ' . $placeholder; // Using `attrubute_` to avoid conflicts with custom names;
return $attribute . ' ' . $this->getSQLOperator($method) . ' ' . $placeholder; // Using `attrubute_` to avoid conflicts with custom names;
break;
}
}

/**
* Get SQL Operator
*
* @param string $operator
* @param string $method
* @return string
* @throws Exception
*/
protected function getSQLOperator(string $operator): string
protected function getSQLOperator(string $method): string
{
switch ($operator) {
switch ($method) {
case Query::TYPE_EQUAL:
return '=';

Expand All @@ -1426,7 +1428,8 @@ protected function getSQLOperator(string $operator): string
return '>=';

default:
throw new Exception('Unknown Operator:' . $operator);
throw new Exception('Unknown method:' . $method);
break;
}
}

Expand Down Expand Up @@ -1486,7 +1489,7 @@ protected function getSQLIndex(string $collection, string $id, string $type, ar
break;
}

return "CREATE {$type} `{$id}` ON `{$this->getDefaultDatabase()}`.`{$this->getNamespace()}_{$collection}` ( ". implode(', ', $attributes) ." )";
return "CREATE {$type} `{$id}` ON `{$this->getDefaultDatabase()}`.`{$this->getNamespace()}_{$collection}` ( " . implode(', ', $attributes) . " )";
}

/**
Expand Down Expand Up @@ -1582,7 +1585,7 @@ protected function getPDO()
/**
* Returns default PDO configuration
*/
public static function getPdoAttributes():array
public static function getPdoAttributes(): array
{
return [
PDO::ATTR_TIMEOUT => 3, // Specifies the timeout duration in seconds. Takes a value of type int.
Expand All @@ -1593,6 +1596,4 @@ public static function getPdoAttributes():array
PDO::ATTR_STRINGIFY_FETCHES => true // Returns all fetched data as Strings
];
}


}
Loading