[RTM] Quote reserved words in database queries #8813

ausi · 2017-11-20T23:46:18Z

As discussed in contao/core-bundle#1106 and the Contao Mumble call, we want to escape all column names that are reserved keywords.

In Contao 4 we will use \System::getContainer()->get('connection')->getDatabasePlatform()->quoteSingleIdentifier($name);. In Contao 3.5 (this pull request) we only quote rows.

There is no need to fix the install tool in 3.5, as it uses quoted column names already.

@leofeyer Do you have an automated workflow to port this pull request to Contao 4.4, or should I do it by hand?

leofeyer · 2017-11-21T07:39:03Z

I have a semi-automated way (creating split branches and then rebasing), so you don't have to do it by hand. But are the code changes actually the same in Contao 3 and Contao 4?

In Contao 4 we will use \System::getContainer()->get('connection')->getDatabasePlatform()->quoteSingleIdentifier($name);. In Contao 3.5 (this pull request) we only quote rows.

ausi · 2017-11-21T08:45:02Z

But are the code changes actually the same in Contao 3 and Contao 4?

Most of them are the same. The quoteColumnName method has to be different. Can you create a branch in the core-bundle where you merge these changes, and I add the additional modifications for Contao 4?

leofeyer · 2017-11-21T09:20:05Z

Sadly this is not how it works. I can only create the split branches after the PR has been merged into Contao 3.5 and a new version has been tagged. So I guess we have to do it manually. 😢

ausi · 2017-11-21T09:28:59Z

K, I do it :)

leofeyer · 2017-11-21T09:33:59Z

Thank you very much.

Description ----------- This is the port of contao/core#8813. Also see #1106 and contao/installation-bundle#70 Commits ------- 84c4746 Quote reserved words in database queries 7bf64b1 Use static instead of \Database as self-reference.

leofeyer · 2017-12-27T10:46:57Z

Before merging this we should check if this fix needs to be applied here as well (I think it does):

contao/core-bundle@4de1d20

ausi · 2017-12-27T10:54:37Z

Yes, I think so. Needs to be applied in both Mysql.php and Mysqli.php.

leofeyer · 2017-12-27T15:37:31Z

And this one as well: contao/core-bundle@541a8ca

leofeyer · 2017-12-28T08:38:40Z

Unfortunately, there are even more problems. Sometimes the identifier is not a column but an expression such as CONCAT(firstname, ' ', lastname), which leads to a "Column not found" exception.

Of course we could add something like this:

// Not an identifier
if (!preg_match('/^[a-z0-9_$.]+$/', $strName))
{
	return $strName;
}

However this would leave the column names inside the CONCAT() function unquoted.

EDIT: The regex should be A-Za-z0-9_$. as mentioned below.

leofeyer · 2017-12-28T08:44:41Z

@contao/developers I don't think that quoting identifiers is the correct solution as it leads to unforeseen issues. We have to discuss this topic again.

I am going to revert the changes in Contao 4.4.11 meanwhile.

ausi · 2017-12-28T14:13:44Z

I still think quoting identifiers is the right solution and your fix with the regular expression is correct. Although we should allow uppercase letters too: /^[a-z0-9_$.]+$/i. Is there a specific reason why you included the $ character?

However this would leave the column names inside the CONCAT() function unquoted.

Which is correct IMO. If you as a developer pass such a string to the model you have to do the quoting yourself, because you are passing an SQL expression and not a column name.

leofeyer · 2017-12-29T08:42:57Z

Apparently the dollar sign is a permitted character in an unquoted identifier:

https://dev.mysql.com/doc/refman/5.7/en/identifiers.html

From release announce: 4.4.9 (2017/12/14) * Fixes several minor PHP 7.2 related issues. 4.4.10 (2017/12/27) * Fixes a few minor issues including a problem with the comments bundle. 4.4.11 (2017/12/28) * Reverts the identifier quoting changes. MySQL 8 and MariaDB 10.2 In MySQL 8.0.2 and MariaDB 10.2.7 rows has been added as reserved word. Since we are using a field named rows in the tl_layout table, Contao is no longer compatible with the mentioned database versions. To avoid renaming the field in Contao and potentially causing theme import issues, we have added identifier quoting in version 4.4.10(*1). Unfortunately this led to several unforeseen problems(*2), so we had to revert the changes in version 4.4.11. (*1) contao/core-bundle#1262 (*2) contao/core#8813 (comment) 4.4.12 (2018/1/3) * Optimizes adding pages to the search index and fixes a few minor issues.

DanielSchwiperich · 2018-01-17T15:42:18Z

@ausi the PR also needs (if get recontinued) to take the creation into account:
https://github.com/madeyourday/contao-core/blob/e33738ab880063afc6453c17305479f58f98cf16/system/modules/core/drivers/DC_Table.php#L688

creating a new layout will otherwise not works because of the rows field in tl_layout
Same for contao/core-bundle#1262 if gets rereverted.

There is also another issue regarding mariadb, will create an issue for it in core-bundle

ausi · 2018-01-17T17:38:46Z

@DanielSchwiperich The set method is already handled, see https://github.com/contao/core/pull/8813/files#diff-a715adcad847fd730a703866083b4ae5R200

leofeyer · 2018-01-18T15:55:57Z

As discussed in Mumble on January 18th, 2018, we want to merge this with the changes from contao/core-bundle@4de1d20 and contao/core-bundle@541a8ca.

ausi · 2018-01-18T16:48:12Z

And add the regex check /^[a-z0-9_$.]+$/i

ausi · 2018-01-18T16:53:36Z

I looked again and think we must not include the . so the correct regex is: /^[a-z0-9_$]+$/i

leofeyer · 2018-01-18T17:01:18Z

What if it is tl_content.singleSRC?

ausi · 2018-01-18T17:04:38Z

Same as with CONCAT(...), the string tl_content.singleSRC is an expression and not a column name. The qouting would have to be done before. Like so 'tl_content.'.\Database::quoteColumnName('singleSRC');

leofeyer · 2018-01-18T19:54:16Z

Ok.

I wonder if the method should be called quoteIdentifier() instead of quoteColumnName(). The MySQL manual speaks of "unquoted identifiers" and the Doctrine method is called quoteIdentifier(), too. Any objections if I change it?

ausi · 2018-01-18T23:30:00Z

Any objections if I change it?

No, seems reasonable 👍
(but we should not forget to rename it in the Contao 4 pull request too)

ausi · 2018-01-19T13:45:48Z

system/modules/core/library/Contao/Database.php

 	{
-		if (in_array(strtolower($strName), array('rows'), true))
+		if ($strName == 'rows')


@leofeyer why did you remove the strtolower()?
This would make it no longer work for Rows or ROWS.

leofeyer · 2018-01-19T13:50:18Z

		// The identifier is quoted already
		if (strncmp($strName, '`', 1) === 0)
		{
			return $strName;
		}

		// Not an identifier
		if (!preg_match('/^[A-Za-z0-9_$]+$/', $strName))
		{
			return $strName;
		}

This is actually only required in Contao 4.4 as we are only quoting rows in Contao 3.5.

ausi · 2018-01-19T14:03:03Z

system/modules/core/library/Contao/Database/Mysql.php

@@ -95,7 +95,10 @@ protected function get_error()
 	 */
 	protected function find_in_set($strKey, $varSet, $blnIsField=false)
 	{
-		$strKey = \Database::quoteIdentifier($strKey);
+		if (preg_match('/^[A-Za-z0-9_$]+$/', $strKey))


This check is not necessary as it is handled by quoteIdentifier() itself.

ausi · 2018-01-19T14:03:18Z

system/modules/core/library/Contao/Database/Mysqli.php

@@ -85,7 +85,10 @@ protected function get_error()
 	 */
 	protected function find_in_set($strKey, $varSet, $blnIsField=false)
 	{
-		$strKey = \Database::quoteIdentifier($strKey);
+		if (preg_match('/^[A-Za-z0-9_$]+$/', $strKey))


leofeyer · 2018-01-19T14:13:10Z

Thank you @ausi.

ausi · 2018-01-19T14:15:29Z

system/modules/core/library/Contao/Database/Mysql.php

@@ -95,6 +95,8 @@ protected function get_error()
 	 */
 	protected function find_in_set($strKey, $varSet, $blnIsField=false)
 	{
+		$strKey = static::quoteIdentifier($strKey);
+
 		if ($blnIsField)
 		{
 			return "FIND_IN_SET(" . $strKey . ", " . $varSet . ")";


In this case we should probably use "FIND_IN_SET(" . $strKey . ", " . static::quoteIdentifier($varSet) . ")"; because $varSet is then an identifier too.

ausi · 2018-01-19T14:15:45Z

system/modules/core/library/Contao/Database/Mysqli.php

@@ -85,6 +85,8 @@ protected function get_error()
 	 */
 	protected function find_in_set($strKey, $varSet, $blnIsField=false)
 	{
+		$strKey = static::quoteIdentifier($strKey);
+
 		if ($blnIsField)
 		{
 			return "FIND_IN_SET(" . $strKey . ", " . $varSet . ")";


Quote reserved words in database queries

e33738a

leofeyer added the defect label Nov 21, 2017

leofeyer added this to the 3.5.32 milestone Nov 21, 2017

leofeyer assigned ausi Nov 21, 2017

ausi mentioned this pull request Dec 20, 2017

[RTM] Quote reserved words in database queries contao/core-bundle#1262

Merged

leofeyer added the up for discussion label Dec 28, 2017

leofeyer removed this from the 3.5.32 milestone Jan 15, 2018

leofeyer closed this Jan 15, 2018

leofeyer changed the base branch from master to 3.5 January 15, 2018 15:24

leofeyer reopened this Jan 15, 2018

DanielSchwiperich mentioned this pull request Jan 17, 2018

Contao from 4.5 on is not compatible to any version of mariadb anymore contao/core-bundle#1317

Closed

leofeyer removed the up for discussion label Jan 18, 2018

leofeyer added this to the 3.5.33 milestone Jan 18, 2018

Rename quoteColumnName() to quoteIdentifier().

5d5c2af

ausi commented Jan 19, 2018

View reviewed changes

leofeyer added 2 commits January 19, 2018 14:56

Only quote identifiers in the find_in_set() methods.

2eefd3f

Make the column check case-insensitive again.

57803a5

leofeyer approved these changes Jan 19, 2018

View reviewed changes

ausi commented Jan 19, 2018

View reviewed changes

Remove the redundant regex check.

3b692c4

leofeyer merged commit db8de81 into contao:3.5 Jan 19, 2018

ausi commented Jan 19, 2018

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RTM] Quote reserved words in database queries #8813

[RTM] Quote reserved words in database queries #8813

ausi commented Nov 20, 2017

leofeyer commented Nov 21, 2017 •

edited

Loading

ausi commented Nov 21, 2017

leofeyer commented Nov 21, 2017

ausi commented Nov 21, 2017

leofeyer commented Nov 21, 2017

leofeyer commented Dec 27, 2017

ausi commented Dec 27, 2017

leofeyer commented Dec 27, 2017

leofeyer commented Dec 28, 2017 •

edited

Loading

leofeyer commented Dec 28, 2017

ausi commented Dec 28, 2017

leofeyer commented Dec 29, 2017

DanielSchwiperich commented Jan 17, 2018

ausi commented Jan 17, 2018

leofeyer commented Jan 18, 2018 •

edited

Loading

ausi commented Jan 18, 2018

ausi commented Jan 18, 2018 •

edited

Loading

leofeyer commented Jan 18, 2018

ausi commented Jan 18, 2018

leofeyer commented Jan 18, 2018

ausi commented Jan 18, 2018

ausi Jan 19, 2018

leofeyer commented Jan 19, 2018

ausi Jan 19, 2018

ausi Jan 19, 2018

leofeyer commented Jan 19, 2018

ausi Jan 19, 2018

ausi Jan 19, 2018

[RTM] Quote reserved words in database queries #8813

[RTM] Quote reserved words in database queries #8813

Conversation

ausi commented Nov 20, 2017

leofeyer commented Nov 21, 2017 • edited Loading

ausi commented Nov 21, 2017

leofeyer commented Nov 21, 2017

ausi commented Nov 21, 2017

leofeyer commented Nov 21, 2017

leofeyer commented Dec 27, 2017

ausi commented Dec 27, 2017

leofeyer commented Dec 27, 2017

leofeyer commented Dec 28, 2017 • edited Loading

leofeyer commented Dec 28, 2017

ausi commented Dec 28, 2017

leofeyer commented Dec 29, 2017

DanielSchwiperich commented Jan 17, 2018

ausi commented Jan 17, 2018

leofeyer commented Jan 18, 2018 • edited Loading

ausi commented Jan 18, 2018

ausi commented Jan 18, 2018 • edited Loading

leofeyer commented Jan 18, 2018

ausi commented Jan 18, 2018

leofeyer commented Jan 18, 2018

ausi commented Jan 18, 2018

ausi Jan 19, 2018

Choose a reason for hiding this comment

leofeyer commented Jan 19, 2018

ausi Jan 19, 2018

Choose a reason for hiding this comment

ausi Jan 19, 2018

Choose a reason for hiding this comment

leofeyer commented Jan 19, 2018

ausi Jan 19, 2018

Choose a reason for hiding this comment

ausi Jan 19, 2018

Choose a reason for hiding this comment

leofeyer commented Nov 21, 2017 •

edited

Loading

leofeyer commented Dec 28, 2017 •

edited

Loading

leofeyer commented Jan 18, 2018 •

edited

Loading

ausi commented Jan 18, 2018 •

edited

Loading