Feature/suggestions #2447

Tezd · 2018-05-30T15:00:08Z

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

This PR adds autocomplete to client for base queries, types, engines, formats, and functions except combinators.

Please bear with me on this PR since i am not very familiar with C++ language and its toolchain. My apologies in advance.
P.S Can I use Clients sendQuery method for gathering data about dbs, tables?

Rewrite completion hash
Load functions, dbs, tables, columns using system tables with limit
Use threading to load completion data in order to not block client
Rewrite code to modern standard
Fix safety threads, fix memory leaks
Remove TST, use parser instead
Allow completion disabling
Implement max_block_size for system.tables and system.columns
Ensure compatibility with old servers (old layout of system database)

alexey-milovidov · 2018-05-31T12:58:12Z

dbms/src/Client/Completion.cpp

+        return h;
+    }
+
+    int init_hash_table(HashTable *ht, size_t size)


From what source this code was copy-pasted?

It is from https://github.com/mysql/mysql-server/blob/8.0/client/completion_hash.cc. I've put source in Completion.h file.

alexey-milovidov · 2018-05-31T12:58:22Z

dbms/src/Client/Completion.cpp

+
+namespace Completion
+{
+    static uint hashpjw(const char *arKey, uint nKeyLength)


Bad hash function.

alexey-milovidov · 2018-05-31T13:02:30Z

P.S Can I use Clients sendQuery method for gathering data about dbs, tables?

This is perfectly Ok, but care should be taken when there are huge amount of tables.
We have servers with about million of small StripeLog tables for chunks.
You can define a limit for maximum number of fetched tables. And probably do it in a separate thread.
Command line client should never stuck.

For example, it is a typical issue for MySQL CLI: when you have huge number of tables, it will startup slowly.

alexey-milovidov · 2018-05-31T13:24:34Z

dbms/src/Client/Completion.h

+
+#include <sys/types.h>
+
+//All of functionality for hash was taken from mysql-server project from completion_hash.cpp file


Missing license. The license is most likely to be incompatible.

Yeah, its only one way compatible. Will rewrite completion hash.

alexey-milovidov · 2018-05-31T13:26:27Z

dbms/src/Client/QueryParts.h

+        {(char *)"Expression"},
+        {(char *)"Set"},
+        //FUNCTIONS
+        {(char *)"plus"},


It will be difficult to support unless we will use FunctionFactory directly (or query system.functions table for looser coupling).

filimonov · 2018-06-15T07:25:13Z

dbms/programs/client/Client.cpp

+        std::vector<Block> blocks;
+
+        //preload all functions
+        sendQuery("SELECT name FROM system.functions", blocks);


Aggregation function can also have suffixes

Wont it be too much repetition and will pollute suggestions? For that reason i didn't want to include combinators, but can add them regardless.

May be you right.

filimonov · 2018-06-15T07:26:44Z

dbms/programs/client/Client.cpp

@@ -171,6 +242,9 @@ class Client : public Poco::Util::Application
    /// External tables info.
    std::list<ExternalTable> external_tables;

+    /// Suggestion limit for how many databases and tables to fetch
+    int suggestion_limit = 100;


why so low by default? For databases that can be reasonable, for tables - a bit too low

@filimonov what limit will you suggest? twice the amount? not really sure :(

I would say that having dozens of tables is quite common. Clickhouse have 24 system tables in default distribution. At that point I have 93 tables on my local clickhouse. And it can be very irritating when autocompletion works for "almost" every table.

Limit? May be 256 ? Even you you will have 64 columns per table, and 32 chars per column name it will give 512 Kb of memory, which is not a big number nowadays. Also 256 is more 'round' number :)

@filimonov Bumped limit to 256. One can always adjust it to higher/lower limits by adding --suggestion_limit= to your CLI client on startup ex. ./clickhouse client --suggestion_limit=512

filimonov · 2018-06-15T07:28:09Z

dbms/programs/client/Client.cpp

+                    delete implodedName;
+            }
+        }
+        delete query;


Also dictionary autocomplete is VERY helpful.

Check that:
https://github.com/filimonov/chc/blob/8bb7b4a1a7917ff5cb75f84e7b76091463f23ecc/autocomplete.go#L196

@filimonov Added dictionary name and attribute name as part of completion. Can add whole dictGetT() template as well. But I think there will be problems with complex keys.

filimonov · 2018-06-15T07:31:59Z

dbms/src/Client/QueryParts.h

+
+QUERYPART queryParts[] = {
+        // CREATE DATABASE, TABLE, VIEW
+        {(const char *)"CREATE"},


Good. You can also compare with https://github.com/filimonov/chc/blob/8bb7b4a1a7917ff5cb75f84e7b76091463f23ecc/autocomplete.go#L12

@filimonov Compared. Added missing query keywords

filimonov · 2018-06-15T07:32:09Z

dbms/src/Client/QueryParts.h

+        {(const char *)"Tuple"},
+        {(const char *)"Nested"},
+        {(const char *)"Expression"},
+        {(const char *)"Set"},


@filimonov added Nullable as well)

alexey-milovidov · 2018-06-16T22:43:01Z

TODO: It shouldn't work when you paste a text with tabs in terminal.

…, fixes and cleanup

Tezd · 2018-06-19T16:10:34Z

@alexey-milovidov it doesn't trigger completion on paste, but will clear tabs from pasted text (at least this is behavior in my env) If it shows different behavior I will investigate it. Also not sure why builds are failing from time to time.

Tezd · 2018-07-10T15:30:40Z

@alexey-milovidov Are there other changes that are needed ?

alexey-milovidov · 2018-07-10T16:12:09Z

Yes, it requires multiple changes. I will address them later.

alexey-milovidov · 2018-07-28T22:57:02Z

Now we have system tables for all required entities except language keywords.

Tezd · 2018-08-02T15:19:50Z

@alexey-milovidov updated PR according to information from new system tables.

Tezd · 2018-08-15T09:06:20Z

@alexey-milovidov sorry to bother, but just want to know if there are any more changes that are needed?

alexey-milovidov · 2018-08-21T02:42:32Z

It's in my queue.

We have to rewrite code to modern style
(remove malloc, free, new, delete, sprintf, strdup, strlen, std::thread::detach)
Fix memory leaks. Fix thread safety errors.
Remove ternary search tree as it is not needed.
Extract completion for keywords from the parser (similar to "expected one of ... " report)
instead of explicit listing.
Allow to completely disable completion just in case.

Implement max_block_size for system.tables and system.columns table because we have servers with about one million of StripeLog tables with about 500 columns and when you do SELECT FROM system.columns even with limit, it will execute slowly and eat excessive amount of memory.

Check what happens when you copy-paste a query with tab characters.

Check and ensure compatibility of new client with old servers (completion may not work but client should work as before).

It is possible to implement in about three days.

…2447

…ou type SELECT DIS<TAB> #2447

filimonov · 2018-08-26T09:44:46Z

dbms/programs/client/Client.cpp

+        sendQuery(
+                "SELECT name FROM system.functions"
+                " UNION ALL "
+                "SELECT name FROM system.table_engines"


Will client be able to work with older versions of server, which doesn't have those tables?

alexey-milovidov · 2018-08-26T11:04:51Z

The diff in this pull request is outdated. See master branch.

morty added 3 commits May 25, 2018 19:52

First implementation

dcff1bb

Added all clickhouse query keywords

12a68a9

Free hash after usage and dont use it at all if no readline present

be0f0ec

alexey-milovidov reviewed May 31, 2018

View reviewed changes

morty added 9 commits June 4, 2018 17:50

Removed hashtable use ternary search tree

3ad80a6

Removed unneeded inclusion of source and header

9b32d1e

Free memory before exit

2479453

Added autoload and limit

fba5fda

Resolved conflicts

823068f

Load suggestions in different thread

3e14bdd

Merge branch 'master' into feature/suggestions

f99433e

Turn off autocompletion if no readline library

ec10e58

Merge branch 'master' into feature/suggestions

a1fa590

alexey-milovidov added a commit that referenced this pull request Jun 9, 2018

Formats: better modularity (development) #2447

20949b4

alexey-milovidov added a commit that referenced this pull request Jun 9, 2018

Formats: better modularity (development) #2447

ba28c75

alexey-milovidov added a commit that referenced this pull request Jun 9, 2018

Formats: better modularity (development) #2447

86769a2

alexey-milovidov added a commit that referenced this pull request Jun 9, 2018

Formats: better modularity (development) #2447

ebca3aa

alexey-milovidov added a commit that referenced this pull request Jun 9, 2018

Formats: better modularity (development) #2447

9ebf4f3

alexey-milovidov added a commit that referenced this pull request Jun 9, 2018

Formats: better modularity (development): removed very old tests #2447

c4e6434

alexey-milovidov added a commit that referenced this pull request Jun 9, 2018

Formats: better modularity (development) #2447

e3a445a

alexey-milovidov added a commit that referenced this pull request Jun 9, 2018

Formats: better modularity (development) #2447

252c09a

alexey-milovidov added a commit that referenced this pull request Jun 9, 2018

Formats: better modularity (development) #2447

21bfed4

alexey-milovidov added a commit that referenced this pull request Jun 9, 2018

Formats: better modularity (development) #2447

bf1788b

alexey-milovidov added a commit that referenced this pull request Jun 9, 2018

Formats: better modularity (development) #2447

1f22875

alexey-milovidov added a commit that referenced this pull request Jun 9, 2018

Formats: better modularity (development) #2447

85f3341

alexey-milovidov added a commit that referenced this pull request Jun 9, 2018

Formats: better modularity (development) #2447

fe21e0d

filimonov reviewed Jun 15, 2018

View reviewed changes

morty added 2 commits June 19, 2018 17:07

Added missing query keywords, preload dictionary names and attributes…

47c6dc1

…, fixes and cleanup

Merge branch 'master' into feature/suggestions

c063d5c

morty added 2 commits June 25, 2018 11:22

Merge branch 'master' into feature/suggestions

492040b

Merge branch 'master' into feature/suggestions

35c0ab9

morty added 2 commits August 2, 2018 14:01

Resolved conflicts

fac492a

Use system tables for formats, engines, functions, collations, types

69f7948

Update Client.cpp

11a7825

alexey-milovidov added a commit that referenced this pull request Aug 25, 2018

Allow to disable suggestion; check server revision #2447

9293882

alexey-milovidov added a commit that referenced this pull request Aug 25, 2018

Separate files #2447

24083e3

alexey-milovidov added a commit that referenced this pull request Aug 25, 2018

Additions #2447

417cc1a

alexey-milovidov added a commit that referenced this pull request Aug 25, 2018

Support for max_block_size in system.tables and system.columns tables #…

068e503

…2447

alexey-milovidov mentioned this pull request Aug 25, 2018

Support for max_block_size in system.tables and system.columns tables #2953

Merged

alexey-milovidov added a commit that referenced this pull request Aug 25, 2018

Made suggest case sensitive again. Otherwise there is a glitch when y…

93b44fb

…ou type SELECT DIS<TAB> #2447

alexey-milovidov added a commit that referenced this pull request Aug 25, 2018

Added test #2447

9c79c5f

alexey-milovidov added a commit that referenced this pull request Aug 25, 2018

Added check for max_block_size = 0 #2447

aa936b0

alexey-milovidov merged commit 11a7825 into ClickHouse:master Aug 25, 2018

filimonov reviewed Aug 26, 2018

View reviewed changes

alexey-milovidov added a commit that referenced this pull request Aug 29, 2018

Removed useless field #2447

87dfd8e

alexey-milovidov added a commit that referenced this pull request Sep 3, 2018

Removed wrong code from StorageSystemColumns #2447

8fae37f

proller pushed a commit to proller/ClickHouse that referenced this pull request Sep 3, 2018

Removed wrong code from StorageSystemColumns ClickHouse#2447

e170de5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/suggestions #2447

Feature/suggestions #2447

Tezd commented May 30, 2018 •

edited by alexey-milovidov

Loading

alexey-milovidov May 31, 2018

Tezd May 31, 2018

alexey-milovidov May 31, 2018

alexey-milovidov commented May 31, 2018

alexey-milovidov May 31, 2018

Tezd May 31, 2018

alexey-milovidov May 31, 2018

filimonov Jun 15, 2018

Tezd Jun 15, 2018

filimonov Jun 15, 2018

filimonov Jun 15, 2018 •

edited

Loading

Tezd Jun 15, 2018

filimonov Jun 15, 2018

Tezd Jun 19, 2018

filimonov Jun 15, 2018

Tezd Jun 19, 2018

filimonov Jun 15, 2018

Tezd Jun 19, 2018

filimonov Jun 15, 2018

Tezd Jun 19, 2018

alexey-milovidov commented Jun 16, 2018

Tezd commented Jun 19, 2018

Tezd commented Jul 10, 2018

alexey-milovidov commented Jul 10, 2018

alexey-milovidov commented Jul 28, 2018

Tezd commented Aug 2, 2018

Tezd commented Aug 15, 2018

alexey-milovidov commented Aug 21, 2018 •

edited

Loading

filimonov Aug 26, 2018

alexey-milovidov commented Aug 26, 2018


		#include <sys/types.h>

		//All of functionality for hash was taken from mysql-server project from completion_hash.cpp file

Feature/suggestions #2447

Feature/suggestions #2447

Conversation

Tezd commented May 30, 2018 • edited by alexey-milovidov Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexey-milovidov commented May 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

filimonov Jun 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexey-milovidov commented Jun 16, 2018

Tezd commented Jun 19, 2018

Tezd commented Jul 10, 2018

alexey-milovidov commented Jul 10, 2018

alexey-milovidov commented Jul 28, 2018

Tezd commented Aug 2, 2018

Tezd commented Aug 15, 2018

alexey-milovidov commented Aug 21, 2018 • edited Loading

Choose a reason for hiding this comment

alexey-milovidov commented Aug 26, 2018

Tezd commented May 30, 2018 •

edited by alexey-milovidov

Loading

filimonov Jun 15, 2018 •

edited

Loading

alexey-milovidov commented Aug 21, 2018 •

edited

Loading