Skip to content

Commit

Permalink
Merge remote-tracking branch 'wa2/main'
Browse files Browse the repository at this point in the history
  • Loading branch information
roberthovsepyan committed Feb 13, 2024
2 parents 30f6376 + 6fe9eae commit 7517481
Showing 265 changed files with 283,231 additions and 97,896 deletions.
23 changes: 5 additions & 18 deletions .eslintrc
Original file line number Diff line number Diff line change
@@ -5,28 +5,15 @@
"@gravity-ui/eslint-config/prettier"
],
// These files are generated, so they don't have to be linted
"ignorePatterns": ["**/*AutocompleteParser.js", "src/generator/dist/**/*"],
"ignorePatterns": ["src/autocomplete/*/generated/**/*", "src/autocomplete/*/grammar/**/*"],
"rules": {
"array-callback-return": "off",
"consistent-return": "off",
"no-negated-condition": "off",
"@typescript-eslint/explicit-function-return-type": "error",
"object-shorthand": "error",
"no-implicit-globals": "off"
},
"overrides": [
{
"files": "*.js",
"rules": {
"@typescript-eslint/explicit-function-return-type": "off"
}
},
{
"files": ["**/parser-extension.js", "src/autocomplete/lib/parsing.js"],
"rules": {
// Parser is extended via reassignment, so this rule doesn't make sense
"no-param-reassign": "off"
}
}
]
"no-implicit-globals": "off",
"camelcase": "off",
"complexity": "off"
}
}
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -8,3 +8,4 @@ node_modules
# Artifacts
dist

.ds_store
File renamed without changes.
13 changes: 3 additions & 10 deletions CODE_CONVENTIONS.md
Original file line number Diff line number Diff line change
@@ -5,15 +5,8 @@ You can always link those rules in your PR without hesitation if you see that th

## Testing

- Each test should only test a granular added piece of logic, and shouldn't test functionality of other's. E.g. if you are using OptionalIfNotExists, you shouldn't test if you get suggestions when writing '', 'IF ', 'IF NOT', just test if `IF NOT EXISTS` is suggesting, and it's enough.
- Each test file should contain at least:
- A test that checks a complete statement for errors
- A test that checks a complete statement locations (can be merged with the one above)
- Tests should be put in `src/tests/{dialect}` folders
- All tests with cursor (expect when testing with newlines) should be written using `parse{dialect}QueryWithCursor` functions. This will help us avoid passing cursor position to functions and make tests more readable
- Each test should only test a granular added piece of logic, and shouldn't test functionality of other's.
- Don't use `foo` or `bar` custom names, always use `test_{object}`, e.g. `SELECT * FROM test_table`, not `SELECT * FROM hehe_haha`. If you need multiple names, use `_{number}` suffix, e.g. `SELECT test_field, test_field_2 FROM test_table;`
- Write all the static tokens in UPPER_CASE, and all the custom variables in lower_case, e.g. `SELECT test_field`
- Always test your statements on errors, and if there's an unexpected error, just add `TODO: fix unhandled error` error

What we'll do later:

- Add required in-between tests, e.g. `parseGeneralSql('DROP DATABASE', ' test_database')`. Currently most of the tests are only checking if query is good when the cursor is at the end.
- Add more shared grouping tests.
22,714 changes: 0 additions & 22,714 deletions NOTICE.txt

This file was deleted.

49 changes: 8 additions & 41 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,57 +1,24 @@
# WebSQL autocomplete

![ci_badge](https://img.shields.io/github/actions/workflow/status/gravity-ui/websql-autocomplete/ci.yml)
![npm_package_version_badge](https://img.shields.io/npm/v/websql-autocomplete)
![last_commit_badge](https://img.shields.io/github/last-commit/gravity-ui/websql-autocomplete)
![license_badge](https://img.shields.io/github/license/gravity-ui/websql-autocomplete)

A tool that provides autocompletion for various sql dialects.

# Parser theory
# Autocomplete theory

In order to parse any language, you need a [lexer](https://en.wikipedia.org/wiki/Lexical_analysis) (tokenizer) and a [parser](https://en.wikipedia.org/wiki/Parsing#Parser) (ast builder)

Resources to research:

- Bison grammar specification: https://www.gnu.org/software/bison/manual/html_node/Rules-Syntax.html
- Playlist with the explanation on how parser and lexer work together: https://www.youtube.com/playlist?list=PLIrl0f9NJZy4oOOAVPU6MyRdFjJFGtceu
- The ANTLR Mega Tutorial: https://tomassetti.me/antlr-mega-tutorial/
- Code Completion with ANTLR4-c3: https://tomassetti.me/code-completion-with-antlr4-c3/

# How everything works

1. Autocomplete parser source code for different `{dialect}` are defined in `src/autocomplete/parsers/{dialect}` directories.
2. Lexer configuration is specified in `src/autocomplete/parsers/{dialect}/grammar/sql.jisonlex` file.
3. SQL grammar is specified in `src/autocomplete/parsers/{dialect}/grammar/**/*.jison`. It is placed in multiple files for easier understanding. Test files in the format `*.test.ts` are placed nearby. Jison is basically bison, but for javascript.
4. `src/autocomplete/parsers/{dialect}/grammar/structure.json` specifies paths to a lexer, and to all the grammar files.
5. Parser extension (basically the autocomplete features) is specified in `src/autocomplete/parsers/{dialect}/parser-extension.ts`.
6. `src/generator/main.ts` concatenates all the jison files into a single big jison file, and runs the jison tool with the specified lexer, then wires everything up with the `parser-extension.ts`, generating `src/autocomplete/parsers/{dialect}/{dialect}AutocompleteParser.js`.
7. The generated file is in plain javascript, so we create a convenient typescript wrapper in `src/autocomplete/index.ts` with all the types and functions. Our users should include this file in their own projects.
1. ANTLR4 grammar and lexer for different `{dialect}` are defined in `src/autocomplete/{dialect}/grammar` directories.
2. Actual parsers and lexers for different `{dialect}` are generated into `src/autocomplete/{dialect}/generated` directories.
3. Dialect specific code, which is required for the core logic to work, is put in `src/autocomplete/{dialect}/{dialect}Autocomplete.ts` files.
4. The core logic is inside `src/index.ts` file. It uses dialect specific helpers to generate autocomplete suggestions based on current cursor position.

# Main scripts

- `npm run generate` - Generate parsers
- `npm run generate` - Generate parsers and lexers
- `npm run test` - Run tests

# Contributing

- When adding a new file, just add Yandex copyright header
- When editing a file that has not Yandex copyright, add Yandex copyright header underneath the existent copyright from Cloudera, and update changed files list in [NOTICE.txt](NOTICE.txt) at the bottom
- Delete and move files freely
- You should name your pull requests and commits in [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) naming. We update release version based on the commits

Yandex copyright:

```text
// Please note that the code below is the modified code distributed on the terms, mentioned below.
// The copyright for the changes belongs to YANDEX LLC.
//
// Copyright 2023 YANDEX LLC
//
// Licensed under the Apache License, Version 2.0 (the "License")
// You may not use this file except in compliance with the License.
// You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software distributed under
// the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
// either express or implied. See the License for the specific language governing permissions
// and limitations under the License.
```
31 changes: 31 additions & 0 deletions antlr-formatter.config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"main": {
"columnLimit": 500,
"minEmptyLines": 1,
"maxEmptyLinesToKeep": 1,
"useTab": false,
"reflowComments": false,
"breakBeforeBraces": false,
"keepEmptyLinesAtTheStartOfBlocks": false,
"allowShortRulesOnASingleLine": false,
"alignSemicolons": "hanging",
"alignColons": "hanging",
"alignTrailingComments": true
},
"lexer": {
"alignTrailingComments": true,
"columnLimit": 500,
"maxEmptyLinesToKeep": 1,
"reflowComments": false,
"useTab": false,
"allowShortRulesOnASingleLine": true,
"allowShortBlocksOnASingleLine": true,
"minEmptyLines": 0,
"alignSemicolons": "ownLine",
"alignColons": "trailing",
"singleLineOverrulesHangingColon": true,
"alignLexerCommands": true,
"alignLabels": true,
"alignTrailers": true
}
}
11 changes: 11 additions & 0 deletions build.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import {build} from 'esbuild';

build({
entryPoints: ['src/index.ts'],
external: ['antlr4ng', 'antlr4-c3'],
bundle: true,
minify: true,
format: 'esm',
outfile: 'dist/index.js',
tsconfig: './tsconfig.build.json',
});
1 change: 1 addition & 0 deletions commitlint.config.js → commitlint.config.cjs
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
/* eslint-env node */
module.exports = {extends: ['@commitlint/config-conventional']};
15 changes: 15 additions & 0 deletions jest.config.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
/* eslint-env node */
module.exports = {
extensionsToTreatAsEsm: ['.ts'],
moduleNameMapper: {
'^(\\.{1,2}/.*)\\.js$': '$1',
},
transform: {
'^.+\\.ts$': [
'ts-jest',
{
useESM: true,
},
],
},
};
29 changes: 0 additions & 29 deletions jest.config.js

This file was deleted.

Loading

0 comments on commit 7517481

Please sign in to comment.