feat: request contextualisation - core functionality #65

jcierocki · 2024-06-26T08:28:38Z

Closes #54

…turn List[MethodParamWithTyping] required to construct ExposedFunction dataclass

github-actions · 2024-06-26T08:29:59Z

Code Coverage Summary

Filename                                                  Stmts    Miss  Cover    Missing
------------------------------------------------------  -------  ------  -------  ---------------------------------------------------------------------
dbally/_main.py                                              13       1  92.31%   10
dbally/_types.py                                              8       1  87.50%   24
dbally/context.py                                             5       0  100.00%
dbally/exceptions.py                                          1       0  100.00%
dbally/assistants/base.py                                    24       0  100.00%
dbally/assistants/openai.py                                  59       2  96.61%   59-76
dbally/audit/event_tracker.py                                36       8  77.78%   38-40, 53, 64, 74, 91, 97
dbally/audit/events.py                                       34       0  100.00%
dbally/audit/spans.py                                         7       0  100.00%
dbally/audit/event_handlers/base.py                          15       0  100.00%
dbally/audit/event_handlers/buffer_event_handler.py           8       0  100.00%
dbally/audit/event_handlers/cli_event_handler.py             56      35  37.50%   11-13, 47-55, 65-66, 79-98, 120-127, 138-145
dbally/audit/event_handlers/langsmith_event_handler.py       29      25  13.79%   6-106
dbally/audit/event_handlers/otel_event_handler.py            74      27  63.51%   19, 123, 126, 138-139, 159-179, 191-199, 209-219, 222-223
dbally/collection/collection.py                             127       3  97.64%   137, 154, 329
dbally/collection/exceptions.py                              13       0  100.00%
dbally/collection/results.py                                 14       0  100.00%
dbally/embeddings/base.py                                     5       0  100.00%
dbally/embeddings/exceptions.py                              15       6  60.00%   10-11, 20, 29-30, 39
dbally/embeddings/litellm.py                                 28      12  57.14%   7-8, 44, 68-84
dbally/gradio/interface.py                                  104     104  0.00%    1-362
dbally/iql/_exceptions.py                                    56       0  100.00%
dbally/iql/_processor.py                                    100       3  97.00%   79, 184, 190
dbally/iql/_query.py                                         26       2  92.31%   11-12
dbally/iql/_type_validators.py                               48       2  95.83%   40, 44
dbally/iql/syntax.py                                         36       9  75.00%   6-9, 27, 36, 60, 63-66
dbally/iql_generator/iql_generator.py                        65       3  95.38%   246-248
dbally/iql_generator/prompt.py                               35       2  94.29%   43, 75
dbally/llms/base.py                                          28       1  96.43%   34
dbally/llms/litellm.py                                       23      10  56.52%   7-8, 47-53, 60, 77
dbally/llms/local.py                                         18      18  0.00%    1-60
dbally/llms/clients/base.py                                  23       2  91.30%   46-47
dbally/llms/clients/exceptions.py                            15       6  60.00%   10-11, 20, 29-30, 39
dbally/llms/clients/litellm.py                               44      21  52.27%   8-9, 65-71, 97-120
dbally/llms/clients/local.py                                 33      33  0.00%    1-95
dbally/nl_responder/nl_responder.py                          24       4  83.33%   74-85
dbally/nl_responder/prompts.py                               19       4  78.95%   64-67
dbally/prompt/elements.py                                    25       1  96.00%   29
dbally/prompt/template.py                                    65       8  87.69%   33, 41, 49, 96, 110, 127, 153, 205
dbally/similarity/chroma_store.py                            37       0  100.00%
dbally/similarity/elastic_vector_search.py                   19      16  15.79%   5-102
dbally/similarity/elasticsearch_store.py                     22      19  13.64%   5-107
dbally/similarity/faiss_store.py                             38      35  7.90%    5-103
dbally/similarity/fetcher.py                                  5       0  100.00%
dbally/similarity/index.py                                   30       2  93.33%   26, 70
dbally/similarity/sqlalchemy_base.py                         44      19  56.82%   35-37, 46, 68, 77, 86-89, 99-105, 123-126
dbally/similarity/store.py                                    7       0  100.00%
dbally/view_selection/base.py                                 7       0  100.00%
dbally/view_selection/llm_view_selector.py                   17       0  100.00%
dbally/view_selection/prompt.py                               9       0  100.00%
dbally/view_selection/random_view_selector.py                10      10  0.00%    1-36
dbally/views/base.py                                         18       1  94.44%   58
dbally/views/decorators.py                                   11       0  100.00%
dbally/views/exceptions.py                                    7       3  57.14%   20-22
dbally/views/exposed_functions.py                            42       3  92.86%   60-63
dbally/views/methods_base.py                                 35       2  94.29%   86, 94
dbally/views/pandas_base.py                                  52       1  98.08%   97
dbally/views/sqlalchemy_base.py                              40       7  82.50%   64, 79-82, 99-103
dbally/views/structured.py                                   52       1  98.08%   82
dbally/views/freeform/text2sql/config.py                     21       1  95.24%   47
dbally/views/freeform/text2sql/exceptions.py                  7       3  57.14%   12-14
dbally/views/freeform/text2sql/prompt.py                     11       0  100.00%
dbally/views/freeform/text2sql/view.py                       94      22  76.60%   50, 53, 56, 59, 62, 74, 154, 158-161, 164, 194, 209-210, 218, 231-236
dbally_cli/main.py                                            5       5  0.00%    1-13
dbally_cli/text2sql.py                                       94      94  0.00%    1-248
dbally_codegen/autodiscovery.py                             122      18  85.25%   54-57, 241-243, 264-278, 281-284, 353-354, 450-455
dbally_codegen/generator.py                                 175       7  96.00%   81, 91, 314, 342, 360, 374, 420
TOTAL                                                      2389     622  73.96%

Diff against main

Filename                                  Stmts    Miss  Cover
--------------------------------------  -------  ------  --------
dbally/context.py                            +5       0  +100.00%
dbally/collection/collection.py              +1       0  +0.02%
dbally/iql/_exceptions.py                    +7      -1  +2.04%
dbally/iql/_processor.py                     +6      -2  +2.32%
dbally/iql/_query.py                         +2      +1  -3.52%
dbally/iql/_type_validators.py              +10       0  +1.09%
dbally/iql_generator/iql_generator.py        -4      -2  +2.63%
dbally/iql_generator/prompt.py               +2       0  +0.35%
dbally/views/base.py                         +2       0  +0.69%
dbally/views/exposed_functions.py            +7       0  +1.43%
dbally/views/methods_base.py                 -6       0  -0.83%
dbally/views/structured.py                   +3       0  +0.12%
dbally/views/freeform/text2sql/view.py       +1       0  +0.26%
TOTAL                                       +36      -4  +0.56%

Results for commit: fab9d3f

Minimum allowed coverage is 60%

♻️ This comment has been updated with latest results

…bugfix; fixed linting issues

…aseCallerContext subclasses from the Union[] args; linting fix

…hed)

…ss + type hint improvements: fixes, new generics and aliases

…nt when it detectes the additional context is required

…jc/issue-54-request-context

…pylint; run pre-commit auto refactor

github-actions · 2024-07-04T11:23:08Z

Trivy scanning results.

…ity and properly handle types: Union[Type1, Type2, ...], __main__.SomeCustomClass

…xecutionResult]."context" -> "metadata"

…as other IQL errors, inherting from IQLError, thus enabled its handling by self-reflection mechanism

…ly in 'structured_view.ask()' and than stored in 'ExposedFunction' instances

…and putting filter args in quotation marks

…uest-context; type hint fixes, missing docstrings, additional refactor

Jakub Cierocki and others added 3 commits June 21, 2024 16:36

context logic subpackage; type-hint context extraction

21506e6

reworked type hint info extraction; extended functionality to also re…

a87e8e2

…turn List[MethodParamWithTyping] required to construct ExposedFunction dataclass

hidden args handling enabled

3ad4ecd

jcierocki added the feature New feature or request label Jun 26, 2024

jcierocki requested a review from mhordynski June 26, 2024 08:28

jcierocki self-assigned this Jun 26, 2024

jcierocki changed the title ~~Request contextualisation - core functionality~~ feat: request contextualisation - core functionality Jun 26, 2024

ds-jakub-cierocki added 15 commits June 28, 2024 13:16

improved type hints parsing and compatibility using package

b0cc0ae

dedicated exceptions for contex-related operations

4ff5f62

useful classmethods for context-related operations

c479c50

make whole context utils module protected; added IQL parsing helper; …

e3bb127

…bugfix; fixed linting issues

parsing type hints _extract_params_and_context() no longer excludes B…

de72c7c

…aseCallerContext subclasses from the Union[] args; linting fix

adjusted the existing code to be aware of contexts (promts yet untouc…

d3958c0

…hed)

adjusted _type_validators.validate_arg_type() to handle typing.Union[]

be338bf

context._utils._does_arg_allow_context() fix

78f1535

context record is now based on pydantic.BaseModel rather than datacla…

308e2e1

…ss + type hint improvements: fixes, new generics and aliases

type hint lifting

73741d9

IQL generating LLM prompt passes BaseCallerContext() as filter argume…

902f5ff

…nt when it detectes the additional context is required

comments cleanup

6309070

type hint fixes

d523bf7

Merge branch 'main' (which includes a large refactor by Michal) into …

efe212f

…jc/issue-54-request-context

post-merge fixes + minor refactor

9ba89e5

jcierocki force-pushed the jc/issue-54-request-context branch from 0ded684 to 9ba89e5 Compare July 3, 2024 12:58

added missing docstrings; fixed type hints; fixed issues detected by …

5fd802f

…pylint; run pre-commit auto refactor

jcierocki force-pushed the jc/issue-54-request-context branch from 7a49b4d to 3c87020 Compare July 4, 2024 18:31

ds-jakub-cierocki added 3 commits July 4, 2024 23:16

reworked parse_param_type() function to increase performance, general…

09bac55

…ity and properly handle types: Union[Type1, Type2, ...], __main__.SomeCustomClass

fix: removed duplicated line from the prompt template

d42a369

adjusted existing unit tests to work with new contextualization logic

c0b0522

jcierocki force-pushed the jc/issue-54-request-context branch 3 times, most recently from 7f8b5f4 to 5398636 Compare July 15, 2024 10:36

ds-jakub-cierocki added 3 commits July 15, 2024 17:58

docs + benchmarks adjusted to meet new naming [ExecutionResult, ViewE…

25fbfa6

…xecutionResult]."context" -> "metadata"

redesigned context-not-available error to follow the same principles …

a154577

…as other IQL errors, inherting from IQLError, thus enabled its handling by self-reflection mechanism

EXPERIMENTAL: reworked context injection such it is handled immediate…

623effd

…ly in 'structured_view.ask()' and than stored in 'ExposedFunction' instances

jcierocki force-pushed the jc/issue-54-request-context branch from 5398636 to 623effd Compare July 15, 2024 15:59

ds-jakub-cierocki added 7 commits July 19, 2024 16:13

additional unit tests for the new contextualization mechanism

afacf5b

context benchmark script and data

dd8b339

refactored main prompt (too long lines), missing end-of-line characters

6bb0816

better error handling

f388f92

context benchmark dataset fix

fbecc51

added polars-based accuracy summary to the benchmark

5d4ff64

adjusted prompt to reduce halucinations: nested filter/context calls …

e7e8826

…and putting filter args in quotation marks

jcierocki force-pushed the jc/issue-54-request-context branch from c5d7b65 to e7e8826 Compare July 23, 2024 08:54

merged main (inc. new benchmarks + large refactor) -> jc/issue-54-req…

f8bf64e

…uest-context; type hint fixes, missing docstrings, additional refactor

micpst assigned micpst and unassigned jcierocki Aug 27, 2024

micpst added 12 commits September 23, 2024 09:06

merge main

c1c871b

fix linters

8eefd9b

fix tests

c28091f

fix tests

69a8d58

fix tests

d6c8fc6

rm old benchmarks

d7026d4

some renames and stuff

e8271ac

fix benchmarks

bdcc7b3

merge main

71f53be

rm chroma file

c82e579

add contexts to benchmarks + fix types

f5a40cb

small refactor

fab9d3f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: request contextualisation - core functionality #65

feat: request contextualisation - core functionality #65

jcierocki commented Jun 26, 2024 •

edited

Loading

github-actions bot commented Jun 26, 2024 •

edited

Loading

github-actions bot commented Jul 4, 2024

feat: request contextualisation - core functionality #65

Are you sure you want to change the base?

feat: request contextualisation - core functionality #65

Conversation

jcierocki commented Jun 26, 2024 • edited Loading

github-actions bot commented Jun 26, 2024 • edited Loading

Code Coverage Summary

Diff against main

github-actions bot commented Jul 4, 2024

jcierocki commented Jun 26, 2024 •

edited

Loading

github-actions bot commented Jun 26, 2024 •

edited

Loading