Add auto completions #79

waydegg · 2023-08-26T23:26:37Z

Summary

This PR adds completions for syntax (classes, variables, functions, etc.) and file names for the prompt session.

Completions kept in sync as files are added/updated/deleted (they are "refreshed" every 5 seconds atm). Whenever a user starts a prompt with "/" we return command completions only.

Since we need to keep track of what files are in the context for keeping completions up-to-date, I've added a CodeFileIndex class that currently just holds a set of file names added into the context. CodeFileIndex is shared between CodeFileManager and UserInputManager. This is done so that when CodeFileIndex modifies the filesystem, `UserInputManager can see these modifications and make updates to completions.

To-Do

Update test cases for CodeFileManager
Add test cases for MentatCompleter

PCSwingle · 2023-08-27T00:08:33Z

Just fyi, in my commands PR #59 I change how we use prompt toolkit a lot; you might want to work off of that branch until it gets merged.

waydegg · 2023-08-29T23:28:59Z

@biobootloader would love your thoughts are on CodeFileIndex and if you can think of a more elegant way to share file paths w/ UserInputManager (i.e. less changes required for CodeFileManager)

biobootloader · 2023-08-30T00:01:14Z

mentat/mentat_prompt_session.py

+# 4. Decide on what words to auto complete (right now it's everything)
+#   a. Use ctags to add types (e.g. function, class) to completions
+# 5. Use ctags if available, fallback to pygments lexer if not
+# 6. Consider fuzzy matching completions


I love fuzzy matching in general. I guess the downside could be too many options popping up for everything you type?

Rift has an @ syntax for referencing files, maybe something like that could be good

You can limit the number of completions shown. Prompt Toolkit has a built-in fuzzy matcher that's pretty fast too.

Yea I like the @ directive for Rift and Cursor. Happy to make that character show file completions specifically.

biobootloader · 2023-08-30T00:21:35Z

@biobootloader would love your thoughts are on CodeFileIndex and if you can think of a more elegant way to share file paths w/ UserInputManager (i.e. less changes required for CodeFileManager)

I think it is going to require changes to CodeFileManager. I guess the question is how do we divide responsibilities best between:

CodeFileManager
CodeMap
CodeFileIndex
MentatCompleter

Both CodeFileManager and MentatCompleter need to know which files / partial files are in the context, and this will grow more complicated as we add features to Mentat that automatically add partial files to context as the conversation continues. So your approach is to make CodeFileIndex to do that shared work.

Maybe a better name for CodeFileIndex could something like ContextManager? Maybe it should also own get_code_message() (currently in CodeFileManager), and CodeFileManager should only be responsible for reading and writing to files?

Another question is what we should auto-complete - just stuff in context or also stuff that's not in context, like from the CodeMap? I think including stuff that's outside context will be important because we'll want user to be able add files/classes/functions to context easily, within a conversation. But maybe that'd be too many completions being suggested for big codebases? The completion engine should probably know which completions are coming from within or outside context, so it could prioritize things in context.

I'm not really sure about any of this, but those are some initial thoughts!

biobootloader · 2023-08-30T00:23:35Z

I don't think this branch has the partial files changes from #69, sorry that that complicates things!

waydegg · 2023-08-30T00:58:07Z

Maybe a better name for CodeFileIndex could something like ContextManager? Maybe it should also own get_code_message() (currently in CodeFileManager), and CodeFileManager should only be responsible for reading and writing to files?

Yup that sounds good. Make CodeFileManager just handle CRUD actions for files, and ContextManager is where references to files and partial files are stored.

waydegg · 2023-08-30T01:02:12Z

Another question is what we should auto-complete - just stuff in context or also stuff that's not in context, like from the CodeMap?

Right now it's just stuff inside the context, however I don't think it would hurt to add completions for files/syntax inside git repos? I think we could enable/disable this depending on if the code-map is enabled/disabled.

waydegg · 2023-08-30T01:04:57Z

I don't think this branch has the partial files changes from #69, sorry that that complicates things!

Good callout! Will fetch from main and update this

waydegg · 2023-08-30T07:13:03Z

pyproject.toml

@@ -4,3 +4,4 @@ known_first_party = "mentat"

 [tool.ruff]
 line-length = 120
+ignore = ["E731"]


Added this rule to ignore lambda assignment (https://beta.ruff.rs/docs/rules/lambda-assignment/) as we have these everywhere and imo they're ok as long as you don't have a ton of logic in them.

waydegg · 2023-08-30T19:30:28Z

In an effort to keep the number of changes down in this PR, I think we should add fuzzy matching in a separate feature branch. Same goes for the @ directive since it's basically another completer we're adding (e.g. another feature).

waydegg · 2023-08-30T19:38:39Z

I don't think it's worth the effort to generate completions from ctags. You're only able to get top-level syntax words (can't get anything inside classes/functions), so the pygments lexer does a pretty good job at this as is.

I also decided to not include completions from all files in the git repo (if there is one), only files explicitly added in the code context, since there's too much irrelevant information being suggested. This is the how VSCode does their compeltions anyways which I think is a good choice.

granawkins · 2023-08-31T09:08:28Z

I read through this and tested it out, and it looks good to me. One comment: eventually it'd be nice if this works with different interfaces. Like if I could call refresh_completions from the VSCode extension somehow with its current user input. This can def be done later, just wanted to throw it out there.

biobootloader · 2023-08-31T18:36:45Z

I read through this and tested it out, and it looks good to me. One comment: eventually it'd be nice if this works with different interfaces. Like if I could call refresh_completions from the VSCode extension somehow with its current user input. This can def be done later, just wanted to throw it out there.

Agreed, "core" Mentat would provide lists of available completions to all interfaces

biobootloader · 2023-08-31T18:39:38Z

In an effort to keep the number of changes down in this PR, I think we should add fuzzy matching in a separate feature branch. Same goes for the @ directive since it's basically another completer we're adding (e.g. another feature).

Agreed, let's make followup issues for these. The @ approach also relates to #17

waydegg · 2023-08-31T19:06:34Z

One comment: eventually it'd be nice if this works with different interfaces

Totally. I'd like to have the completions using pygls to pull from language servers (assuming we can get it working outside of VSCode). Would make the auto-completions tool-agnostic and higher quality.

biobootloader · 2023-08-31T23:24:43Z

not a big deal because it's rare, but both kinds of completitons can be suggested at once (the history one take priority if you hit tab):

I don't think it's worth trying to fix - not sure how often most people would use history completions anyway

waydegg · 2023-09-01T00:19:26Z

both kinds of completitons can be suggested at once

Do you have any preference on how we should handle this? It's pretty common for me to see both completion sources, so I think it's worth looking into.

I'm thinking we could copy how ipython does it and have arrow keys finish/cycle command history completions, and have tab/shift+tab or ctrl+n/ctrl+p finish/cycle auto-completions.

biobootloader · 2023-09-01T00:46:06Z

mentat/code_file_manager.py

-        if self.files:
+        if self.code_context.files:
            cprint("Files included in context:", "green")
        else:
            cprint("No files included in context.\n", "red")
            cprint("Git project: ", "green", end="")
-        cprint(self.git_root.name, "blue")
+        cprint(self.config.git_root.name, "blue")
        _print_path_tree(
-            _build_path_tree(self.files, self.git_root),
-            get_paths_with_git_diffs(self.git_root),
-            self.git_root,
+            _build_path_tree(self.code_context.files.values(), self.config.git_root),
+            get_paths_with_git_diffs(self.config.git_root),
+            self.config.git_root,
        )
        print()


I think displaying the project tree / include files should be done by CodeContext. And instead of being in the init should be in something like CodeContext.display_context_tree(), called in app.py after initializing the CodeContext.

biobootloader · 2023-09-01T22:57:12Z

I'm thinking we could copy how ipython does it and have arrow keys finish/cycle command history completions, and have tab/shift+tab or ctrl+n/ctrl+p finish/cycle auto-completions.

That sounds great!

biobootloader

This is awesome!

add completions from pymgents lexer

ef7c577

waydegg mentioned this pull request Aug 27, 2023

Add autocomplete for file, class, and function names #78

Closed

waydegg added 5 commits August 27, 2023 00:13

add file path completions

df747e6

removed imports

e987be0

merge main

6625279

combine command completer with syntax completer

268b454

add CodeFileIndex, auto refresh completions

f9a104f

biobootloader reviewed Aug 30, 2023

View reviewed changes

waydegg added 2 commits August 29, 2023 23:54

merge main, rename CodeIndex to CodeContext, fix tests

798b7c1

fix ruff errors

ecbc8ef

waydegg commented Aug 30, 2023

View reviewed changes

remove comments

c88cfcf

fix merge conflicts

ba68eb7

waydegg marked this pull request as ready for review August 30, 2023 19:54

fix black lint errors

105c040

waydegg requested a review from biobootloader August 31, 2023 03:49

fix edge case

8441c2e

biobootloader reviewed Sep 1, 2023

View reviewed changes

waydegg added 3 commits September 1, 2023 18:57

move print project tree to code context

7fd479d

change history complete key to be right arrow key

ccb42cd

fix ruff errors

f193865

waydegg requested a review from biobootloader September 2, 2023 02:52

display context tree in CodeContext instead of CFM

ed1f6d2

biobootloader approved these changes Sep 2, 2023

View reviewed changes

biobootloader merged commit e99ed70 into AbanteAI:main Sep 5, 2023
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add auto completions #79

Add auto completions #79

waydegg commented Aug 26, 2023 •

edited

Loading

PCSwingle commented Aug 27, 2023

waydegg commented Aug 29, 2023

biobootloader Aug 30, 2023

waydegg Aug 30, 2023

biobootloader commented Aug 30, 2023

biobootloader commented Aug 30, 2023

waydegg commented Aug 30, 2023

waydegg commented Aug 30, 2023

waydegg commented Aug 30, 2023

waydegg Aug 30, 2023

waydegg commented Aug 30, 2023

waydegg commented Aug 30, 2023

granawkins commented Aug 31, 2023

biobootloader commented Aug 31, 2023

biobootloader commented Aug 31, 2023

waydegg commented Aug 31, 2023

biobootloader commented Aug 31, 2023

waydegg commented Sep 1, 2023

biobootloader Sep 1, 2023

biobootloader commented Sep 1, 2023

biobootloader left a comment

Add auto completions #79

Add auto completions #79

Conversation

waydegg commented Aug 26, 2023 • edited Loading

Summary

To-Do

PCSwingle commented Aug 27, 2023

waydegg commented Aug 29, 2023

biobootloader Aug 30, 2023

Choose a reason for hiding this comment

waydegg Aug 30, 2023

Choose a reason for hiding this comment

biobootloader commented Aug 30, 2023

biobootloader commented Aug 30, 2023

waydegg commented Aug 30, 2023

waydegg commented Aug 30, 2023

waydegg commented Aug 30, 2023

waydegg Aug 30, 2023

Choose a reason for hiding this comment

waydegg commented Aug 30, 2023

waydegg commented Aug 30, 2023

granawkins commented Aug 31, 2023

biobootloader commented Aug 31, 2023

biobootloader commented Aug 31, 2023

waydegg commented Aug 31, 2023

biobootloader commented Aug 31, 2023

waydegg commented Sep 1, 2023

biobootloader Sep 1, 2023

Choose a reason for hiding this comment

biobootloader commented Sep 1, 2023

biobootloader left a comment

Choose a reason for hiding this comment

waydegg commented Aug 26, 2023 •

edited

Loading