Cleanup code context and code features

[PCSwingle]

TODO:

• Confirm that embeddings are still cached (I think they should be, but best to check)
• Figure out LLM filter user included files.

[PCSwingle]

The LLM filter no longer knows which files are user included; how should we give it this information? I think just giving it all of the file names would work well, but let me know your ideas.

[granawkins]

It's a great PR, look at those numbers! +507 −732
In hindsight I really overdid it with the caching 😅
Lots of good refactors. I left a few suggestions, take em or leave em.

[granawkins]

Suggested change

	if config.auto_context and prompt:
	if config.auto_context and prompt and auto_tokens > 0:

[PCSwingle]

In my persistent context PR I actually merge auto_context and auto_tokens into auto_context_tokens and I check if it's more than 0 there, so this change is pretty much already made!

[granawkins]

As I understand the system ultimately goes back to model.config. This will make it difficult to use with other models/scenarios, like what we did/plan-to-do with LLMFeatureFilter. Also, using max_tokens seems like a cleaner interface for the function? Just "give me up to N tokens of context".

[granawkins]

tbh our LLMFeatureFilter should have a test to catch that - making a note

[PCSwingle]

I changed it to prompt_tokens instead of max_tokens because we don't actually tell it to give us up to N tokens of context; we tell it to include all of the user's include files (since we no longer truncate it) and give us up to auto_tokens of automatic context (as long as it fits). The prompt tokens is just so we can check if there are enough tokens for user included files, and so we can only add as many auto context files as can fit in context. I think this makes a lot more sense to me (since we aren't truncating user included files anymore).

[PCSwingle]

What do you mean by difficult to use with different models/scenarios? I'm not sure I understand

[granawkins]

Suggested change

	if (
	section.start >= self.interval.start
	and section.start < self.interval.end
	):
	if self.interval.intersects(section):

DiffAnnotations inherit Interval so this works. Also, the earlier wouldn't catch an annotation spanning multiple intervals. Still not ideal, but will guarantee it's there at least.

[PCSwingle]

This isn't actually an intersection check; this is checking to ensure that the diff annotation starts within the interval (if it ends inside of the interval, we don't want to include it because a. it could be included in multiple intervals and b. the start of the diff annotation wouldn't make much sense because those line numbers aren't in this interval)

[biobootloader]

hmm, when I run with auto context on it now takes significantly longer to embed my query / get embeddings. It hangs for 5+ seconds before showing the loading bar for embeddings:

[PCSwingle]

hmm, when I run with auto context on it now takes significantly longer to embed my query / get embeddings. It hangs for 5+ seconds before showing the loading bar for embeddings:

I managed to track it down to us calling 'get_code_message' on every code feature in the project multiple times (which reads the entire file for every feature, meaning we're reading the same file multiple times whenever there are multiple features in a file) when we get embeddings; for now, I'll return the code feature caching on get_code_message, but later on I want to add caching where it should matter in code_file_manager to cache the individual files and only re-read the files if their last modified time has been updated as well as reduce the number of times we call get code message on each feature in get embeddings.
Edit: Actually, after re-adding code feature caching it didn't change anything; after looking into it more I realized I started calling git diff on every single file (not just the ones with git diffs) by accident! This also makes me think we might want to cache git diffs (and again only update them when the file is updated) since this is by far the most expensive operation we do.

[biobootloader]

Actually, after re-adding code feature caching it didn't change anything; after looking into it more I realized I started calling git diff on every single file (not just the ones with git diffs) by accident! This also makes me think we might want to cache git diffs (and again only update them when the file is updated) since this is by far the most expensive operation we do.

Nice work tracking it down! Much faster now. I guess this only matters if there are a lot of files with changes, but that should be super rare. I guess worst case someone is using Mentat to help them review a big PR that changes dozens of files?

Perhaps we could call git diff just once and get diff info for all changed files from that, instead of calling it on each file?