Smart Apply: Use system prompt to encourage code blocks being created #5290
Conversation
|
@abeatrix Seems just as reliable without the weird occasional behaviour in chat... Running evals here sourcegraph/cody-leaderboard#16 |
agent/src/index.test.ts
Outdated
| test('Test animal implementation name property', () => { | ||
| const dog = new Dog() | ||
| expect(dog.name).toEqual('Dog') | ||
| // src/animal.test.ts |
There was a problem hiding this comment.
This is added as a comment instead of markdown format we provided 👀
(It wasn't doing it before the prompt change though so not a regression but an edge case?)
There was a problem hiding this comment.
im seeing a minor regression where it's including the file pointers in the response.
cc @jtibshirani on the response change
lib/shared/src/chat/preamble.ts
Outdated
| * produce code blocks that we can associate with existing file paths. | ||
| * We want to read these file paths to support applying code directly to files from chat. | ||
| */ | ||
| const CHAT_PREAMBLE = psDedent` |
There was a problem hiding this comment.
| const CHAT_PREAMBLE = psDedent` | |
| const SMART_APPLY_PREAMBLE = ps`If your answer contains fenced code blocks in Markdown, include the full file path in the code block tag using this structure: \`\`\`$LANGUAGE:$FILEPATH\n\`\`\`.` | |
| const CHAT_PREAMBLE = DEFAULT_PREAMBLE.join(SMART_APPLY_PREAMBLE) |
I think we don't need to say that this is an additional rules in system prompt. We can probably update it to something like this? ( Haven't tested it though!)
There was a problem hiding this comment.
Happy to try anything, will run an eval on this change and we can see!
There was a problem hiding this comment.
@abeatrix Applied this change and made a couple of more tweaks, looks pretty good!
Eval is here: sourcegraph/cody-leaderboard#16
My comments here are still relevant: sourcegraph/cody-leaderboard#16 (comment)
I think we can go ahead and merge this, we can always tweak later but let's get a patch out today as we may be both OOO next week
Co-authored-by: Beatrix <[email protected]>
Co-authored-by: Beatrix <[email protected]>
|
|
|
Have a failing test because the demo instance is replying with It's likely because the demo config has changed since that test was last updated. Using |
…#5290) ## Description This PR improves the smart apply prompt tweak so that: - We use the system prompt when available - We use the same desired Markdown syntax when _we_ include codebase context, which helps steer the LLM onto the right path. I have ran an eval on this here: sourcegraph/cody-leaderboard#16 ## Test plan - [x] Tested creating code blocks on all major models <!-- Required. See https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles. --> --------- Co-authored-by: Beatrix <[email protected]>
Description
This PR improves the smart apply prompt tweak so that:
I have ran an eval on this here: sourcegraph/cody-leaderboard#16
Quite happy with the results so far
Test plan