-
Notifications
You must be signed in to change notification settings - Fork 291
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Smart Apply: Use system prompt to encourage code blocks being created #5290
Conversation
@abeatrix Seems just as reliable without the weird occasional behaviour in chat... Running evals here sourcegraph/cody-leaderboard#16 |
agent/src/index.test.ts
Outdated
test('Test animal implementation name property', () => { | ||
const dog = new Dog() | ||
expect(dog.name).toEqual('Dog') | ||
// src/animal.test.ts |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is added as a comment instead of markdown format we provided 👀
(It wasn't doing it before the prompt change though so not a regression but an edge case?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
im seeing a minor regression where it's including the file pointers in the response.
cc @jtibshirani on the response change
lib/shared/src/chat/preamble.ts
Outdated
* produce code blocks that we can associate with existing file paths. | ||
* We want to read these file paths to support applying code directly to files from chat. | ||
*/ | ||
const CHAT_PREAMBLE = psDedent` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
const CHAT_PREAMBLE = psDedent` | |
const SMART_APPLY_PREAMBLE = ps`If your answer contains fenced code blocks in Markdown, include the full file path in the code block tag using this structure: \`\`\`$LANGUAGE:$FILEPATH\n\`\`\`.` | |
const CHAT_PREAMBLE = DEFAULT_PREAMBLE.join(SMART_APPLY_PREAMBLE) |
I think we don't need to say that this is an additional rules in system prompt. We can probably update it to something like this? ( Haven't tested it though!)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Happy to try anything, will run an eval on this change and we can see!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@abeatrix Applied this change and made a couple of more tweaks, looks pretty good!
Eval is here: sourcegraph/cody-leaderboard#16
My comments here are still relevant: sourcegraph/cody-leaderboard#16 (comment)
I think we can go ahead and merge this, we can always tweak later but let's get a patch out today as we may be both OOO next week
Co-authored-by: Beatrix <[email protected]>
Co-authored-by: Beatrix <[email protected]>
|
Have a failing test because the demo instance is replying with
It's likely because the demo config has changed since that test was last updated. Using |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great! I haven't encountered any prompt leaks anymore, and I don't see any noticeable regression from the test results, so I think this would be a good change to patch the regression we are seeing. Thanks for fixing this!
…#5290) ## Description This PR improves the smart apply prompt tweak so that: - We use the system prompt when available - We use the same desired Markdown syntax when _we_ include codebase context, which helps steer the LLM onto the right path. I have ran an eval on this here: sourcegraph/cody-leaderboard#16 ## Test plan - [x] Tested creating code blocks on all major models <!-- Required. See https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles. --> --------- Co-authored-by: Beatrix <[email protected]>
Description
This PR improves the smart apply prompt tweak so that:
I have ran an eval on this here: sourcegraph/cody-leaderboard#16
Quite happy with the results so far
Test plan