Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance(ai-help): hash markdown to identify formatting updates #10643

Merged
merged 4 commits into from
Mar 29, 2024

Conversation

caugner
Copy link
Contributor

@caugner caugner commented Mar 4, 2024

Summary

Problem

The scripts/ai-help-macros.ts script updates the document content if the document hash changes, but this hash also changes when building the content from another yari branch (e.g. next), because the branch name is included in the source property:

image

Solution

Use the hash of the markdown content to determine whether the content has changed.

Note: We still use the text content to determine whether the embedding needs to be regenerated.


How did you test this change?

Ran the following query on the Supabase dev project to populate the new markdown_hash column:

update mdn_doc_macro set markdown_hash = encode(digest (markdown, 'sha256'), 'base64');

Then ran yarn ai-help-macros update-index locally.

@caugner caugner requested a review from a team as a code owner March 4, 2024 17:19
Copy link
Contributor Author

@caugner caugner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-approval to avoid merge conflicts, since I'm going to update the script to integrate the new text-embedding-3-small embedding model.

@caugner caugner merged commit de1aae9 into main Mar 29, 2024
13 checks passed
@caugner caugner deleted the improve-ai-help-indexing branch March 29, 2024 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant