Implementing Fundamental Indexing

[tom-ludwig]

Overview

This PR introduces indexing, enhancing CE's search capabilities by indexing files and enabling efficient keyword searches.

TL;DR: Significant search time reduction from 5 sec to 0.09 sec (with usage preview) and 0.007 sec (without preview). Indexing process completes in just 0.1 sec. Tested on CE 16.10.23

Description

I've hooked up Search Kit to supercharge CodeEdit's search game. What's that mean for you? Well, we're talking about a behind-the-scenes indexing party. When you throw a search query, CodeEdit now uses this index it built earlier, slashing search times.

For instance, finding 'struct' used to take a sluggish 5 seconds in our 29,000 lines of code. Now? Blink, and you'll miss it—0.09 seconds, even with a usage preview, this not only locates the file but also identifies usages within the found files.! Skip the usage preview, and it's a blink-and-you'll-really-miss-it 0.007 seconds. And guess what? The startup indexing takes just 0.1 seconds.

How It Works

Curious about the magic under the hood? Here's the simplified rundown:
SearchKit diligently maps every term to a document using inverted indexing, creating a handy map of where each term appears. When you search, CodeEdit skips sifting through files directly. Instead, it goes straight to the index, like a well-organised library catalog. Your searches are now lightning-fast. CodeEdit doesn't need to scan every file; it already knows where to find what you're looking for. For more info check out: SearchKit Documentation
and Documentation Archive

FYI: I'll throw in some tests in another PR after this one gets the green light.

Related Issues

• closes File Indexing Functionality #1473
• closes ✨ Find Navigator: Improve project find performance #1464
• closes 🐞 Search Results are in lowercased letters #1484

Checklist

• I read and understood the contributing guide as well as the code of conduct
• The issues this PR addresses are related to each other
• My changes generate no new warnings
• My code builds and runs on my machine
• My changes are all related to the related issue above
• I documented my code
• Enhance async usage for improved efficiency and responsiveness.
• Utilise the function for searching in CE
• Integrate async search result retrieval

Screenshots

[tom-ludwig]

@austincondiff Search on M1 MacBook Air is as quick as on my M1 Max. All good now!

[thecoolwinter]

Found one small bug and a couple quick spelling errors. Super excited to get this merged!

[0xWDG]

LGTM, Nice work 🎉

[austincondiff]

Looks good! Very nice work @activcoding! 👏

Outdated