The index integration is progressing ..

[admtech]

In the current version 1.4 many PRs for the index and the search have been added. I am very happy about this, because I see the index as very important for DragondlyDB. Even a simple index makes handling the data much easier. Sorting and filtering by tags without loading modules makes FireflyDB very interesting. This has already worked well with JSON support. We now use DragonflyDb in production with over 4M keys and especially loading and saving snapshots is much better and much much faster than with Redis.

Are there already a few things you can test on the index topic?
FT. CREATE and FT.SEARCH are implemented, but the index doesn't seem to work yet. FT.DROPINDEX or FT._LIST do not work yet. With these basics you could already test a little bit.

Thanks for your work.

[dranikpg]

Hi!

Search and indexing are currently in the very early stages of development. Basically, we're implementing only core features and are laying the grounds for efficient indexing.

Many practical commands like deleting an index, listing available, etc are missing. Almost all command options are missing too. Most importantly many core features are missing: stemming, scoring, sorting, aggregating, index compression.

However it should already be possible to construct an index with FT.CREATE and query it with FT.QUERY. I recently ran benchmarks with up to a million indexed keys on fairly basic queries (text/tags and 1-3 intersections or unions) - performance should be more or less comparable to RedisSearch.

How did you try querying? You can also take a look at our tests to see what cases should work (for example TEST_F(SearchFamilyTest, Tags)):

https://github.com/dragonflydb/dragonfly/blob/main/src/server/search/search_family_test.cc

---

We are really interested in your use case so we can adapt our future development in that area 🚀

First we should match on core features, that require possibly more work and time to implement. Then we can add all auxiliary commands and options required to fully use those features

Some basic questions:

• I see you only mentioned tags which are a very basic indexing type. Do you intend to use any other data types (text, numbers, vectors)?
• Can you tell us a little more about the queries you're going to run? Is it simply filtering by a tag or does it include: intersections, unions, negations, numeric queries, etc?
• How much data do you plan to index (all 4M keys?)
• Do you plan to sort, score or aggregate?

Main core features (tag index and query execution) should be almost ready. There can be query scenarios that will perform worse (negation + intersection for example). The most important main feature that is missing is index compression, which allows to shrink the index size by a factor of two or three.

Happy to see interest in this new feature 🙂

[admtech]

Hi,

I see you only mentioned tags which are a very basic indexing type. Do you intend to use any other data types (text, numbers, vectors)?

Yes, of course. We have different scenarios like

• A full text search with tags like "instructions", "questions", etc
• A job search by text and geo distance, with tags like full-time, part-time, etc.
• An index of a specific area with tags like "solved" or "not yet solved" or sorted by "creation date" or "last update". With limits and offset for "browsing" within areas.
• Easy searching in the FAQs with tags like "General", "Members", etc.

Different sorts, limits and offsets are also used. Text and numbers are definitely included. Vectors we have not currently used, but the topic is certainly interesting.

Can you tell us a little more about the queries you're going to run? Is it simply filtering by a tag or does it include: intersections, unions, negations, numeric queries, etc?

Mainly filtering by tags or numbers. We use intersections and unions, but rarely. Negations rarely, numeric queries we use for timestamps or filtering by statistic (more than x comments, more than x profile views, etc.).

How much data do you plan to index (all 4M keys?)

In total, there are about 4 million data. But divided into different indexes. One index e.g. for the content, one index for the users, one index for comments, one index for news, etc.

Do you plan to sort, score or aggregate?

Sorting and score, of course. Aggregate would be great - but we take what comes :-)

Main core features (tag index and query execution) should be almost ready. There can be query scenarios that will perform worse (negation + intersection for example).

I personally still miss the Basic commands FT.INFO, FT._LIST and FT.DROPINDEX. Otherwise you can't see what already exists or has been tested. At the moment the index is not yet permanent. So that there is no index anymore when restarting. Helpful would also be an info about the status of the indexing of the existing prefix. With large contents one does not know otherwise, how much are already indexed.

But first tests show that it already works quite well. The possibilities to sort by score or a sortable tag, numbers (e.g. timestamp),etc. should be the next steps. If these few basics are fulfilled, you can test the whole thing properly. No matter if features are still missing.

Happy to see interest in this new feature

Anytime. If you want us to test something, just let us know.

greetings from Germany :-)