GH-528 cache the locations in the super blocks #529

hmottestad · 2024-11-27T11:36:20Z

Issue resolved (if any): #528

Description of this pull request:

Please check all the lines before posting the pull request:

I've created tests for all my changes
My pull request isn't fixing or changing multiple unlinked elements (please create one pull request for each element)
I've applied the code formatter (mvn formatter:format on the backend, npm run format on the frontend) before posting my pull request, mvn formatter:validate to validate the formatting on the backend, npm run validate on the frontend
All my commits have relevant names
I've squashed my commits (if necessary)

hmottestad · 2024-11-27T11:38:50Z

qendpoint-core/src/main/java/com/the_qa_company/qendpoint/core/compact/bitmap/Bitmap375Big.java

+
+	@Override
+	public String toString() {
+		return "Bitmap375Big{}";
+	}


IntelliJ likes to show the objects in the debugger, which is usually a good idea except that Bitmap375Big is usually a very very big object...so overriding toString() forces IntelliJ to use the results of that instead of analyzing and displaying the entire object.

qendpoint-core/src/main/java/com/the_qa_company/qendpoint/core/compact/bitmap/Bitmap375Big.java

hmottestad · 2024-11-27T11:43:26Z

qendpoint-core/src/main/java/com/the_qa_company/qendpoint/core/compact/bitmap/Bitmap375Big.java

+			mid = (min + max) / 2;
+		}
+
+		prevFound[index] = min;


Here we are filling the prevFound (cache) dynamically, but it should be possible to do this at startup instead.

...int-core/src/main/java/com/the_qa_company/qendpoint/core/compact/sequence/SequenceInt32.java

hmottestad · 2024-12-05T10:44:15Z

@ate47 This is still work in progress, but I've implemented an approach that scales with the size of the data and calculates the values at startup. I found out that a lot of the performance improvements come from dynamically updating the cache while the query is running, since the next value will usually be located very close to where we found the previous value. Also found a useful little hack where I on the second and third iteration of the binary search read the value at min + 1 to see if that allows us to shorten the number of iterations.

With the EU dataset I'm seeing a decent improvement in performance:

I'm using qendpoint.mergejoin=true.

The query below goes from 4.592 seconds to 2.959 seconds

select (count(distinct ?a) as ?count) where {
    ?a <https://linkedopendata.eu/prop/P20> ?b.
    ?b a <http://wikiba.se/ontology#BestRank>.
    ?a <http://schema.org/dateModified> ?dt.
}

The query below goes from 1.042 seconds to 0.708 seconds

SELECT (COUNT(DISTINCT ?s0) as ?c ) WHERE {
    ?s0 <https://linkedopendata.eu/prop/direct/P35> <https://linkedopendata.eu/entity/Q9934> .
    ?s0 <https://linkedopendata.eu/prop/direct/P32> <https://linkedopendata.eu/entity/Q27> .
    {
        ?s0 <https://linkedopendata.eu/prop/direct/P888> ?category .
        ?category <https://linkedopendata.eu/prop/direct/P1849> <https://linkedopendata.eu/entity/Q2547985> .
    } UNION {
        ?s0 <https://linkedopendata.eu/prop/direct/P1849> <https://linkedopendata.eu/entity/Q2547985> .
    }
}

The query below goes from 1.437 seconds to 1.082 seconds

select (count(*) as ?count) where {
    ?a a <http://wikiba.se/ontology#Item>.
    ?a <http://www.w3.org/2000/01/rdf-schema#label> ?prop1.
}

The query below goes from 17.308 seconds to 9.892 seconds

This query uses a combination of nested loop join (JoinIterator) and merge join, and it was actually faster when merge join was disabled (13.929 seconds). Without merge join but with the new binary search it runs in 10.723 seconds and with both merge join and the new binary search is runs in 9.892 seconds. So now it's faster with merge join :)

select (count(?a) as ?count) where {
    ?a a <http://www.w3.org/2002/07/owl#Restriction>.
    ?d a <http://www.w3.org/2002/07/owl#Restriction>.
    ?a <http://www.w3.org/2002/07/owl#onProperty> ?prop1.
    ?d <http://www.w3.org/2002/07/owl#onProperty> ?prop2.

    ?a ?p <http://www.w3.org/2002/07/owl#Thing>.
    ?d ?r <http://www.w3.org/2002/07/owl#Thing>.

    FILTER(?a != ?d)  
}

The query below goes from 2.068 seconds to 1.801 seconds

Query uses merge join!

select (count(*) as ?count) where {
    ?a a <http://www.w3.org/2002/07/owl#Restriction>.
    ?d a <http://www.w3.org/2002/07/owl#Restriction>.
    ?a <http://www.w3.org/2002/07/owl#onProperty> ?prop1.
    ?d <http://www.w3.org/2002/07/owl#onProperty> ?prop2.

    FILTER(?a != ?d)  
}

The query below goes from 1.13 seconds to 1.129 seconds

Query uses merge join, which is this case is probably the reason why it's not any faster with the new binary search. When I run it with the old binary search and with merge join disabled it runs in 6.887 seconds, so merge join makes a big difference here!

select (count(*) as ?count) where {
	?a <https://linkedopendata.eu/prop/P35> ?b.
  	?a <http://schema.org/dateModified> ?c.
  	?a <http://schema.org/version> ?d.
  	?a <http://wikiba.se/ontology#statements> ?e.
  	?a <https://linkedopendata.eu/prop/P32> ?h.
}

PS: If you want to use the old binary search: java -DuseOldBinarySearch=true -Xms32G -Xmx32G -jar qendpoint-backend-2.2.0-exec.jar

hmottestad · 2024-12-05T10:46:03Z

And I've rebased this branch on dev, so it contains the previous improvements that were merged. All the above results are from after the rebase.

hmottestad · 2024-12-11T08:58:53Z

@ate47 I saw an interface called DynamicSequence that extends LongArray. Since it's called "dynamic" I assume that it wouldn't be safe for us to implement the caching of the estimated location since the cache might not be re-calculated when things change. Is this a correct assumption?

hmottestad · 2024-12-11T09:12:59Z

Updated results

The query below goes from 4.592 seconds to 2.265 seconds

select (count(distinct ?a) as ?count) where {
    ?a <https://linkedopendata.eu/prop/P20> ?b.
    ?b a <http://wikiba.se/ontology#BestRank>.
    ?a <http://schema.org/dateModified> ?dt.
}

The query below goes from 1.042 seconds to 0.481 seconds

SELECT (COUNT(DISTINCT ?s0) as ?c ) WHERE {
    ?s0 <https://linkedopendata.eu/prop/direct/P35> <https://linkedopendata.eu/entity/Q9934> .
    ?s0 <https://linkedopendata.eu/prop/direct/P32> <https://linkedopendata.eu/entity/Q27> .
    {
        ?s0 <https://linkedopendata.eu/prop/direct/P888> ?category .
        ?category <https://linkedopendata.eu/prop/direct/P1849> <https://linkedopendata.eu/entity/Q2547985> .
    } UNION {
        ?s0 <https://linkedopendata.eu/prop/direct/P1849> <https://linkedopendata.eu/entity/Q2547985> .
    }
}

The query below goes from 1.437 seconds to 0.933 seconds

select (count(*) as ?count) where {
    ?a a <http://wikiba.se/ontology#Item>.
    ?a <http://www.w3.org/2000/01/rdf-schema#label> ?prop1.
}

The query below goes from 17.308 seconds to 8.204 seconds

select (count(?a) as ?count) where {
    ?a a <http://www.w3.org/2002/07/owl#Restriction>.
    ?d a <http://www.w3.org/2002/07/owl#Restriction>.
    ?a <http://www.w3.org/2002/07/owl#onProperty> ?prop1.
    ?d <http://www.w3.org/2002/07/owl#onProperty> ?prop2.

    ?a ?p <http://www.w3.org/2002/07/owl#Thing>.
    ?d ?r <http://www.w3.org/2002/07/owl#Thing>.

    FILTER(?a != ?d)  
}

hmottestad · 2024-12-11T09:56:22Z

qendpoint-store/src/main/java/com/the_qa_company/qendpoint/store/EndpointTripleSource.java

-		if (subjectID == 0 || subjectID == -1) {
-			newSubj = subj;
-		} else {
-			newSubj = this.endpoint.getHdtConverter().subjectIdToIRI(subjectID);
-		}
-		if (predicateID == 0 || predicateID == -1) {
-			newPred = pred;
-		} else {
-			newPred = this.endpoint.getHdtConverter().predicateIdToIRI(predicateID);
-		}
-		if (objectID == 0 || objectID == -1) {
-			newObj = obj;
-		} else {
-			newObj = this.endpoint.getHdtConverter().objectIdToIRI(objectID);
-		}
-
 		if (graph) {
 			graphID = new long[contexts.length];
-			newContextes = this.endpoint.getHdtConverter().graphIdToIRI(contexts, graphID);


If the NativeStore isn't in use then we don't need these variables. So it's simplest if we can delay creating them until we are sure we need them.

hmottestad · 2024-12-11T09:58:39Z

qendpoint-core/src/main/java/com/the_qa_company/qendpoint/core/util/BitUtil.java

-			rank -= value & 1;
-			bitpos++;
-			value >>>= 1;
+			int trailingZeros = Long.numberOfTrailingZeros(value);


This is faster. Not certain why, but I assume that the Long.numberOfTrailingZeros(value) method uses a SIMD instruction to check multiple bits in a single operation.

… track of the estimated location of values in the underlying long array. This assumes that the values are ordered.

hmottestad · 2024-12-11T10:49:26Z

@ate47 This PR should be ready for review now.

ate47

Except my 2 comments, these changes are fine for me.

...e/src/main/java/com/the_qa_company/qendpoint/core/util/io/compress/WriteLongArrayBuffer.java

ate47 · 2024-12-11T13:09:28Z

qendpoint-backend/pom.xml


        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>

    <dependencies>
+        <dependency>


Any reason for this add? Was this part of a compilation issue?

Runtime issue actually. Complained of a method not found when generating the query explanation json. The project inherits two different versions of Jackson, so we apparently need to decide which one to use.

It would be cleaner to use dependency management instead, but I couldn't find that in any of the poms. Maybe I overlooked something?

hmottestad changed the base branch from master to dev November 27, 2024 11:36

hmottestad commented Nov 27, 2024

View reviewed changes

qendpoint-core/src/main/java/com/the_qa_company/qendpoint/core/compact/bitmap/Bitmap375Big.java Outdated Show resolved Hide resolved

hmottestad commented Nov 27, 2024

View reviewed changes

qendpoint-core/src/main/java/com/the_qa_company/qendpoint/core/compact/bitmap/Bitmap375Big.java Outdated Show resolved Hide resolved

hmottestad commented Nov 27, 2024

View reviewed changes

...int-core/src/main/java/com/the_qa_company/qendpoint/core/compact/sequence/SequenceInt32.java Outdated Show resolved Hide resolved

hmottestad force-pushed the GH-528-cache-estimated-locations-in-super-blocks branch from f7e8e5e to ee858be Compare December 4, 2024 10:37

we need to specify the version of jackson

f5bb5ce

hmottestad force-pushed the GH-528-cache-estimated-locations-in-super-blocks branch from ebe036a to dbb90b7 Compare December 11, 2024 08:00

hmottestad commented Dec 11, 2024

View reviewed changes

hmottestad added 3 commits December 11, 2024 11:01

the-qa-companyGH-528 improve binary search in super blocks by keeping…

5a45912

… track of the estimated location of values in the underlying long array. This assumes that the values are ordered.

improved bit shifting in select1 by using Long.numberOfTrailingZeroes

9b7e6e2

only create the new Values if we are going to query the native store

4d56b27

hmottestad force-pushed the GH-528-cache-estimated-locations-in-super-blocks branch from dafcffc to 4d56b27 Compare December 11, 2024 10:02

reduce memory pressure

342e3e8

hmottestad force-pushed the GH-528-cache-estimated-locations-in-super-blocks branch from 5fc81db to 342e3e8 Compare December 11, 2024 10:36

ate47 self-requested a review December 11, 2024 12:47

ate47 requested changes Dec 11, 2024

View reviewed changes

revert some changes based on review

c97d4f1

ate47 approved these changes Dec 12, 2024

View reviewed changes

ate47 merged commit 2492a84 into the-qa-company:dev Dec 12, 2024
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-528 cache the locations in the super blocks #529

GH-528 cache the locations in the super blocks #529

hmottestad commented Nov 27, 2024

hmottestad Nov 27, 2024

hmottestad Nov 27, 2024

hmottestad commented Dec 5, 2024 •

edited

Loading

hmottestad commented Dec 5, 2024 •

edited

Loading

hmottestad commented Dec 11, 2024

hmottestad commented Dec 11, 2024 •

edited

Loading

hmottestad Dec 11, 2024

hmottestad Dec 11, 2024

hmottestad commented Dec 11, 2024

ate47 left a comment

ate47 Dec 11, 2024

hmottestad Dec 11, 2024

GH-528 cache the locations in the super blocks #529

GH-528 cache the locations in the super blocks #529

Conversation

hmottestad commented Nov 27, 2024

hmottestad Nov 27, 2024

Choose a reason for hiding this comment

hmottestad Nov 27, 2024

Choose a reason for hiding this comment

hmottestad commented Dec 5, 2024 • edited Loading

The query below goes from 4.592 seconds to 2.959 seconds

The query below goes from 1.042 seconds to 0.708 seconds

The query below goes from 1.437 seconds to 1.082 seconds

The query below goes from 17.308 seconds to 9.892 seconds

The query below goes from 2.068 seconds to 1.801 seconds

The query below goes from 1.13 seconds to 1.129 seconds

hmottestad commented Dec 5, 2024 • edited Loading

hmottestad commented Dec 11, 2024

hmottestad commented Dec 11, 2024 • edited Loading

Updated results

The query below goes from 4.592 seconds to 2.265 seconds

The query below goes from 1.042 seconds to 0.481 seconds

The query below goes from 1.437 seconds to 0.933 seconds

The query below goes from 17.308 seconds to 8.204 seconds

hmottestad Dec 11, 2024

Choose a reason for hiding this comment

hmottestad Dec 11, 2024

Choose a reason for hiding this comment

hmottestad commented Dec 11, 2024

ate47 left a comment

Choose a reason for hiding this comment

ate47 Dec 11, 2024

Choose a reason for hiding this comment

hmottestad Dec 11, 2024

Choose a reason for hiding this comment

hmottestad commented Dec 5, 2024 •

edited

Loading

hmottestad commented Dec 5, 2024 •

edited

Loading

hmottestad commented Dec 11, 2024 •

edited

Loading