Eventual consistency

[ll13j1209]

Hi, currently, the delta replica needs to catch up with the patch log server before returning result to the user. Is there a way to support eventual consistency when querying the delta? Assume if one has frequent streaming ingestion going on and strong consistency will increase the response time. Thank you

[afs]

Hi @ll13j1209 ,

Do you have a situation where this actually happens? It would be interesting to hear about the setup in more detail.

It's certainly possible within the design. It has been done in some place that use RDF-Delta server with a modified client.

There was some code to poll for changes instead of sync. That doesn't seem to be active. Noted as issue #241

One way in the current system that night help is to use an "external" setting in the assembler with property delta:dataset. This way, the data is available via route two URLs - one goes straight to database, one goes to a synchronization dataset.

In code there is setting the sync policy to the DeltaConnection. The options are "None" (i.e the app calls sync), "on read" (the normal Fuseki+Delta setup) and "on write" (read transactions don't sync, but the system is synced before a write).
DeltaClient.attachExternal.

[ll13j1209]

Hi @afs , thanks for reaching out! Currently, I have a primary delta server which only takes write request and a cluster of read replica that handles read. All of them sync to the same patch log server.
Sometimes, when I try to query, it takes a long time to get the result. I am suspicious that the sync process might be the reason. Hence the ask.
You mentioned that we could use an "external" setting in the assembler with property delta:dataset and expose two urls. Could you explain a little more in tail how the config would look like? Below is my current config for the read delta instances.

`<#service1> rdf:type fuseki:Service ;
fuseki:name "ds" ;
fuseki:serviceQuery "sparql" ;
fuseki:serviceQuery "query" ;
fuseki:serviceUpdate "update" ;
fuseki:serviceUpload "upload" ;
fuseki:serviceReadWriteGraphStore "data" ;
fuseki:serviceReadGraphStore "get" ;
fuseki:dataset <#dataset> ;
.

<#dataset> rdf:type delta:DeltaDataset ;
delta:changes "${DELTA_SERVER}" ;
delta:patchlog "${PATCHLOG}";
delta:storage "tdb2";
delta:zone "/fuseki";`

Thank you!

[ll13j1209]

BTW, is it possible that in the config, we can use other fuseki type such as DatasetTDB2 instead of delta:DeltaDataset? This way I can have a service point to the same directory without specify a patchlog server for quick serach

[afs]

This is what external is for. It uses delta:dataset to link to another dataset. delta:DeltaDataset is a wrapper.

<#dataset>  rdf:type delta:DeltaDataset ;
...
     delta:storage "external" ;
     delta:dataset  <#otherDataset> ;
     .

<#otherDataset> rdf:type ja:MemoryDataset .

You can (with care) have a fuseki:Service expose the <#otherDataset>. Of course, it won't record changes.

[ll13j1209]

Awesome! Thank you for the clarification!
One more thing, right now, does the config support changing the sync policy? If we want to set the policy to only sync on write request for example. If the config does not support, anyother approach we can use to change the default sync policy?

[afs]

The machinery is there and there are tests.
DeltaConnection.create(,...SyncPolicy.TXN_W) but not exposed in the assembler which calls LibBuildDC.setupDataset.