Prototype Persistence provider for AGL persistence API

[SebastianSchildt]

Hello there,

as we do not yet know where the persistence API will end up, and want to write this in Rust/C++ anyway I just open this issue for discussion.

here is a complete - albeit beyond ugly - provider that does everything:

• Starts up and reads a complete VSS json file, and filters out all entires where "persist" is set to true
• Checlk AGL persistence store and in case data exists publish it t KUKSA
• Subscribing all paths and in case of changes pushing it back to persoistence layer

The code

import json
import subprocess
from kuksa_client.grpc import VSSClient
from kuksa_client.grpc import Datapoint

KUKSA_PORT=55556

def extract(path, vssdata, persistPaths):
    #print(f"Extracting {path}")
    if vssdata['type'] == "branch":
        if "children" in vssdata:
            for k, v in vssdata['children'].items():
                extract(path + "." + k, v, persistPaths)
    elif vssdata['type'] == "sensor" or vssdata['type'] == "actuator":
        #print(f"Checking {path}")
        if "persist" in vssdata and vssdata['persist'] is True:
            persistPaths.append(path)
            #print(f"Persisting {path}")
    

def extract_path_to_persist(vssdata, persistPaths):
    extract("Vehicle", vssdata['Vehicle'], persistPaths)

def read_and_publish_to_kuksa(persistPaths):
    for path in persistPaths:
        print(f"Reading {path}")
        output = subprocess.check_output(["grpcurl", "-plaintext", "-d", f'{{"key": "{path}", "namespace": "vss"}}', "localhost:50054", "storage_api.Database/Read"])
        outputDict = json.loads(output)
        with VSSClient('127.0.0.1', KUKSA_PORT) as client:
            if "success" in outputDict and outputDict["success"] is True:
                print(f"Pushing value {outputDict['result']} to {path} is ")
                client.set_current_values({ path: Datapoint(outputDict['result']),
    })
            else:
                print(f"No data for {path}")
   
def main():
    print("Hello Persistent world!")
    # we are reading it from a complete VSS JSON, parsing may
    # be more complicated tha reading a simple config file
    # but the VSS way would be to define a persist field in
    # an overlay, and furthermore, the datatype information
    # might be needed when doing this in Rust or C++ as the 
    # GRPC API of KUKSA is typed for performance reasons
    # (Python bindings just hid this fact. At a cost)
    vssdata = json.load(open("vss_release_4.0.json"))
    persistPaths = []
    extract_path_to_persist(vssdata, persistPaths)

    print(f"Persisting {persistPaths}")

    # try to get path form AGL persistence API and push
    # to KUKSA. 
    read_and_publish_to_kuksa(persistPaths)

    # No we subscribe all Paths and update them in
    # AGL persistence API
    print("Subscribing to all paths")
    with VSSClient('127.0.0.1', KUKSA_PORT) as client:
        for updates in client.subscribe_current_values(persistPaths):
            for path, data in updates.items():
                if data is not None:
                    print(f"Updating {path} with {data.value}")
                    # update path in AGL persistence API
                    subprocess.check_output(["grpcurl", "-plaintext", "-d", f'{{"key": "{path}", "namespace": "vss", "value": "{data.value}"}}', "localhost:50054", "storage_api.Database/Write"])




if __name__ == "__main__":
    main()

This is based on https://github.com/LSchwiedrzik/agl-persistent-storage-api/tree/master

@LSchwiedrzik "just" need to rewirte in Rust.... As seperate component, or maybe an (optional) feature directly in persistent-storage

Databroker needs to run, everything to get started with that should be in https://github.com/eclipse-kuksa/kuksa-databroker/blob/main/doc/quickstart.md

fyi @eriksven @g-scott-murray

[SebastianSchildt]

vss_release_4.0.json
Example VSS with two persist paths:

Vehicle.Cabin.Infotainment.HMI.DisplayOffDuration 
Vehicle.Cabin.Infotainment.Media.Volume

[SebastianSchildt]

@g-scott-murray you commented elsewhere

I'm not much of a fan of having to manually edit a VSS JSON or work with vss-tools overlays

I can understand, that it feels cumbersome. In the end "who does the work, can decide" :D The advantage is, some people like to consolidate all VSS related info in... well VSS....

Maybe an option would be writing an acutal overlay (in YAML), and parsing that. New VSS stools become quite relaxed wrt to "Incomplete" VSS specs as overlay, so a config for the example might look like this

Vehicle.Cabin.Infotainment.HMI.DisplayOffDuration:
    persist: true
    datatype: "uint8"


Vehicle.Cabin.Infotainment.Media.Volume:
    persist: true
    datatype: "uint16"

That might be easily parsed instead of the json in my example (again, actually not even datatype is necessary but I assume it might be needed to type it directly in GRPC).

People who do want a consolidated VSS could do (assuming the file above called test.vspec)

vspec export json -s ../vehicle_signal_specification/spec/VehicleSignalSpecification.vspec -o test.json --pretty --overlays ../agl-persistence-prototype/test.vspec 

But users of the persistence plugin/provider would never need to do that

(side note: I was thinking "persist: false" might still be usfeul for the plugin, maybe there are datapoints we want to populate upon start but NOT subscribe/update. E.g. is somebody wants to set VIN, Vehicle color or such things...)