Merge pull request #236 from boschresearch/feature/load_overlays

Feature/load overlays

doc/liveUpdateVSSTree.md

@@ -1,7 +1,8 @@
 # Updating/Extending VSS structure during runtime
 
 Upon startup the KUKSA.val server loads a JSON file describing the data points it manages. Usually this is a version of the [Geniivi Vehicle Signal Specification](https://github.com/GENIVI/vehicle_signal_specification/).
-If you want to extend or modify this structure in your application, it is not neccesary to modify Genivi VSS, instead you can extend the loaded VSS structure during runtime.
+
+If you want to extend or modify the VSS structure from your application during runtime, this is possible using the `updateVSSTree` and `updateMetaData` requests described here.
 
 ## Overlays using updateVSSTree
 The updateVSSTree method takes an input JSON that follows the same rules as the initial VSS JSON. This is added as an _overlay_ meaning:
@@ -72,4 +73,4 @@ updateMetaData updateMetaData Vehicle.Speed '{"max":9999}'
 As the Input data to `updateVSSTree` as well as `updateMetaData`is merged with the existing data structure, there is no way to _remove_ previously added elements.
 
 **This feature schould be considered Beta quality.**
-Similar to the initial JSON, there is not much input verification going on. We _check_ that you provide syntactically valid JSON, we _trust_ that it is a valid VSS structure. Therefore, as a system desing suggestion, we recommend you perform any neccesary, potentially dangerous, data structure modifcations during an initialization phase.
+Similar to the initial JSON, there is not much input verification going on. We _check_ that you provide syntactically valid JSON, we _trust_ that it is a valid VSS structure. Therefore, as a system design suggestion, we recommend you perform any neccesary, potentially dangerous, data structure modifcations during system initialization phase, preferably using the `--overlays` parameter of `kuksa-val-server`.

doc/usage.md

@@ -4,7 +4,6 @@ Using `./kuksa-val-server -h`, you can get a list of supported options:
 
 ```
 Usage: ./kuksa-val-server OPTIONS
-
 OPTIONS:
   -h [ --help ]                         Help screen
   -c [ --config-file ] arg (="config.ini")
@@ -24,10 +23,14 @@ OPTIONS:
   --vss arg                             [mandatory] Path to VSS data file 
                                         describing VSS data tree structure 
                                         which `kuksa-val-server` shall handle. 
-                                        Sample 'vss_release_2.0.json' file can be 
-                                        found under [unit-test](./data/vss-core/
-                                        vss_release_2.0.json)
-                                        
+                                        Sample 'vss_release_2.0.json' file can 
+                                        be found under [data](./data/vss-core/v
+                                        ss_release_2.0.json)
+  --overlays arg                        Path to a directory cotaiing additional
+                                        VSS models. All json files will be 
+                                        applied on top of the main vss file 
+                                        given by the -vss parameter in 
+                                        alphanumerical order
   --cert-path arg (=".")                [mandatory] Directory path where 
                                         'Server.pem', 'Server.key' and 
                                         'jwt.key.pub' are located. 
@@ -46,18 +49,13 @@ OPTIONS:
   --port arg (=8090)                    If provided, `kuksa-val-server` shall 
                                         use different server port than default 
                                         '8090' value
-  --record (=0)                         Enable record feature for replaying into
-                                        the server later on. Record file will be 
-                                        stored to the specified path with timestamp.
-                                        File format is *.csv
-                                        possible values:
-                                          - noRecord: no Recording will be applied
-                                          - recordSet: Record Set Value (inputs of the server)
-                                          - recordSetandGet: Record Set Value and Get Value (in- and outputs of the server)
-                      
-  --record-path (='.')                  Specify path where the record file should 
-                                        be stored to. 
-
+  --record arg (=noRecord)              Enables recording into log file, for 
+                                        later being replayed into the server 
+                                        noRecord: no data will be recorded
+                                        recordSet: record setting values only
+                                        recordSetAndGet: record getting value 
+                                        and setting value
+  --record-path arg (=.)                Specifies record file path.
   --log-level arg                       Enable selected log level value. To 
                                         allow for different log level 
                                         combinations, parameter can be provided
@@ -81,7 +79,7 @@ MQTT Options:
                                         client. Defaults to 60
   --mqtt.retry arg (=3)                 Times of retry via connections. 
                                         Defaults to 3
-  --mqtt.topic-prefix arg               Prefix to add for each mqtt topics
+  --mqtt.topic-prefix arg (=vss)        Prefix to add for each mqtt topics
   --mqtt.publish arg                    List of vss data path (using readable 
                                         format with `.`) to be published to 
                                         mqtt broker, using ";" to seperate 

include/OverlayLoader.hpp

@@ -0,0 +1,28 @@
+/*
+ * ******************************************************************************
+ * Copyright (c) 2021 Robert Bosch GmbH.
+ *
+ * All rights reserved. This program and the accompanying materials
+ * are made available under the terms of the Eclipse Public License v2.0
+ * which accompanies this distribution, and is available at
+ * https://www.eclipse.org/org/documents/epl-2.0/index.php
+ *
+ * *****************************************************************************
+ */
+
+#include <vector>
+#include <boost/filesystem.hpp>
+
+#include "ILogger.hpp"
+#include "IVssDatabase.hpp"
+
+
+/** Finding overlays to load. Returns an alphanumerically list of paths pointing
+ * to *.json files in  directory 
+ */
+std::vector<boost::filesystem::path> gatherOverlays(std::shared_ptr<ILogger> log, boost::filesystem::path directory);
+
+/** Iterates over paths in vector, tries to parse as JSON and merge with existing
+ *  structure in database
+ */
+void applyOverlays(std::shared_ptr<ILogger> log, const std::vector<boost::filesystem::path> overlayfiles, std::shared_ptr<IVssDatabase> db);

src/OverlayLoader.cpp

@@ -0,0 +1,68 @@
+/*
+ * ******************************************************************************
+ * Copyright (c) 2021 Robert Bosch GmbH.
+ *
+ * All rights reserved. This program and the accompanying materials
+ * are made available under the terms of the Eclipse Public License v2.0
+ * which accompanies this distribution, and is available at
+ * https://www.eclipse.org/org/documents/epl-2.0/index.php
+ *
+ * *****************************************************************************
+ */
+
+/* helper fuctions to load overlays during server startup */
+
+#include <stdexcept>
+#include <jsoncons/json.hpp>
+
+#include "OverlayLoader.hpp"
+#include "kuksa.pb.h"
+
+using jsoncons::json;
+
+std::vector<boost::filesystem::path> gatherOverlays(
+    std::shared_ptr<ILogger> logger, boost::filesystem::path overlaydir) {
+  std::vector<boost::filesystem::path> overlayfiles;
+  logger->Log(LogLevel::VERBOSE,
+              "Searching overlays in \"" + overlaydir.generic_string() + "\"");
+  if (!boost::filesystem::is_directory(overlaydir)) {
+    throw std::runtime_error("Overlays need to be a directory. \"" +
+                             overlaydir.generic_string() + "\" is not.");
+  }
+  for (auto &p : boost::filesystem::directory_iterator(overlaydir)) {
+    if (boost::filesystem::is_regular_file(p.path()) &&
+        p.path().extension() == ".json") {
+      logger->Log(LogLevel::VERBOSE, "Found " + p.path().generic_string());
+      overlayfiles.push_back(p);
+    }
+  }
+
+  // Sort according to filename
+  std::sort(overlayfiles.begin(), overlayfiles.end(),
+            [](boost::filesystem::path a, boost::filesystem::path b) {
+              return a.filename() < b.filename();
+            });
+
+  return overlayfiles;
+}
+
+void applyOverlays(std::shared_ptr<ILogger> log,
+                   const std::vector<boost::filesystem::path> overlayfiles,
+                   std::shared_ptr<IVssDatabase> db) {
+  // Mock a channel with permissions to change VSS tree
+  auto mockChannel = std::make_shared<kuksa::kuksaChannel>();
+  mockChannel->set_modifytree(true);
+
+  for (auto const &p : overlayfiles) {
+    log->Log(LogLevel::INFO, "Loading overlay \"" + p.generic_string() + "\"");
+    ;
+    try {
+      std::ifstream is(p.generic_string());
+      jsoncons::json overlay = json::parse(is);
+      db->updateJsonTree(*mockChannel, overlay);
+    } catch (std::exception &e) {
+      throw std::runtime_error("Error loading \"" + p.generic_string() +
+                               "\": " + e.what());
+    }
+  }
+}

src/main.cpp

@@ -1,6 +1,6 @@
 /*
  * ******************************************************************************
- * Copyright (c) 2019-2020 Robert Bosch GmbH.
+ * Copyright (c) 2019-2021 Robert Bosch GmbH.
  *
  * All rights reserved. This program and the accompanying materials
  * are made available under the terms of the Eclipse Public License v2.0
@@ -39,6 +39,8 @@
 #include "MQTTPublisher.hpp"
 #include "exception.hpp"
 #include "grpcHandler.hpp"
+#include "OverlayLoader.hpp"
+
 
 #include "../buildinfo.h"
 
@@ -107,6 +109,7 @@ int main(int argc, const char *argv[]) {
       "cert-path = . \n"
       "log-level = ALL\n")
     ("vss", program_options::value<boost::filesystem::path>()->required(), "[mandatory] Path to VSS data file describing VSS data tree structure which `kuksa-val-server` shall handle. Sample 'vss_release_2.0.json' file can be found under [data](./data/vss-core/vss_release_2.0.json)")
+    ("overlays", program_options::value<boost::filesystem::path>(), "Path to a directory cotaiing additional VSS models. All json files will be applied on top of the main vss file given by the -vss parameter in alphanumerical order")
     ("cert-path", program_options::value<boost::filesystem::path>()->required()->default_value(boost::filesystem::path(".")),
       "[mandatory] Directory path where 'Server.pem', 'Server.key' and 'jwt.key.pub' are located. ")
     ("insecure", program_options::bool_switch()->default_value(false), "By default, `kuksa-val-server` shall accept only SSL (TLS) secured connections. If provided, `kuksa-val-server` shall also accept plain un-secured connections for Web-Socket and REST API connections, and also shall not fail connections due to self-signed certificates.")
@@ -188,6 +191,14 @@ int main(int argc, const char *argv[]) {
   try {
     // initialize server
     auto vss_path = variables["vss"].as<boost::filesystem::path>();
+
+    //check overlaypath
+    std::vector<boost::filesystem::path> overlayfiles;
+    if ( variables.count("overlays") )
+    {
+      overlayfiles = gatherOverlays(logger, variables["overlays"].as<boost::filesystem::path>());
+    }
+
     // initialize pseudo random number generator
     std::srand(std::time(nullptr));
 
@@ -236,6 +247,7 @@ int main(int argc, const char *argv[]) {
         logger, database, tokenValidator, accessCheck, subHandler);
 
     database->initJsonTree(vss_path);
+    applyOverlays(logger, overlayfiles ,database);
 
     if(variables.count("mqtt.publish")){
         string path_to_publish = variables["mqtt.publish"].as<string>();