Refactoring the persistence layer to be able to persist any Java Obje…

…ct (#407)

* Refactoring the Persistence layer

Fixes: #399

The persistence layer takes care of durably storing the RCA results (and in future decisions from deciders) and providing them when asked for it. The layer also takes care of periodic file rotations and cleaning up old DB files.

Although, it wasn't intended, the _Persistence_ layer is tightly coupled to the _RCA graph_ today. If you look at the `write()` method in  `Persistable` interface, you will find this signature:
`<T extends ResourceFlowUnit> void write(Node<?> node, T flowUnit) throws SQLException, IOException;`

So the Persistable Object knows about the graph node. Therefore, in future as the remediation system evolves and we have components, that are not nodes in the RCA-Graph, we might still want to persist their outputs and this interface can't do that.

Therefore, the goal is to make the methods of Persistable generic, so that it can persist any object given to it and be able to read out any object the caller asks for.

Say we have to persist this class:
```java

Outer.java
________________

class Outer {
  int x;
  int y;
  boolean z;
  String name;
  List<T> myList;

 B b_obj;
}
-------------------------------------------------------

A.java
_______

class B {
  int x;
  String y;
}
```

when the above code is annotated as:
```java

Outer.java
________________

// This will create a table named "Outer"
@table
class Outer {
  int x;
  int y;
  // Its not annotated as column, therefore, it will not be persisted.
  boolean z;
  String name;
  B b_obj;
  List<T> myList;

  // Add a column named x to Outer table. The value stored in the row for that column will be the value of x.
  @column
  int getX() { .. }

  @column
  int getY() { .. }

  @column
   String getName() { .. }

  // For all fields that are not [primitive types](https://docs.oracle.com/javase/6/docs/api/java/lang/Class.html#isPrimitive()), they will be persisted in their own tables as a new row. The auto-increment ID
for this row will be persisted as a value of the column named "__table__B", so that we have a link to the nested element/table from this table.
  @column
  B getBObj() {..}

// If the Column annotation exists for a collection type, then they will be written to a table of their own.
// The name of the table is obtained by calling T.class.getSimpleName(). If all of them evaluates to the same
// table name, then they get written to the same table as different rows. And the corresponding Outer table's
// column will have a string [<rowId1]>, <rowId2>].
// If `T.class.getSimpleName()` evaluates to different names, then they are linked back in Outer table as
// different columns.
  @column
  List<T> getMyList() { .. }
}
-------------------------------------------------------

B.java
_______

@table
class B {
  int x;
  String y;
}
```

this creates these set of tables:
Here assuming that

`getMyList()` returns a list of three elements where

```java
for (T in getMyList) {
  // T.getClass().getSimpleName() returns `T` for two of the objects and `TT` for the third.
}
```

Table Outer
timestamp|ID|X|Y|Name|__table__BObj|__table__T|__table__TT
--|--|--|-|------|--------------|----------------|---
..|53|1|2|name|34|[23,24]|32

Table B
timestamp|ID|X|Y
--|--|--|--
..|34|23|y

Table T
timestamp|ID| col2| ..
--|---|-|--------
..|23|24|..
..|24|24|..

Table TT
timestamp|ID| col2| ..
--|---|-|--------
..|32|24|..

- The Java code resembles how the tables and nested tables will be laid out.
- Because the nested table can be obtained just by reading the column name, someone can write a tool to read the RCAs from the SQLite files. The won't need the java package to figure out the nesting.
- If new columns or nestings are added, they can also be added in the DB without schema mismatch. This is because, we create a new SQLite file on each restart of the RCA agent process (on top of periodic file rotations).
- The table name and column names are derived from the classname and getter name (methodName with `get` stripped out). This provides a 1-1 mapping from the persistor class and fields to the table name and column names.
- Over and above, we are able to persist any Java Object.

* Better comments and error message handling

* Adding some more tests

* Addressing PR comments

* Added test to cover list of ints

* Added more comments for readability and addressing PR review comments

src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/rca/persistence/Persistable.java

@@ -20,11 +20,15 @@
 import com.google.common.annotations.VisibleForTesting;
 import com.google.gson.JsonElement;
 import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
 import java.sql.SQLException;
 import java.util.List;
 import java.util.Map;
+import org.checkerframework.checker.nullness.qual.NonNull;
+import org.checkerframework.checker.nullness.qual.Nullable;
 import org.jooq.Record;
 import org.jooq.Result;
+import org.jooq.exception.DataAccessException;
 
 public interface Persistable {
   /**
@@ -48,6 +52,21 @@ public interface Persistable {
    */
   JsonElement read(String rca);
 
+  /**
+   * This API reads the latest row from the table corresponding to the Object.
+   * @param clz The Class whose Object is desired.
+   * @param <T> The generic type of the class
+   * @return An instantiated Object of the class with the fields populated with the data from the latest row in the table and other
+   *     referenced tables or null if the table does not exist yet.
+   * @throws NoSuchMethodException If the expected setter does not exist.
+   * @throws IllegalAccessException If the setter is not Public
+   * @throws InvocationTargetException If invoking the setter by reflection threw an exception.
+   * @throws InstantiationException Creating an Object of the class failed for some reason.
+   * @throws DataAccessException Thrown by the DB layer.
+   */
+  <T> @Nullable T read(Class<T> clz)
+      throws NoSuchMethodException, IllegalAccessException, InvocationTargetException, InstantiationException, DataAccessException;
+
   /**
    * Write data to the database.
    *
@@ -56,6 +75,36 @@ public interface Persistable {
    */
   <T extends ResourceFlowUnit> void write(Node<?> node, T flowUnit) throws SQLException, IOException;
 
+  /**
+   * This API helps us write any Java Object into the Database and it does not need to be a Graph node anymore.
+   * This is required because:
+   * - in future we would like to persist 'remediation Actions' and they are not necessarily
+   *    FlowUnits or Graph nodes.
+   * - This removes de-coupling of the persistence layer from the RCA runtime.
+   * Restrictions on the Object
+   * --------------------------
+   * The Object that can be persisted has certain restrictions:
+   * - The Fields of the class that are to be persisted should be annotated with '@ValueColumn' or '@RefColumn'.
+   *    A Field should be annotated as @ValueColumn if the field type is a Java primitive type or String type.
+   *    The accepted types are boolean, byte, char, short, int, long, float, and double and their Boxed counterparts.
+   *    A Column can be marked as @RefColumn if the class Field is an user defined Java Class or a collection of the same.
+   * - The annotated Fields should have a 'getter' and a 'setter'. getters are expected to be named as 'get' or 'is'
+   *     prepended to the _capitalized_ fieldName. And the setters are expected to be named 'set' and the capitalized
+   *     field name. The type of the field should match the type of the getter's return type or the setter's argument
+   *     type. Remember, int and Integer are two different types. Therefore, if the field is of type 'int' and your
+   *     'getter' returns a value of type Integer, you will still get NoSuchMethodException.
+   * - The annotated fields and the getter/setters should be declared in the Object and not in a Super class of it.
+   * @param object The Object that needs to be persisted.
+   * @param <T> The type of the Object.
+   * @throws SQLException This is thrown if the underlying DB throws an error.
+   * @throws IOException This is thrown in case the DB file rotation fails.
+   * @throws IllegalAccessException If the getters and setters of the fields to be persisted are not Public.
+   * @throws NoSuchMethodException If getter or setters don't exist with the expected naming convention.
+   * @throws InvocationTargetException Invoking the getter or setter throws an exception.
+   */
+  <T> void write(@NonNull T object) throws SQLException, IOException, IllegalAccessException, NoSuchMethodException,
+      InvocationTargetException;
+
   void close() throws SQLException;
 
   /**

src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/rca/persistence/PersistorBase.java

@@ -23,6 +23,7 @@
 import com.google.gson.JsonElement;
 import java.io.File;
 import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.sql.Connection;
@@ -34,6 +35,7 @@
 import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
+import java.util.Objects;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
 import org.apache.logging.log4j.LogManager;
@@ -188,6 +190,30 @@ public synchronized <T extends ResourceFlowUnit> void write(Node<?> node, T flow
     }
   }
 
+  public synchronized <T> void write(T obj)
+      throws SQLException, IOException, IllegalAccessException, NoSuchMethodException, InvocationTargetException {
+    Objects.requireNonNull(obj);
+    rotateRegisterGarbageThenCreateNewDB(RotationType.TRY_ROTATE);
+    try {
+      writeImpl(obj);
+    } catch (IllegalStateException | IllegalArgumentException | IllegalAccessException | InvocationTargetException | NoSuchMethodException
+        illegalEx) {
+      throw illegalEx;
+    } catch (SQLException e) {
+      LOG.info("RCA: Fail to write.", e);
+      rotateRegisterGarbageThenCreateNewDB(RotationType.FORCE_ROTATE);
+      try {
+        writeImpl(obj);
+      } catch (SQLException ex) {
+        LOG.error("Failed to write multiple times. Giving up.");
+        // We rethrow this exception so that framework can take appropriate action.
+        throw e;
+      }
+    }
+  }
+
+  abstract <T> void writeImpl(T obj) throws IllegalAccessException, InvocationTargetException, NoSuchMethodException, SQLException;
+
   private synchronized void rotateRegisterGarbageThenCreateNewDB(RotationType type) throws IOException, SQLException {
     Path rotatedFile = null;
     long currTime = System.currentTimeMillis();

src/main/java/com/amazon/opendistro/elasticsearch/performanceanalyzer/rca/persistence/RefColumn.java

@@ -0,0 +1,11 @@
+package com.amazon.opendistro.elasticsearch.performanceanalyzer.rca.persistence;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+@Retention(RetentionPolicy.RUNTIME)
+@Target(ElementType.FIELD)
+public @interface RefColumn {
+}