Changing oracle backend from using varray to json array

[danielporterda]

Pull Request Checklist

• Read and understand the contribution guidelines
• Include appropriate tests
• Set a descriptive title and thorough description
• Add a reference to the issue this PR will solve, if appropriate
• Include changelog additions in one or more commit message bodies between the CHANGELOG_BEGIN and CHANGELOG_END tags
• Normal production system change, include purpose of change in description

NOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with /AzurePipelines run to
trigger the build.

This change modifies how we store several fields in oracle from using varrays to JSON arrays. It changes the database and makes concomitant changes to the DAO. No interface changes. Should be low regression risk, minimal overlap with postgres or h2 backend.

[danielporterda]

was intending just to put this up for Brian to do a first pass over

[cocreature]

was intending just to put this up for Brian to do a first pass over

You can turn it into a draft to make it clear that it’s not quite ready for the general public.

[dasormeter]

@danielporterda as discussed on slack also make sure you rerun ./fmt.sh - there is one bazel formatting and a few scalafmt thing it is moaning about

	modified:   ledger/participant-integration-api/src/main/scala/platform/store/Conversions.scala
	modified:   ledger/participant-integration-api/src/main/scala/platform/store/dao/JdbcLedgerDao.scala
	modified:   ledger/participant-integration-api/src/main/scala/platform/store/dao/events/EventsTable.scala
	modified:   ledger/participant-integration-api/src/main/scala/platform/store/dao/events/EventsTableFlatEventsRangeQueries.scala
	modified:   ledger/participant-integration-api/src/main/scala/platform/store/dao/events/SqlFunctions.scala```

[dasormeter]

hiya @danielporterda you can also cleanup the VARRAY handling code now as part of this PR

1. Remove definition of the various VARRAY types in the Oracle V1__init.sql
   

   daml/ledger/participant-integration-api/src/main/resources/db/migration/oracle/V1__Init.sql

   Lines 12 to 21 in f5e50ba

   	create type VARCHAR_ARRAY as VARRAY (32767) OF VARCHAR2(4000);
   	/
   	create type SMALLINT_ARRAY as VARRAY (32767) of SMALLINT;
   	/
   	create type BYTE_ARRAY_ARRAY as VARRAY (32767) of RAW(2000);
   	/
   	create type TIMESTAMP_ARRAY as VARRAY (32767) of TIMESTAMP;
   	/
   	create type BOOLEAN_ARRAY as VARRAY (32767) of NUMBER(1, 0);
   	/

2. Remove oracle specific import and createArray conversion code in Conversions.scala
   

   daml/ledger/participant-integration-api/src/main/scala/platform/store/Conversions.scala

   Line 27 in f5e50ba

   import oracle.jdbc.OracleConnection

daml/ledger/participant-integration-api/src/main/scala/platform/store/Conversions.scala

Lines 59 to 143 in f5e50ba

	@SuppressWarnings(Array("org.wartremover.warts.ArrayEquals"))
	abstract sealed class ArrayToStatement[T](oracleTypeName: String)
	extends ToStatement[Array[T]]
	with NotNullGuard {
	override def set(s: PreparedStatement, index: Int, v: Array[T]): Unit = {
	if (v == (null: AnyRef)) {
	s.setNull(index, JDBCType.ARRAY.getVendorTypeNumber, oracleTypeName)
	} else {
	s.setObject(
	index,
	unwrapConnection(s).createARRAY(oracleTypeName, v.asInstanceOf[Array[AnyRef]]),
	JDBCType.ARRAY.getVendorTypeNumber,
	)
	}
	}
	}
	implicit object ByteArrayArrayToStatement
	extends ArrayToStatement[Array[Byte]]("BYTE_ARRAY_ARRAY")
	implicit object TimestampArrayToStatement extends ArrayToStatement[Timestamp]("TIMESTAMP_ARRAY")
	implicit object RefPartyArrayToStatement extends ArrayToStatement[Ref.Party]("VARCHAR_ARRAY")
	implicit object CharArrayToStatement extends ArrayToStatement[String]("VARCHAR_ARRAY")
	implicit object IntegerArrayToStatement extends ArrayToStatement[Integer]("SMALLINT_ARRAY")
	implicit object BooleanArrayToStatement
	extends ArrayToStatement[java.lang.Boolean]("BOOLEAN_ARRAY")
	implicit object InstantArrayToStatement extends ToStatement[Array[Instant]] {
	override def set(s: PreparedStatement, index: Int, v: Array[Instant]): Unit = {
	s.setObject(
	index,
	unwrapConnection(s).createARRAY("TIMESTAMP_ARRAY", v.map(java.sql.Timestamp.from)),
	JDBCType.ARRAY.getVendorTypeNumber,
	)
	}
	}
	@SuppressWarnings(Array("org.wartremover.warts.ArrayEquals"))
	implicit object StringOptionArrayArrayToStatement extends ToStatement[Option[Array[String]]] {
	override def set(s: PreparedStatement, index: Int, stringOpts: Option[Array[String]]): Unit = {
	stringOpts match {
	case None => s.setNull(index, JDBCType.ARRAY.getVendorTypeNumber, "VARCHAR_ARRAY")
	case Some(arr) =>
	s.setObject(
	index,
	unwrapConnection(s)
	.createARRAY("VARCHAR_ARRAY", arr.asInstanceOf[Array[AnyRef]]),
	JDBCType.ARRAY.getVendorTypeNumber,
	)
	}
	}
	}
	object IntToSmallIntConversions {
	implicit object IntOptionArrayArrayToStatement extends ToStatement[Array[Option[Int]]] {
	override def set(s: PreparedStatement, index: Int, intOpts: Array[Option[Int]]): Unit = {
	val intOrNullsArray = intOpts.map(_.map(new Integer(_)).orNull)
	s.setObject(
	index,
	unwrapConnection(s)
	.createARRAY("SMALLINT_ARRAY", intOrNullsArray.asInstanceOf[Array[AnyRef]]),
	JDBCType.ARRAY.getVendorTypeNumber,
	)
	}
	}
	}
	private def unwrapConnection[T](s: PreparedStatement): OracleConnection = {
	s.getConnection match {
	case hikari: HikariProxyConnection =>
	hikari.unwrap(classOf[OracleConnection])
	case oracle: OracleConnection =>
	oracle
	case c: Connection =>
	sys.error(
	s"Unsupported connection type for creating Oracle integer array: ${c.getClass.getSimpleName}"
	)
	}
	}
	}

[nmarton-da]

@dasormeter note for going forward with append-only these parse to json and then render string should go to the batching phase, and the DTO for injection should have already the string so we avoid blocking ingestion threads with dataconversion

[dasormeter]

approved, with small suggestion to avoid whitespace-only changes to EventsTableFlatEventsRangeQueries.scala

participant-integration-api bazel.build cleanup for ojdbc8.jar will be left to a follow-on PR tracked under issue #9976

ensuring indices are hit for json array fields in follow-on ticket #9975 pending discovery of that as a bottleneck

[nmarton-da]

nice bunch of code removed 👍