Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Play-json-jsoniter #13

Merged
merged 29 commits into from
Jan 10, 2020
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
b9dad57
Add play-json-jsoniter subproject
haghard Dec 29, 2019
168a2fb
Add play-json-jsoniter subproject
haghard Dec 29, 2019
c45eb41
Clean up things
haghard Dec 29, 2019
74e56bc
Clean up deps
haghard Dec 29, 2019
d6609cd
1. Add playJsonJsoniter subproject to the root project aggregated set…
haghard Dec 30, 2019
4381d62
Fix a typo
haghard Jan 8, 2020
2a3781b
Improve imports
haghard Jan 8, 2020
09abc0f
Improve TestDataGenerators
haghard Jan 8, 2020
2269831
Since currently there is no build for json-scala-values-generator for…
haghard Jan 8, 2020
aeff208
Clean up
haghard Jan 8, 2020
0083a75
Following standard scala guidelines
haghard Jan 8, 2020
d3a2ec7
Bring back test/scala folder
haghard Jan 8, 2020
fd4b4a7
Switching on java LinkedHashMap for storing pairs
haghard Jan 8, 2020
9bf5a6a
Minor cleanup
haghard Jan 8, 2020
527e5f1
Switched back on mutable.ArrayBuffer
haghard Jan 8, 2020
0c83c71
sbt 1.3.6
haghard Jan 9, 2020
d873091
Fix comments
haghard Jan 9, 2020
7ab45d7
Code review: a)Switch back on java.util.LinkedHashMap b) No more .toV…
haghard Jan 9, 2020
db359fc
Added missing things for previous commit
haghard Jan 9, 2020
518a546
Replace Vector with Array
haghard Jan 9, 2020
ac942f3
PlayJsonJsoniter uses config params to parse numbers form PlayJson li…
haghard Jan 9, 2020
d924ee6
Add more tests
haghard Jan 9, 2020
6bb95fe
Cleanup
haghard Jan 9, 2020
1d3b146
More tests
haghard Jan 9, 2020
1bd477d
Fix typos
haghard Jan 9, 2020
0565e03
Minor changes
haghard Jan 9, 2020
b51444b
Local scoped JsonParserSettings
haghard Jan 9, 2020
ccdb349
Minor changes
haghard Jan 10, 2020
46773e3
Deserialize method returns Try
haghard Jan 10, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

1. play-json-tools - Set of implicit Play-JSON `Format` helper classes. Example in [FlatFormatSpec](play-json-tools/src/test/scala/com/evolutiongaming/util/FlatFormatSpec.scala)
1. play-json-generic - provides Format derivation for enum like adt's (sealed trait/case objects'). Examples in [EnumerationDerivalSpec](play-json-generic/src/test/scala/com/evolutiongaming/util/generic/EnumerationDerivalSpec.scala)
1. play-json-jsoniter - provides the fastest way to convert an instance of `play.api.libs.json.JsValue` to byte array and read it back.

## Setup

Expand Down
20 changes: 18 additions & 2 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ val commonSettings = Seq(
crossScalaVersions := Seq("2.13.1", "2.12.10"),
)


lazy val root = project
.in(file("."))
.disablePlugins(MimaPlugin)
Expand All @@ -28,6 +27,7 @@ lazy val root = project
.aggregate(
playJsonTools,
playJsonGeneric,
playJsonJsoniter
)


Expand Down Expand Up @@ -55,4 +55,20 @@ lazy val playJsonTools = project
playJson,
nel,
scalaTest % Test,
).map(excludeLog4j)))
).map(excludeLog4j)))

//++ 2.12.10 or ++ 2.13.1
lazy val playJsonJsoniter = project
.in(file("play-json-jsoniter"))
.settings(commonSettings)
.settings(Seq(
moduleName := "play-json-jsoniter",
name := "play-json-jsoniter",
libraryDependencies ++= {
CrossVersion.partialVersion(scalaVersion.value) match {
case Some((2, v)) if v >= 13 =>
Seq(playJson, nel, jsoniter, scalaTest % Test, jsonGenerator % Test).map(excludeLog4j)
case _ =>
Seq(playJson, nel, jsoniter, collectionCompact, scalaTest % Test).map(excludeLog4j)
}
}))
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
package com.evolutiongaming.jsonitertool

import com.github.plokhotnyuk.jsoniter_scala.core._
import play.api.libs.json._

import scala.collection.IndexedSeq

object PlayJsonJsoniter {

implicit val jsValueCodec: JsonValueCodec[JsValue] =
new JsonValueCodec[JsValue] {

private val playJsonSettings = JsonParserSettings.settings.bigDecimalParseSettings
haghard marked this conversation as resolved.
Show resolved Hide resolved

/**
* The implementation was borrowed from: https://github.com/plokhotnyuk/jsoniter-scala/blob/e80d51019b39efacff9e695de97dce0c23ae9135/jsoniter-scala-benchmark/src/main/scala/io/circe/CirceJsoniter.scala
* and adapted to meet PlayJson criteria.
*/
override def decodeValue(in: JsonReader, default: JsValue): JsValue = {
val b = in.nextToken()
if (b == 'n') in.readNullOrError(default, "expected `null` value")
else if (b == '"') {
in.rollbackToken()
JsString(in.readString(null))
} else if (b == 'f' || b == 't') {
in.rollbackToken()
if (in.readBoolean()) JsTrue else JsFalse
} else if ((b >= '0' && b <= '9') || b == '-') {
in.rollbackToken()
//In order to stay consistent with PlayJson which can parse 310 characters length numbers
val dLimit = playJsonSettings.digitsLimit + 1
JsNumber(in.readBigDecimal(null, playJsonSettings.mathContext, playJsonSettings.scaleLimit, dLimit))
} else if (b == '[') {
val array: IndexedSeq[JsValue] =
if (in.isNextToken(']')) new Array[JsValue](0)
else {
in.rollbackToken()
var i = 0
var arr = new Array[JsValue](4)
do {
if (i == arr.length) arr = java.util.Arrays.copyOf(arr, i << 1)
arr(i) = decodeValue(in, default)
i += 1
} while (in.isNextToken(','))

if (in.isCurrentToken(']'))
if (i == arr.length) arr else java.util.Arrays.copyOf(arr, i)
else in.arrayEndOrCommaError()
}
JsArray(array)
} else if (b == '{') {
/*
* Because of DoS vulnerability in Scala 2.12 HashMap https://github.com/scala/bug/issues/11203
* we use a Java LinkedHashMap because it better handles hash code collisions for Comparable keys.
*/
val kvs =
if (in.isNextToken('}')) new java.util.LinkedHashMap[String, JsValue]()
else {
val underlying = new java.util.LinkedHashMap[String, JsValue]()
in.rollbackToken()
do {
underlying.put(in.readKeyAsString(), decodeValue(in, default))
} while (in.isNextToken(','))

if (!in.isCurrentToken('}'))
in.objectEndOrCommaError()

underlying
}
import scala.jdk.CollectionConverters._
JsObject(kvs.asScala)
} else in.decodeError("expected JSON value")
}

override def encodeValue(jsValue: JsValue, out: JsonWriter): Unit =
jsValue match {
case JsBoolean(b) =>
out.writeVal(b)
case JsString(value) =>
out.writeVal(value)
case JsNumber(value) =>
out.writeVal(value)
case JsArray(items) =>
out.writeArrayStart()
items.foreach(encodeValue(_, out))
out.writeArrayEnd()
case JsObject(kvs) =>
out.writeObjectStart()
kvs.foreach {
haghard marked this conversation as resolved.
Show resolved Hide resolved
case (k, v) =>
out.writeKey(k)
encodeValue(v, out)
}
out.writeObjectEnd()
case JsNull =>
out.writeNull()
}

override val nullValue: JsValue = JsNull
}

def serialize(payload: JsValue): Array[Byte] =
writeToArray(payload)

def deserialize(bytes: Array[Byte]): JsValue =
readFromArray[JsValue](bytes)
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
package com.evolutiongaming.jsonitertool

import play.api.libs.json.Json
import com.evolutiongaming.jsonitertool.TestDataGenerators.{Address, User}

trait PlayJsonImplicits {

implicit val e = Json.reads[Address]
implicit val f = Json.writes[Address]

implicit val g = Json.reads[User]
implicit val h = Json.writes[User]

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
package com.evolutiongaming.jsonitertool

import com.evolutiongaming.jsonitertool.TestDataGenerators.{User, genUser}
import org.scalacheck.Prop.forAll
import org.scalacheck.{Arbitrary, Gen, Test}
import play.api.libs.json.Json

//sbt playJsonJsoniter/test:"runMain com.evolutiongaming.jsonitertool.PlayJsonWithJsoniterBackendSpec"
object PlayJsonWithJsoniterBackendSpec extends org.scalacheck.Properties("PlayJsonWithJsoniterBackend") {
val Size = 200

override def overrideParameters(p: Test.Parameters): Test.Parameters =
p.withMinSuccessfulTests(Size)

implicit def generator: Arbitrary[User] = Arbitrary(genUser)

property("Write using PlayJson -> Read using Jsoniter") = forAll { user: User =>
val jsValue = Json.toJson(user)
val bts = Json.toBytes(jsValue)
val actJsValue = PlayJsonJsoniter.deserialize(bts)
user == Json.fromJson[User](actJsValue).get
}

property("Write using PlayJson -> Read using Jsoniter. Batch") = forAll(
Gen.containerOfN[Vector, User](Size, genUser),
) { batch: Vector[User] =>
val bools = batch.map { user =>
val jsValue = Json.toJson(user)
val bts = Json.toBytes(jsValue)
val actJsValue = PlayJsonJsoniter.deserialize(bts)
user == Json.fromJson[User](actJsValue).get
}

bools.find(_ == false).isEmpty
}

property("Write using Jsoniter -> Read using Jsoniter. Batch") = forAll(
Gen.containerOfN[Vector, User](Size, genUser),
) { batch: Vector[User] =>
val bools = batch.map { user =>
val jsValue = Json.toJson(user)
val bts = PlayJsonJsoniter.serialize(jsValue)
val actJsValue = PlayJsonJsoniter.deserialize(bts)
user == Json.fromJson[User](actJsValue).get
}

bools.find(_ == false).isEmpty
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
package com.evolutiongaming.jsonitertool

import org.scalacheck.Prop.forAll
import org.scalacheck.{Arbitrary, Gen, Test}
import play.api.libs.json.Json
import valuegen.RandomJsArrayGen

//sbt playJsonJsoniter/test:"runMain com.evolutiongaming.jsonitertool.RandomJsonArraysSpec"
object RandomJsonArraysSpec extends org.scalacheck.Properties("RandomJsonSpec") {

val Size = 5000

//produces any imaginable Json array
def randomArrayGen: Gen[value.JsArray] = RandomJsArrayGen()

implicit def generator: Arbitrary[value.JsArray] = Arbitrary(randomArrayGen)

override def overrideParameters(p: Test.Parameters): Test.Parameters =
p.withMinSuccessfulTests(Size)

property("Random json arrays") = forAll { array: value.JsArray =>
val json = array.toString
val jsValue = Json.parse(json)
val bts = PlayJsonJsoniter.serialize(jsValue)
val actJsValue = PlayJsonJsoniter.deserialize(bts)
jsValue == actJsValue
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
package com.evolutiongaming.jsonitertool

import org.scalacheck.Prop.forAll
import org.scalacheck.{Arbitrary, Gen, Test}
import play.api.libs.json.Json
import valuegen.RandomJsObjGen

//sbt playJsonJsoniter/test:"runMain com.evolutiongaming.jsonitertool.RandomJsonObjectsSpec"
object RandomJsonObjectsSpec extends org.scalacheck.Properties("RandomJsonObjectsSpec") {

val Size = 5000

//produces any imaginable Json object
def randomObjGen: Gen[value.JsObj] = RandomJsObjGen()

implicit def generator: Arbitrary[value.JsObj] = Arbitrary(randomObjGen)

override def overrideParameters(p: Test.Parameters): Test.Parameters =
p.withMinSuccessfulTests(Size)

property("Random json objects") = forAll { obj: value.JsObj =>
val json = obj.toString
val jsValue = Json.parse(json)
val bts = PlayJsonJsoniter.serialize(jsValue)
val actJsValue = PlayJsonJsoniter.deserialize(bts)
jsValue == actJsValue
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
package com.evolutiongaming.jsonitertool

import play.api.libs.json.Json
import valuegen.Preamble._
import valuegen.{JsArrayGen, JsObjGen}
import value.JsObj
import org.scalacheck.{Arbitrary, Gen}

object TestDataGenerators extends PlayJsonImplicits {

val ALPHABET: Vector[String] = "abcdefghijklmnopqrstuvwzyz".split("").toVector

def str(len: Int): Gen[String] = Gen.listOfN(len, Gen.choose(0, ALPHABET.size - 1).map(ALPHABET(_))).map(_.mkString(""))

def flagGen: Gen[Boolean] = Arbitrary.arbitrary[Boolean]

def nameGen: Gen[String] = Gen.choose(5, 20).flatMap(str)

def birthDateGen: Gen[String] = Gen.choose(5, 20).flatMap(str)

def latitudeGen: Gen[Double] = Arbitrary.arbitrary[Double]

def longitudeGen: Gen[Double] = Arbitrary.arbitrary[Double]

def longGen: Gen[Long] = Arbitrary.arbitrary[Long]

def ints: Gen[Int] = Arbitrary.arbitrary[Int]

def emailGen: Gen[String] = Gen.choose(5, 20).flatMap(str)

def countryGen: Gen[String] = Gen.choose(5, 20).flatMap(str)

def friendGen(depth: Int): Gen[JsObj] = JsObjGen(
"name" -> nameGen,
"email" -> emailGen,
"flag" -> flagGen,
"l" -> longGen,
"from" -> JsObjGen("country" -> countryGen, "doubles" -> JsArrayGen(latitudeGen, longitudeGen)),
"metrics" -> JsArrayGen(ints, ints),
"friends" -> (if (depth > 0) JsArrayGen(friendGen(depth - 1)) else JsArrayGen())
)

def objGen: Gen[JsObj] = JsObjGen(
"name" -> nameGen,
"email" -> emailGen,
"flag" -> flagGen,
"l" -> longGen,
"from" -> JsObjGen("country" -> countryGen, "doubles" -> JsArrayGen(latitudeGen, longitudeGen)),
"metrics" -> JsArrayGen(ints, ints, ints, ints, ints),
"friends" -> JsArrayGen(friendGen(depth = 50))
)

def genUser: Gen[User] = objGen
.map(json => Json.fromJson[User](Json.parse(json.toString)).get)

case class Address(country: String, doubles: List[Double])

case class User(name: String, email: String, flag: Boolean, from: Address, l: Long, metrics: List[Int],
friends: List[User])
}
Loading