Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cross-build for Scala.js #515

Merged
merged 50 commits into from
Aug 10, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
ea80181
Cross-build for Scala.js
armanbilge Jul 29, 2021
d614a86
Fixup remaining linking issues
armanbilge Jul 29, 2021
b69fc2f
Poke ci
armanbilge Jul 29, 2021
dc7dc14
Set eviction rule for Scala.js
armanbilge Aug 2, 2021
de451e2
Disable scoverage for Scala.js
armanbilge Aug 2, 2021
f80fde7
JS hashing bug fix
armanbilge Aug 2, 2021
e6f2ef6
Enable Scala.js bundler for tests
armanbilge Aug 2, 2021
12f37ac
Fix cast exception
armanbilge Aug 2, 2021
e40f79b
Implement ByteVector <-> ArrayBuffer conversion
armanbilge Aug 2, 2021
b3def34
Use Uint8Array
armanbilge Aug 2, 2021
07d9060
Use yarn for test npm modules
armanbilge Aug 2, 2021
8a9ce4a
Move #231 to jvm-only
armanbilge Aug 3, 2021
0d6a531
Add US locale
armanbilge Aug 3, 2021
82da921
Add scala-java-time to eviction rules
armanbilge Aug 3, 2021
6a30a4b
Disable parallel execution, for a moment
armanbilge Aug 3, 2021
b99cf2a
Run tests alphabetically
armanbilge Aug 3, 2021
39641e0
Move parallel test setting
armanbilge Aug 3, 2021
1c765af
Revert "Run tests alphabetically"
armanbilge Aug 3, 2021
23196a1
Disable several test suites
armanbilge Aug 3, 2021
baaff4e
Disable several test suites
armanbilge Aug 3, 2021
fab9edf
Bump munit
armanbilge Aug 3, 2021
a45e7fa
Remove framework declaration
armanbilge Aug 3, 2021
44c3685
Add junit-interface to deps
armanbilge Aug 3, 2021
d496282
Undo all the damage
armanbilge Aug 3, 2021
ee6a718
Dependency adjustments
armanbilge Aug 3, 2021
92962b9
Disable failing test
armanbilge Aug 3, 2021
5eb60ce
Tryint to isolate
armanbilge Aug 3, 2021
c2e9d41
Isolating...
armanbilge Aug 3, 2021
0eb4153
Isolating...
armanbilge Aug 3, 2021
3bbb286
Isolating...
armanbilge Aug 3, 2021
f7029e2
Isolating...
armanbilge Aug 3, 2021
333c343
Isolating...
armanbilge Aug 3, 2021
6091dd0
Isolating...
armanbilge Aug 3, 2021
92024dc
Undo all the damage
armanbilge Aug 3, 2021
272e121
One more isolation
armanbilge Aug 3, 2021
23ec9f1
And now it shouldn't crash
armanbilge Aug 3, 2021
ee6dae2
Try fixing scram
armanbilge Aug 3, 2021
291841f
Re-enable SslTest
armanbilge Aug 3, 2021
f42b0c9
Bump fs2
armanbilge Aug 4, 2021
bb8a60c
Bump Scalajs version
armanbilge Aug 4, 2021
0882e77
Try out new locale dependency
armanbilge Aug 5, 2021
7f3c2c1
Bump fs2
armanbilge Aug 7, 2021
14b21f5
Bump fs2
armanbilge Aug 7, 2021
89535fd
Bump fs2, fix failing test
armanbilge Aug 7, 2021
9bc52c9
Fix 2.12 and 3.0 compile
armanbilge Aug 7, 2021
68a4bf0
Add scala-java-time dep
armanbilge Aug 7, 2021
1983d99
Fix eviction warning
armanbilge Aug 7, 2021
694f0d5
Correctly enable/disable logging
armanbilge Aug 9, 2021
ddb57ba
Restore sourcepos
armanbilge Aug 10, 2021
cdc08ed
Address review comments
armanbilge Aug 10, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 53 additions & 41 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ lazy val `scala-2.13` = "2.13.6"
lazy val `scala-3.0` = "3.0.1"

// This is used in a couple places
lazy val fs2Version = "3.0.5"
lazy val fs2Version = "3.0-121-8013a95"
lazy val natchezVersion = "0.1.5"

// We do `evictionCheck` in CI
Expand Down Expand Up @@ -65,9 +65,9 @@ lazy val commonSettings = Seq(
Compile / unmanagedSourceDirectories ++= {
val sourceDir = (Compile / sourceDirectory).value
CrossVersion.partialVersion(scalaVersion.value) match {
case Some((3, _)) => Seq(sourceDir / "scala-2.13+")
case Some((3, _)) => Seq(sourceDir / "scala-2.13+", file(sourceDir.getPath.replaceFirst("jvm", "shared").replaceFirst("js", "shared")) / "scala-2.13+")
case Some((2, 12)) => Seq()
case Some((2, _)) => Seq(sourceDir / "scala-2.13+")
case Some((2, _)) => Seq(sourceDir / "scala-2.13+", file(sourceDir.getPath.replaceFirst("jvm", "shared").replaceFirst("js", "shared")) / "scala-2.13+")
case _ => Seq()
}
},
Expand All @@ -89,58 +89,69 @@ lazy val skunk = project
.enablePlugins(AutomateHeaderPlugin)
.settings(commonSettings)
.settings(publish / skip := true)
.dependsOn(core, tests, circe, refined, example)
.aggregate(core, tests, circe, refined, example)
.dependsOn(core.jvm, core.js, tests.jvm, tests.js, circe.jvm, circe.js, refined.jvm, refined.js, example)
.aggregate(core.jvm, core.js, tests.jvm, tests.js, circe.jvm, circe.js, refined.jvm, refined.js, example)

lazy val core = project
lazy val core = crossProject(JVMPlatform, JSPlatform)
.crossType(CrossType.Full)
.in(file("modules/core"))
.enablePlugins(AutomateHeaderPlugin)
.jsEnablePlugins(ScalaJSBundlerPlugin)
.settings(commonSettings)
.settings(
name := "skunk-core",
description := "Tagless, non-blocking data access library for Postgres.",
resolvers += "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots",
libraryDependencies ++= Seq(
"org.typelevel" %% "cats-core" % "2.6.1",
"org.typelevel" %% "cats-effect" % "3.1.1",
"co.fs2" %% "fs2-core" % fs2Version,
"co.fs2" %% "fs2-io" % fs2Version,
"org.scodec" %% "scodec-core" % (if (scalaVersion.value.startsWith("3.")) "2.0.0" else "1.11.8"),
"org.scodec" %% "scodec-cats" % "1.1.0",
"org.tpolecat" %% "natchez-core" % natchezVersion,
"org.tpolecat" %% "sourcepos" % "1.0.0",
"com.ongres.scram" % "client" % "2.1",
"org.scala-lang.modules" %% "scala-collection-compat" % "2.4.4",
"org.typelevel" %%% "cats-core" % "2.6.1",
"org.typelevel" %%% "cats-effect" % "3.1.1",
"co.fs2" %%% "fs2-core" % fs2Version,
"co.fs2" %%% "fs2-io" % fs2Version,
"org.scodec" %%% "scodec-core" % (if (scalaVersion.value.startsWith("3.")) "2.0.0" else "1.11.8"),
"org.scodec" %%% "scodec-cats" % "1.1.0",
"org.tpolecat" %%% "natchez-core" % natchezVersion,
"org.scala-lang.modules" %%% "scala-collection-compat" % "2.4.4",
) ++ Seq(
"com.beachape" %% "enumeratum" % "1.6.1",
"com.beachape" %%% "enumeratum" % "1.6.1",
).filterNot(_ => scalaVersion.value.startsWith("3."))
).jvmSettings(
libraryDependencies ++= Seq(
"org.tpolecat" %%% "sourcepos" % "1.0.0",
"com.ongres.scram" % "client" % "2.1"
)
).jsSettings(
Compile / npmDependencies += "saslprep" -> "1.0.3",
useYarn := true
)

lazy val refined = project
lazy val refined = crossProject(JVMPlatform, JSPlatform)
.crossType(CrossType.Pure)
.in(file("modules/refined"))
.dependsOn(core)
.enablePlugins(AutomateHeaderPlugin)
.settings(commonSettings)
.settings(
libraryDependencies ++= Seq(
"eu.timepit" %% "refined" % "0.9.27",
"eu.timepit" %%% "refined" % "0.9.27",
)
)

lazy val circe = project
lazy val circe = crossProject(JVMPlatform, JSPlatform)
.crossType(CrossType.Pure)
.in(file("modules/circe"))
.dependsOn(core)
.enablePlugins(AutomateHeaderPlugin)
.settings(commonSettings)
.settings(
name := "skunk-circe",
libraryDependencies ++= Seq(
"io.circe" %% "circe-core" % "0.14.1",
"io.circe" %% "circe-parser" % "0.14.1"
"io.circe" %%% "circe-core" % "0.14.1",
"io.circe" %%% "circe-parser" % "0.14.1"
)
)

lazy val tests = project
lazy val tests = crossProject(JVMPlatform, JSPlatform)
.crossType(CrossType.Full)
.in(file("modules/tests"))
.dependsOn(core, circe)
.enablePlugins(AutomateHeaderPlugin)
Expand All @@ -149,39 +160,40 @@ lazy val tests = project
publish / skip := true,
scalacOptions -= "-Xfatal-warnings",
libraryDependencies ++= Seq(
"org.typelevel" %% "scalacheck-effect-munit" % "1.0.2",
"org.typelevel" %% "munit-cats-effect-3" % "1.0.5",
"org.typelevel" %% "cats-free" % "2.6.1",
"org.typelevel" %% "cats-laws" % "2.6.1",
"org.typelevel" %% "discipline-munit" % "1.0.9",
"org.typelevel" %%% "scalacheck-effect-munit" % "1.0.2",
"org.typelevel" %%% "munit-cats-effect-3" % "1.0.5",
"org.typelevel" %%% "cats-free" % "2.6.1",
"org.typelevel" %%% "cats-laws" % "2.6.1",
"org.typelevel" %%% "discipline-munit" % "1.0.9",
) ++ Seq(
"io.chrisdavenport" %% "cats-time" % "0.3.4",
"io.chrisdavenport" %%% "cats-time" % "0.3.4",
).filterNot(_ => scalaVersion.value.startsWith("3.")),
testFrameworks += new TestFramework("munit.Framework")
)
.jsSettings(Test / scalaJSLinkerConfig ~= (_.withModuleKind(ModuleKind.CommonJSModule)))

lazy val example = project
.in(file("modules/example"))
.dependsOn(core)
.dependsOn(core.jvm)
.enablePlugins(AutomateHeaderPlugin)
.settings(commonSettings)
.settings(
publish / skip := true,
libraryDependencies ++= Seq(
"org.tpolecat" %% "natchez-honeycomb" % natchezVersion,
"org.tpolecat" %% "natchez-jaeger" % natchezVersion,
"org.tpolecat" %%% "natchez-honeycomb" % natchezVersion,
"org.tpolecat" %%% "natchez-jaeger" % natchezVersion,
)
// ) ++ Seq(
// "org.http4s" %% "http4s-dsl" % "0.21.22",
// "org.http4s" %% "http4s-blaze-server" % "0.21.22",
// "org.http4s" %% "http4s-circe" % "0.21.22",
// "io.circe" %% "circe-generic" % "0.13.0",
// "org.http4s" %%% "http4s-dsl" % "0.21.22",
// "org.http4s" %%% "http4s-blaze-server" % "0.21.22",
// "org.http4s" %%% "http4s-circe" % "0.21.22",
// "io.circe" %%% "circe-generic" % "0.13.0",
// ).filterNot(_ => scalaVersion.value.startsWith("3."))
)

lazy val docs = project
.in(file("modules/docs"))
.dependsOn(core)
.dependsOn(core.jvm)
.enablePlugins(AutomateHeaderPlugin)
.enablePlugins(ParadoxPlugin)
.enablePlugins(ParadoxSitePlugin)
Expand All @@ -196,11 +208,11 @@ lazy val docs = project
paradoxTheme := Some(builtinParadoxTheme("generic")),
version := version.value.takeWhile(_ != '+'), // strip off the +3-f22dca22+20191110-1520-SNAPSHOT business
paradoxProperties ++= Map(
"scala-versions" -> (core / crossScalaVersions).value.map(CrossVersion.partialVersion).flatten.map(_._2).mkString("2.", "/", ""),
"scala-versions" -> (core.jvm / crossScalaVersions).value.map(CrossVersion.partialVersion).flatten.map(_._2).mkString("2.", "/", ""),
"org" -> organization.value,
"scala.binary.version" -> s"2.${CrossVersion.partialVersion(scalaVersion.value).get._2}",
"core-dep" -> s"${(core / name).value}_2.${CrossVersion.partialVersion(scalaVersion.value).get._2}",
"circe-dep" -> s"${(circe / name).value}_2.${CrossVersion.partialVersion(scalaVersion.value).get._2}",
"core-dep" -> s"${(core.jvm / name).value}_2.${CrossVersion.partialVersion(scalaVersion.value).get._2}",
"circe-dep" -> s"${(circe.jvm / name).value}_2.${CrossVersion.partialVersion(scalaVersion.value).get._2}",
"version" -> version.value,
"scaladoc.skunk.base_url" -> s"https://static.javadoc.io/org.tpolecat/skunk-core_2.12/${version.value}",
"scaladoc.fs2.io.base_url"-> s"https://static.javadoc.io/co.fs2/fs2-io_2.12/${fs2Version}",
Expand All @@ -210,6 +222,6 @@ lazy val docs = project
makeSite := makeSite.dependsOn(mdoc.toTask("")).value,
mdocExtraArguments := Seq("--no-link-hygiene"), // paradox handles this
libraryDependencies ++= Seq(
"org.tpolecat" %% "natchez-jaeger" % natchezVersion,
"org.tpolecat" %%% "natchez-jaeger" % natchezVersion,
)
)
9 changes: 9 additions & 0 deletions modules/core/js/src/main/scala/SSLPlatform.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
// Copyright (c) 2018-2021 by Rob Norris
// This software is licensed under the MIT License (MIT).
// For more information see LICENSE or https://opensource.org/licenses/MIT

package skunk

private[skunk] trait SSLCompanionPlatform {

}
tpolecat marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
// Copyright (c) 2018-2021 by Rob Norris
// This software is licensed under the MIT License (MIT).
// For more information see LICENSE or https://opensource.org/licenses/MIT

package skunk.net.message

import scala.scalajs.js
import scala.scalajs.js.typedarray._

private[message] trait PasswordMessagePlatform {

private val crypto = js.Dynamic.global.require("crypto")

// See https://www.postgresql.org/docs/9.6/protocol-flow.html#AEN113418
// and https://github.com/pgjdbc/pgjdbc/blob/master/pgjdbc/src/main/java/org/postgresql/util/MD5Digest.java
def md5(user: String, password: String, salt: Array[Byte]): PasswordMessage = {

// Hash with this thing
val md = crypto.createHash("md5")

// First round
md.update(password)
md.update(user)
var hex = BigInt(1, new Int8Array(md.digest().asInstanceOf[ArrayBuffer]).toArray).toString(16)
while (hex.length < 32)
hex = "0" + hex

// Second round
md.update(hex)
md.update(salt.toTypedArray)
hex = BigInt(1, new Int8Array(md.digest().asInstanceOf[ArrayBuffer]).toArray).toString(16)
while (hex.length < 32)
hex = "0" + hex

// Done
new PasswordMessage("md5" + hex) {}

}

}
136 changes: 136 additions & 0 deletions modules/core/js/src/main/scala/net/message/Scram.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
// Copyright (c) 2018-2021 by Rob Norris
// This software is licensed under the MIT License (MIT).
// For more information see LICENSE or https://opensource.org/licenses/MIT

package skunk.net.message

import scodec.bits.ByteVector
import scodec.codecs.utf8

import scala.scalajs.js
import scala.scalajs.js.typedarray.TypedArrayBuffer
import scala.scalajs.js.typedarray.TypedArrayBufferOps._

/**
* Partial implementation of [RFC5802](https://tools.ietf.org/html/rfc5802), as needed by PostgreSQL.
*
* That is, only features used by PostgreSQL are implemented -- e.g., channel binding is not supported and
* optional message fields omitted by PostgreSQL are not supported.
*/
private[skunk] object Scram {
val SaslMechanism = "SCRAM-SHA-256"

val NoChannelBinding = ByteVector.view("n,,".getBytes)

private implicit class StringOps(val value: String) extends AnyVal {
def bytesUtf8: ByteVector = ByteVector.view(value.getBytes(java.nio.charset.StandardCharsets.UTF_8))
}

private val normalize = js.Dynamic.global.require("saslprep").asInstanceOf[String => String]

def clientFirstBareWithRandomNonce: ByteVector = {
val nonce = {
val arr = new Array[Byte](32)
java.security.SecureRandom.getInstanceStrong().nextBytes(arr)
ByteVector.view(arr).toBase64
}
clientFirstBareWithNonce(nonce)
}

def clientFirstBareWithNonce(nonce: String): ByteVector =
s"n=,r=${nonce}".bytesUtf8

case class ServerFirst(nonce: String, salt: ByteVector, iterations: Int)
object ServerFirst {
private val Pattern = """r=([\x21-\x2B\x2D-\x7E]+),s=([A-Za-z0-9+/]+={0,2}),i=(\d+)""".r

def decode(bytes: ByteVector): Option[ServerFirst] =
utf8.decodeValue(bytes.bits).toOption.flatMap {
case Pattern(r, s, i) =>
Some(ServerFirst(r, ByteVector.fromValidBase64(s), i.toInt))
case _ =>
None
}
}

case class ClientProof(value: String)

case class ClientFinalWithoutProof(channelBinding: String, nonce: String) {
override def toString: String = s"c=$channelBinding,r=$nonce"
def encode: ByteVector = toString.bytesUtf8
def encodeWithProof(proof: ClientProof): ByteVector = (toString ++ s",p=${proof.value}").bytesUtf8
}

case class Verifier(value: ByteVector)

case class ServerFinal(verifier: Verifier)
object ServerFinal {
private val Pattern = """v=([A-Za-z0-9+/]+={0,2})""".r
def decode(bytes: ByteVector): Option[ServerFinal] =
utf8.decodeValue(bytes.bits).toOption.flatMap {
case Pattern(v) =>
Some(ServerFinal(Verifier(ByteVector.fromValidBase64(v))))
case _ =>
None
}
}

private val crypto = js.Dynamic.global.require("crypto")

private def bufferToByteVector(buffer: js.Dynamic): ByteVector =
ByteVector.view(TypedArrayBuffer.wrap(
buffer.asInstanceOf[js.typedarray.ArrayBuffer],
buffer.byteOffset.asInstanceOf[Int],
buffer.byteLength.asInstanceOf[Int]
))

private def HMAC(key: ByteVector, str: ByteVector): ByteVector = {
val mac = crypto.createHmac("sha256", key.toByteBuffer.arrayBuffer())
mac.update(str.toByteBuffer.arrayBuffer())
bufferToByteVector(mac.digest())
}

private def H(input: ByteVector): ByteVector = input.digest("SHA-256")

private def Hi(str: String, salt: ByteVector, iterations: Int): ByteVector = {
// TODO It is unfortunate that we have to use a sync API here when an async is available
// To make the change here will require running an F[_]: Async up the hiearchy
val salted = crypto.pbkdf2Sync(str, salt.toByteBuffer.arrayBuffer(), iterations, 8 * 32, "sha256")
bufferToByteVector(salted)
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate? PBKDF-2 should be completely CPU bound

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Node.js offers an async version of this method with a callback. I assume they have a good reason for doing so.

My best guess would be that you are indeed right that it is CPU-bound, but it's CPU intensive enough that it would detrimentally block a single-threaded Node.js app.


private def makeClientProofAndServerSignature(password: String, salt: ByteVector, iterations: Int, clientFirstMessageBare: ByteVector, serverFirstMessage: ByteVector, clientFinalMessageWithoutProof: ByteVector): (ClientProof, Verifier) = {
val saltedPassword = Hi(normalize(password), salt, iterations)
val clientKey = HMAC(saltedPassword, "Client Key".bytesUtf8)
val storedKey = H(clientKey)
val comma = ",".bytesUtf8
val authMessage = clientFirstMessageBare ++ comma ++ serverFirstMessage ++ comma ++ clientFinalMessageWithoutProof
val clientSignature = HMAC(storedKey, authMessage)
val proof = clientKey xor clientSignature
val serverKey = HMAC(saltedPassword, "Server Key".bytesUtf8)
val serverSignature = HMAC(serverKey, authMessage)
(ClientProof(proof.toBase64), Verifier(serverSignature))
}

def saslInitialResponse(channelBinding: ByteVector, clientFirstBare: ByteVector): SASLInitialResponse =
SASLInitialResponse(SaslMechanism, channelBinding ++ clientFirstBare)

def saslChallenge(
password: String,
channelBinding: ByteVector,
serverFirst: ServerFirst,
clientFirstBare: ByteVector,
serverFirstBytes: ByteVector
): (SASLResponse, Verifier) = {
val clientFinalMessageWithoutProof = ClientFinalWithoutProof(channelBinding.toBase64, serverFirst.nonce)
val (clientProof, expectedVerifier) =
makeClientProofAndServerSignature(
password,
serverFirst.salt,
serverFirst.iterations,
clientFirstBare,
serverFirstBytes,
clientFinalMessageWithoutProof.encode)
(SASLResponse(clientFinalMessageWithoutProof.encodeWithProof(clientProof)), expectedVerifier)
}
}
Loading