Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial Tableau Reading Support #10733

Merged
merged 36 commits into from
Aug 7, 2024
Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
e9fa255
Add Tableau
jdunkerley Jul 22, 2024
d8136f4
Building up the bits and bobs.
jdunkerley Jul 22, 2024
143c2bd
Pull hyperd.
jdunkerley Jul 24, 2024
5d5bf36
First test of Tableau.
jdunkerley Jul 29, 2024
297ca24
Fix integration with HyperAPI
hubertp Jul 29, 2024
d20ae0b
nit
hubertp Jul 29, 2024
b2dd092
First functional bits.
jdunkerley Jul 31, 2024
0c55cba
Typo.
jdunkerley Jul 31, 2024
6ac10c1
Read structure.
jdunkerley Jul 31, 2024
68c6d1c
Read structure fixed.
jdunkerley Jul 31, 2024
379f9a6
Infrastructure for reading a Tableau Table.
jdunkerley Aug 1, 2024
aa2576f
Move inner classes out and split out a bit.
jdunkerley Aug 1, 2024
3286ba4
Fix Nothing handling.
jdunkerley Aug 1, 2024
7803fc8
Split Enso files.
jdunkerley Aug 1, 2024
1133948
Fix dropdowns.
jdunkerley Aug 1, 2024
eab5a77
Add Changelog.
jdunkerley Aug 1, 2024
2adffe1
Add File_Format for Tableau.
jdunkerley Aug 1, 2024
8139009
Legal review
jdunkerley Aug 1, 2024
cd4c0a3
Scala fmt
jdunkerley Aug 1, 2024
a1f52e7
Dependency resolution is a prerequisite for compilation
hubertp Aug 1, 2024
a0c5357
Add Hyper Copyright
jdunkerley Aug 2, 2024
30f3c47
Useless human...
jdunkerley Aug 2, 2024
6594c50
add more info to sbt
hubertp Aug 2, 2024
35828c3
Block comments.
jdunkerley Aug 2, 2024
3708118
Java Format.
jdunkerley Aug 2, 2024
4c4991a
Add the start of unit tests for Tableau.
jdunkerley Aug 2, 2024
00546cc
Handle process failure more gracefully.
jdunkerley Aug 2, 2024
f5164a6
Java fmt and sample hyper files.
jdunkerley Aug 2, 2024
3bedc2d
sbt now has up-to-date list of unmanaged deps
hubertp Aug 2, 2024
a782089
Read test part 1.
jdunkerley Aug 2, 2024
dfb4b7c
Fix failing test.
jdunkerley Aug 2, 2024
fa6eaf5
Fix failing test.
jdunkerley Aug 2, 2024
6bb721c
Java format.
jdunkerley Aug 2, 2024
fe90653
Ensure `unmanagedJars` fetches jars
hubertp Aug 2, 2024
872384f
Update build.sbt
hubertp Aug 5, 2024
2d1d7c0
Merge branch 'develop' into wip/jd/tableau
radeusgd Aug 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@
comparisons.][10614]
- [Relative paths are now resolved relative to the project location, also in the
Cloud.][10660]
- [Support for reading from Tableau Hyper files.][10733]

[10614]: https://github.com/enso-org/enso/pull/10614
[10660]: https://github.com/enso-org/enso/pull/10660
[10733]: https://github.com/enso-org/enso/pull/10733

# Enso 2023.3

Expand Down
102 changes: 101 additions & 1 deletion build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import src.main.scala.licenses.{
import JPMSPlugin.autoImport._

import java.io.File
import java.nio.file.Files
import java.nio.file.Paths

// ============================================================================
Expand Down Expand Up @@ -167,7 +168,11 @@ GatherLicenses.distributions := Seq(
"Snowflake",
Distribution.sbtProjects(`std-snowflake`)
),
makeStdLibDistribution("Microsoft", Distribution.sbtProjects(`std-microsoft`))
makeStdLibDistribution(
"Microsoft",
Distribution.sbtProjects(`std-microsoft`)
),
makeStdLibDistribution("Tableau", Distribution.sbtProjects(`std-tableau`))
)

GatherLicenses.licenseConfigurations := Set("compile")
Expand Down Expand Up @@ -351,6 +356,7 @@ lazy val enso = (project in file("."))
`std-aws`,
`std-snowflake`,
`std-microsoft`,
`std-tableau`,
`http-test-helper`,
`enso-test-java-helpers`,
`exploratory-benchmark-java-helpers`,
Expand Down Expand Up @@ -526,6 +532,7 @@ val poiOoxmlVersion = "5.2.3"
val redshiftVersion = "2.1.0.15"
val univocityParsersVersion = "2.9.1"
val xmlbeansVersion = "5.1.1"
val tableauVersion = "0.0.19691.r2d7e5bc8"

// === ZIO ====================================================================

Expand Down Expand Up @@ -570,6 +577,7 @@ val apacheArrowVersion = "14.0.1"
val snowflakeJDBCVersion = "3.15.0"
val mssqlserverJDBCVersion = "12.6.2.jre11"
val jsoniterVersion = "2.28.5"
val jnaVersion = "5.14.0"

// ============================================================================
// === Utility methods =====================================================
Expand Down Expand Up @@ -1949,6 +1957,7 @@ lazy val runtime = (project in file("engine/runtime"))
.dependsOn(`std-aws` / Compile / packageBin)
.dependsOn(`std-snowflake` / Compile / packageBin)
.dependsOn(`std-microsoft` / Compile / packageBin)
.dependsOn(`std-tableau` / Compile / packageBin)
.value
)
.dependsOn(`common-polyglot-core-utils`)
Expand Down Expand Up @@ -3269,6 +3278,8 @@ val `std-snowflake-polyglot-root` =
stdLibComponentRoot("Snowflake") / "polyglot" / "java"
val `std-microsoft-polyglot-root` =
stdLibComponentRoot("Microsoft") / "polyglot" / "java"
val `std-tableau-polyglot-root` =
stdLibComponentRoot("Tableau") / "polyglot" / "java"

lazy val `std-base` = project
.in(file("std-bits") / "base")
Expand Down Expand Up @@ -3606,6 +3617,92 @@ lazy val `std-microsoft` = project
.dependsOn(`std-table` % "provided")
.dependsOn(`std-database` % "provided")

lazy val `std-tableau` = project
.in(file("std-bits") / "tableau")
.settings(
frgaalJavaCompilerSetting,
autoScalaLibrary := false,
unmanagedExternalZip := {
val platform = if (Platform.isWindows) {
"windows"
} else if (Platform.isMacOS) {
"macos"
} else if (Platform.isLinux) {
"linux"
}
val arch = if (Platform.isArm64) {
"arm64"
} else {
"x86_64"
}
new URI(
s"https://downloads.tableau.com/tssoftware//tableauhyperapi-java-$platform-$arch-release-main.$tableauVersion.zip"
hubertp marked this conversation as resolved.
Show resolved Hide resolved
).toURL()
},
fetchZipToUnmanaged := {
val unmanagedDirectory = (Compile / unmanagedBase).value
val logger = state.value.log
if (IO.listFiles(unmanagedDirectory).size < 2) { // Heuristic, should have at least hyperapi jar and os-specific one.
logger.log(
Level.Info,
"std-tableau's unmanaged dependencies are not up-to-date. fetching..."
)
unmanagedDirectory.mkdirs()
val unmanagedPath = unmanagedDirectory.toPath
IO.withTemporaryDirectory(
tmp => {
val files = IO.unzipURL(
unmanagedExternalZip.value,
tmp,
f =>
f.endsWith(".jar") && !f.contains("gradle") && !f.contains(
"javadoc"
) && !f.contains("jna")
)
files.map { f =>
IO.move(f, unmanagedPath.resolve(f.getName).toFile)
Attributed.blank(unmanagedPath.resolve(f.getName).toFile)
}.toSeq
},
keepDirectory = false
)
} else {
Seq[Attributed[File]]()
}
},
Compile / unmanagedClasspath := Def.task {
val additionalFiles: Seq[Attributed[File]] = fetchZipToUnmanaged.value
val result = (Compile / unmanagedClasspath).value
result ++ additionalFiles
}.value,
Compile / packageBin / artifactPath :=
`std-tableau-polyglot-root` / "std-tableau.jar",
libraryDependencies ++= Seq(
"org.netbeans.api" % "org-openide-util-lookup" % netbeansApiVersion % "provided",
"net.java.dev.jna" % "jna-platform" % jnaVersion
),
Compile / packageBin := Def.task {
val result = (Compile / packageBin).value
val _ = StdBits
.copyDependencies(
`std-tableau-polyglot-root`,
Seq("std-tableau.jar"),
ignoreScalaLibrary = true
)
.value
result
}.value
)
.dependsOn(`std-base` % "provided")
.dependsOn(`std-table` % "provided")

lazy val fetchZipToUnmanaged =
taskKey[Seq[Attributed[File]]](
"Download zip file from an `unmanagedExternalZip` url and unpack jars to unmanaged libs directory"
)
lazy val unmanagedExternalZip =
settingKey[URL]("URL to zip file with dependencies")

/* Note [Native Image Workaround for GraalVM 20.2]
* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* In GraalVM 20.2 the Native Image build of even simple Scala programs has
Expand Down Expand Up @@ -3826,6 +3923,8 @@ pkgStdLibInternal := Def.inputTask {
(`std-snowflake` / Compile / packageBin).value
case "Microsoft" =>
(`std-microsoft` / Compile / packageBin).value
case "Tableau" =>
(`std-tableau` / Compile / packageBin).value
case _ if buildAllCmd =>
(`std-base` / Compile / packageBin).value
(`enso-test-java-helpers` / Compile / packageBin).value
Expand All @@ -3838,6 +3937,7 @@ pkgStdLibInternal := Def.inputTask {
(`std-aws` / Compile / packageBin).value
(`std-snowflake` / Compile / packageBin).value
(`std-microsoft` / Compile / packageBin).value
(`std-tableau` / Compile / packageBin).value
case _ =>
}
val libs =
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ type JS_Object
JS_Object.Value object_node (make_field_names object_node)

## PRIVATE
Creates a Jackon_Object from a list of key-value pairs.
Creates a Jackson_Object from a list of key-value pairs.
Keys must be `Text` values.
Values will be recursively converted to JSON serializable as needed.
from_pairs : Vector -> JS_Object
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Copyright (c) 2016 - 2023, Salesforce, Inc. and its licensors. All rights reserved.

Protected by U.S. Patents and Trademarks as noted at http://www.tableau.com/ip; Patents pending.
12 changes: 12 additions & 0 deletions distribution/lib/Standard/Tableau/0.0.0-dev/THIRD-PARTY/NOTICE
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
Enso
Copyright 2020 - 2024 New Byte Order sp. z o. o.

'jna', licensed under the Apache-2.0, is distributed with the Tableau.
The license file can be found at `licenses/APACHE2.0`.
Copyright notices related to this dependency can be found in the directory `net.java.dev.jna.jna-5.14.0`.


'jna-platform', licensed under the Apache-2.0, is distributed with the Tableau.
The license file can be found at `licenses/APACHE2.0`.
Copyright notices related to this dependency can be found in the directory `net.java.dev.jna.jna-platform-5.14.0`.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot to say but I think ideally we should add an entry about HyperAPI here. Can be added by modifying notice-header.

But current is probably OK too since we do have the COPYRIGHT-HYPERAPI file, so feel free to ignore.

Loading
Loading