Add graalpy packages to the component directory #8351

Akirathan · 2023-11-21T11:15:14Z

Fixes #8294

Pull Request Description

Adds these JAR modules to the component directory inside Engine distribution:

graal-language-23.1.0
org.bouncycastle.* - these need to be added for graalpy language

Important Notes

Remove org.bouncycastle.* packages from runtime.jar fat jar.
Make sure that the ./run script preinstalls GraalPy standalone distribution before starting engine tests
- Note that using python -m venv is only possible from standalone distribution, we cannot distribute graalpython-launcher.
Make sure that installation of numpy and its polyglot execution example works.
Convert Text to TruffleString before passing to GraalPy - 8ee9a28

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

The documentation has been updated, if necessary.
Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
All code follows the
Scala,
Java,
and
Rust
style guides. In case you are using a language not listed above, follow the Rust style guide.
All code has been tested:
- Unit tests have been written where possible.
- If GUI codebase was changed, the GUI was tested when built using ./run ide build.

Akirathan · 2023-11-22T19:08:50Z

test/Tests/src/Semantic/Python_Interop_Spec.enso

+        Test.specify "should recognize Text as Python string" <|
+            py_is_str "Hello" . should_be_true
+            py_is_str 10 . should_be_false
+            py_is_str Nothing . should_be_false


These tests are failing on develop. I am not sure why the test should correctly marshall strings was passing before.

This reverts commit d288df3.

This reverts commit 64f48df.

…nstalled

# Conflicts: # test/Examples_Tests/src/Python_Examples_Spec.enso

Akirathan · 2023-11-27T18:14:07Z

graalpy correctly installs numpy in a virtual environment - https://github.com/enso-org/enso/actions/runs/7008514362/job/19064977553?pr=8351#step:10:4972

GitHub
Add graalpy packages to the component directory · enso-org/enso@d35fa37
Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

docs/polyglot/python.md

distribution/engine/THIRD-PARTY/org.graalvm.truffle.truffle-nfi-23.1.0/NOTICES

distribution/engine/THIRD-PARTY/org.graalvm.python.python-resources-23.1.0/NOTICES

tools/legal-review/engine/org.graalvm.python.python-language-23.1.0/copyright-ignore

radeusgd · 2023-11-29T12:51:17Z

distribution/engine/THIRD-PARTY/NOTICE

+'profiler-tool', licensed under the GNU General Public License, version 2, with the Classpath Exception, is distributed with the engine.
+The license file can be found at `licenses/GNU_General_Public_License__version_2__with_the_Classpath_Exception`.
+Copyright notices related to this dependency can be found in the directory `org.graalvm.tools.profiler-tool-23.1.0`.


I have no idea if the way we distribute the profiler-tool satisfies the requirements of the GNU GPL license with classpath exception.

If I understand correctly, we are distributing these files inside of our JAR, right? Or is it a separate JAR?

We need to clarify how this dependency is distributed.

Then check if the way we do this is OK with the license - we should consult someone who has more legal knowledge about this.

If I understand correctly, we are distributing these files inside of our JAR, right? Or is it a separate JAR?

It is a separate JAR. We basically take those Jars from Maven central and just put it inside the component directory in our distribution, without any modifications.

Then check if the way we do this is OK with the license - we should consult someone who has more legal knowledge about this.

We have always distributed this tool. Just not as a separate Jar, but as part of the GraalVM distribution. It is only now that the legal review tool complains because it can actually discover the license in the Jar. I doubt that there were any major license changes in the community edition of Graal.

I had been in a legal battle about GPL+CP Exception with ASF and I won. That makes me believe I am qualified...

If I understand correctly, we are distributing these files inside of our JAR, right? Or is it a separate JAR?

It is a separate JAR. We basically take those Jars from Maven central
and just put it inside the component directory in our distribution, without any modifications.

Good. Distributing GPL+CPEx JARs is certainly OK and we can even claim the whole thing is under Apache License. Radek is right that repackaging might be tricker - good that we are not doing it.

mwu-tow · 2023-11-29T14:56:59Z

It seems that notarization gets broken, removed the "ready to merge" so it can be fixed first.

somebody1234

approving the changes to signArchivesMacOs.ts

…org/enso into wip/akirathan/8294-install-graalpy

radeusgd · 2023-12-04T11:33:31Z

build/ci_utils/src/cache/goodie/graalpy.rs

+
+fn graalpy_version_from_str(version_string: &str) -> Result<Version> {
+    let line = version_string.lines().find(|line| line.contains("GraalVM CE")).context(
+        "There is a Java environment available but it is not recognizable as GraalVM one.",


Suggested change

"There is a Java environment available but it is not recognizable as GraalVM one.",

"There is a Java environment available but it is not recognizable as a GraalVM CE.",

We are checking for Graal CE specifically.

I assume if someone runs GraalVM EE, the current message There is a Java environment available but it is not recognizable as GraalVM one. would be false (GraalVM EE is still GraalVM).

radeusgd

Code and config changes look good - approving that.

I'm not sure if the inclusion of profiler-tool in the distribution is allowed by the license, as I noted above - I'm not suggesting it isn't either - I just don't have enough legal experience to be able to assess that.

Add graalpy packages to the component directory

9c80b90

Akirathan self-assigned this Nov 21, 2023

Akirathan added 2 commits November 22, 2023 18:00

Fix assembly of runtime.jar Uber jar

08f30af

Convert Text to TruffleString before passing to python

8ee9a28

Akirathan commented Nov 22, 2023

View reviewed changes

Akirathan added 11 commits November 23, 2023 12:02

Add graalpy dependencies to the legal-review

a96ecf1

Text uses UTF-16 encoding by default

7386c3c

Ignore Python_Example_Test

64f48df

graalpy launcher is included in engine distribution

d288df3

Merge branch 'develop' into wip/akirathan/8294-install-graalpy

c645630

Revert "graalpy launcher is included in engine distribution"

2c5c9a8

This reverts commit d288df3.

Revert "Ignore Python_Example_Test"

d63c0bd

This reverts commit 64f48df.

Update python example doc - graalpy standalone distribution must be i…

84945b8

…nstalled

Python_Examples_Spec.enso test uses standalone graalpy to install numpy

fd27b35

run script install graalpy before executing engine tests

70bbf64

Merge branch 'develop' into wip/akirathan/8294-install-graalpy

d35fa37

# Conflicts: # test/Examples_Tests/src/Python_Examples_Spec.enso

Akirathan marked this pull request as ready for review November 27, 2023 17:21

Akirathan requested review from 4e6, JaroslavTulach, radeusgd, jdunkerley, GregoryTravis, hubertp and mwu-tow as code owners November 27, 2023 17:21

Akirathan added the CI: No changelog needed Do not require a changelog entry for this PR. label Nov 27, 2023

fmt

8e676dc

JaroslavTulach approved these changes Nov 28, 2023

View reviewed changes

docs/polyglot/python.md Outdated Show resolved Hide resolved

Update python.md docs.

758ffcf

Akirathan added the CI: Ready to merge This PR is eligible for automatic merge label Nov 29, 2023

radeusgd reviewed Nov 29, 2023

View reviewed changes

distribution/engine/THIRD-PARTY/org.graalvm.truffle.truffle-nfi-23.1.0/NOTICES Outdated Show resolved Hide resolved

radeusgd reviewed Nov 29, 2023

View reviewed changes

distribution/engine/THIRD-PARTY/org.graalvm.python.python-resources-23.1.0/NOTICES Outdated Show resolved Hide resolved

radeusgd reviewed Nov 29, 2023

View reviewed changes

tools/legal-review/engine/org.graalvm.python.python-language-23.1.0/copyright-ignore Show resolved Hide resolved

radeusgd reviewed Nov 29, 2023

View reviewed changes

mwu-tow approved these changes Nov 29, 2023

View reviewed changes

mwu-tow removed the CI: Ready to merge This PR is eligible for automatic merge label Nov 29, 2023

sign more binaries before notarization

e6bf428

mwu-tow requested review from PabloBuchu, indiv0 and somebody1234 as code owners November 29, 2023 16:01

somebody1234 approved these changes Nov 29, 2023

View reviewed changes

mwu-tow and others added 4 commits November 29, 2023 20:39

one more set of binaries to sign

1f22461

Re-review licenses

1ed637a

Tests in Map_Spec are not dependent on the key ordering

009a54b

Merge branch 'wip/akirathan/8294-install-graalpy' of github.com:enso-…

85cdd4b

…org/enso into wip/akirathan/8294-install-graalpy

Akirathan added the CI: Ready to merge This PR is eligible for automatic merge label Dec 4, 2023

radeusgd reviewed Dec 4, 2023

View reviewed changes

radeusgd approved these changes Dec 4, 2023

View reviewed changes

mergify bot merged commit a67297a into develop Dec 4, 2023
31 checks passed

mergify bot deleted the wip/akirathan/8294-install-graalpy branch December 4, 2023 11:51

JaroslavTulach mentioned this pull request Dec 15, 2023

16% startup regression after including GraalPy #8553

Closed

Akirathan mentioned this pull request Jan 3, 2024

Remove org.bouncycastle dependency #8664

Merged

3 tasks

mwu-tow mentioned this pull request May 16, 2024

Clean build of IDE fails on macOS #9973

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add graalpy packages to the component directory #8351

Add graalpy packages to the component directory #8351

Akirathan commented Nov 21, 2023 •

edited

Loading

Akirathan Nov 22, 2023

Akirathan commented Nov 27, 2023 •

edited by unfurl-links bot

Loading

radeusgd Nov 29, 2023

Akirathan Nov 29, 2023

JaroslavTulach Dec 4, 2023

mwu-tow commented Nov 29, 2023

somebody1234 left a comment

radeusgd Dec 4, 2023

radeusgd left a comment

	"There is a Java environment available but it is not recognizable as GraalVM one.",
	"There is a Java environment available but it is not recognizable as a GraalVM CE.",

Add graalpy packages to the component directory #8351

Add graalpy packages to the component directory #8351

Conversation

Akirathan commented Nov 21, 2023 • edited Loading

Pull Request Description

Important Notes

Checklist

Akirathan Nov 22, 2023

Choose a reason for hiding this comment

Akirathan commented Nov 27, 2023 • edited by unfurl-links bot Loading

radeusgd Nov 29, 2023

Choose a reason for hiding this comment

Akirathan Nov 29, 2023

Choose a reason for hiding this comment

JaroslavTulach Dec 4, 2023

Choose a reason for hiding this comment

mwu-tow commented Nov 29, 2023

somebody1234 left a comment

Choose a reason for hiding this comment

radeusgd Dec 4, 2023

Choose a reason for hiding this comment

radeusgd left a comment

Choose a reason for hiding this comment

Akirathan commented Nov 21, 2023 •

edited

Loading

Akirathan commented Nov 27, 2023 •

edited by unfurl-links bot

Loading