Investigate usage of Python libraries

[jdunkerley]

Want to investigate how to bring in numpy into Enso.

How can we use the function of numpy on an Enso values.

Goal is to create a sample where we compute the cumulative normal using numpy.random.normal.
Should be a function we can call from Enso and get the result back as an Enso number.

Eventually we want a guide on how to use libraries for Python, Java and JavaScript as an end user.

[JaroslavTulach]

Per @wdanilo request I did an experiment. With 2023.2.1-nightly.2023.8.28 version I can run:

from Standard.Base import all

foreign python random_array s = """
    import numpy
    return numpy.random.normal(size=s)

main =
    IO.println "Running main..."
    arr = random_array 100
    IO.println "Got an array with length: "+arr.length.to_text
    IO.println "Array: "+arr.to_text
    IO.println "Done"

and I get following output:

Got an array with length: 100
Array: [-0.09386895315368417, -1.266229800602598, -0.3429757808576437, -1.6792257889695055, -0.3385088461234442, 0.11289795467528857, 0.8916941815051626, -1.0696112287321675, 0.4865905596261328, -0.6397951846022013, 1.052976140979142, 0.8119170266325583, -0.07817607610272753, -0.8193685321790681, 1.0860242485923075, -0.05368266491222449, 0.05746030033663869, -1.3051587927504449, 1.0497666399074084, -2.7464091220543576, 0.012845593568870047, -0.9137102058433068, 0.0007617801778440016, 0.11264312500310081, -0.7640099744595212, -0.4393489820309963, -1.6176242019840605, 1.0589448394846022, 0.3280937911408749, -1.8941000148768066, 0.056400734311041735, 1.4280505594854997, -1.787362975843809, -1.8445015469623112, 1.896998118008599, 1.0529673325440068, -1.9617790372341097, -0.6322390316620973, -0.15956695611442728, 0.598937784779556, -0.07088269067275893, -1.322253697878595, 1.5521054826932357, -2.0027687657303503, 1.3694001132555012, 0.9291459721947709, 1.0074137754411931, 1.9693240280438655, 0.5461347946698242, -0.012776766219722034, -2.078064794259111, -1.0534379585843447, -0.3445812486938666, -0.3824447922622398, -0.3399176978739385, -0.17486973645656176, -0.05387319604142368, 0.16671878855533676, 1.329421186913485, 0.7970311978906027, -0.014128050466247585, 0.35039963740657665, -0.02971711278314848, -2.080225982469118, 0.47440327739048255, -0.7570151553478073, 1.218095404137077, -1.5444572251954123, 0.9302455393089899, -1.066086092943335, -1.6572041846759313, 0.7112987460684057, -0.9900115808034595, -1.3049185518211748, -0.8844078867856275, 0.11063116944168372, 0.8738840337166691, 2.0049436598720423, 0.5164879135213348, -0.26953808928089257, 0.44809002446141427, -0.35087211183833145, 1.3615823883191966, -0.06437149144627778, -0.16933876739735085, 1.5115145783073007, 1.4502282685412136, 2.18696175541435, 0.2770957598736182, -1.1322850347809366, -0.44390951719098704, -0.5112946025063865, 0.5956653667432468, 1.6645352265521942, -0.16317412633142495, -0.3377822866702371, 2.855754596898088, -0.25830209251600544, 1.1032564940924392, 0.990404467603104]
Done

So yes, we can use numpy in Enso.

[JaroslavTulach]

To reproduce the above experiment, we need to create a virtual environment for Graal Python:

/tmp$ ~/bin/graalvm-community-openjdk-17.0.7+7.1/bin/graalpy -m venv /tmp/gpy
/tmp$ source /tmp/gpy/bin/activate
(gpy) /tmp$ pip install numpy
Successfully installed numpy-1.23.5

Then we need to follow instructions on using venv inside of Java programs which leads to following diff:

diff --git engine/runner/src/main/scala/org/enso/runner/ContextFactory.scala engine/runner/src/main/scala/org/enso/runner/ContextFactory.s
cala
index 0faff393e0..30c757b4f2 100644
--- engine/runner/src/main/scala/org/enso/runner/ContextFactory.scala
+++ engine/runner/src/main/scala/org/enso/runner/ContextFactory.scala
@@ -81,6 +81,7 @@ class ContextFactory {
           new DebuggerSessionManagerEndpoint(repl, peer)
         } else null
       }
+      .option("python.Executable", "/tmp/gpy/bin/graalpy")
       .option(
         RuntimeOptions.LOG_LEVEL,
         JavaLoggingLogHandler.getJavaLogLevelFor(logLevel).getName
diff --git engine/runtime-language-epb/src/main/java/org/enso/interpreter/epb/node/ForeignEvalNode.java engine/runtime-language-epb/src/ma
in/java/org/enso/interpreter/epb/node/ForeignEvalNode.java
index 3e8bef5ca7..67f4b276c5 100644
--- engine/runtime-language-epb/src/main/java/org/enso/interpreter/epb/node/ForeignEvalNode.java
+++ engine/runtime-language-epb/src/main/java/org/enso/interpreter/epb/node/ForeignEvalNode.java
@@ -147,6 +149,11 @@ public class ForeignEvalNode extends RootNode {
       Source source =
           Source.newBuilder(code.getLanguage().getTruffleId(), head + indentLines, "").build();
       EpbContext context = EpbContext.get(this);
+      var src = Source.newBuilder("python", """
+        import site
+        """, "enso.py").build();
+      var initSite = context.getEnv().parsePublic(src);
+      initSite.call();
       CallTarget ct = context.getEnv().parsePublic(source);
       ct.call();
       Object fn = context.getEnv().importSymbol("polyglot_enso_python_eval");

The patch tells Graal Python to use /tmp/gpy virtual environment with numpy installed. With these changes one can use numpy (or any other pip installable library) in Enso.

I am not sure how to make this process user friendly, however?

[JaroslavTulach]

How-to document is ready for your review!

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2023-08-28):

Progress: - Use numpy in Enso: https://github.com/enso-org/enso/blob/545f747745b961c527420947d38023b3fccd8b67/docs/polyglot/python.md

• investigating why (numpy.random.normal) . sort stackoverflows
• reported as: StackOverflowError in sort called on numpy array. #7677
• reviewing logging: Replace a custom logger with off the shelf implementation #7559 (review)
• reviewing Change layout of local library search path in order to be able to move Round_Spec.enso back to Tests #7634 (review)
• QA: [/127.0.0.1:30535] Address in use #6730 (comment)
• discussing compatibility: https://github.com/orgs/enso-org/discussions/7639#discussioncomment-6839217
• standup meeting It should be finished by 2023-08-31.

Next Day: Integrate the numpy tutorial

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2023-08-29):

Progress: - Verify installation of numpy in an integration test: a1e16b9

• discussing Enso Table -> numpy conversion
• reviews, suggestion DB: Fix atom types in dropdowns. #7670 (review)
• reviews, type benchmark: Add value_type to Column.from_vector and expected_value_type to Column.map and Column.zip #7637 (review)
• review IR: Towards shallow nesting in IR #7681 (review)
• reporting "instrumentation from Enso" issue: Allow access to caches and instrumentation from Enso #7683
• planning meeting
• instructions for profiling startup
• Talking about libraries sharing
• (demo) meetings It should be finished by 2023-08-31.

Next Day: Bugfixing & planning

[enso-bot]

Jaroslav Tulach reports a new STANDUP for yesterday (2023-08-30):

Progress: - numpy support integrated: #7678

• bookclubing: https://discord.com/channels/401396655599124480/1146122722007580743/1146339567394369557
• bug reported: Segmentation fault (presumably) when parsing Main.enso file #7691
• staying up-to-date with development of GraalVM: https://discord.com/channels/@me/948519891961544714/1146305202916442112
• discussions & meetings
• debugging AppImage & Proxy issues with a friend - bug: AppImage cannot be used expanded: tries to download GraalVM #7698
• initial PR for Vector.sort: Standardization of Vector.sort array access #7697 It should be finished by 2023-08-31.

Next Day: Bugfixing Vector.sort. Towards Any.to.

Discord

Discord - A New Way to Chat with Friends & Communities

Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

Discord

Discord - A New Way to Chat with Friends & Communities

Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.