Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Truffle Interop Types: Array & Vectors #4542

Closed
8 tasks done
wdanilo opened this issue Feb 5, 2023 · 0 comments
Closed
8 tasks done

Truffle Interop Types: Array & Vectors #4542

wdanilo opened this issue Feb 5, 2023 · 0 comments
Assignees
@hubertp @JaroslavTulach
Labels
-compiler -libs Libraries: New libraries to be implemented p-low Low priority x-new-feature Type: new feature request

Comments

@wdanilo
Copy link
Member

wdanilo commented Feb 5, 2023

This task is automatically imported from the old Task Issue Board and it was originally created by jaroslavtulach.
Original issue is here.

Truffle polyglot system allows various types to be exchanged between different languages (Enso, JavaScript, Python, but also calls to hosted Java). These types include JVM primitive types, strings, date & time values as well as arrays: https://www.graalvm.org/truffle/javadoc/com/oracle/truffle/api/interop/InteropLibrary.html

The handling of these types in Enso shall be uniform, based on the InteropLibrary and should be opaque - e.g. not piggyback on the knowledge of the origin of the value. Rather it shall use just the InteropLibrary messages and treat all the values the same. If it “quacks like a date”, it is a date…

Another principle is as late conversion as possible. Truffle language shall let the foreign value flow through as long as possible to minimize the cost of conversion, hoping the conversion may not be needed at all. On the other hand, it is necessary to make sure that the conversion of Enso non-supported types (byte, int, float) happens before they propagate to the internal AST of Enso interpreter.

From Polyglot to Enso

An example of this approach has been demonstrated in #3559 and it consists primarily of:

  • doConvertDate - specialization to invoke messages on the object
  • PolyglotProxy delegation of necessary messages
  • getPolyglotCallType check
  • EnsoDate wrapper representing the date internally in Enso ASTs

The implementation of Array support shall follow the same:

  • doConvertArray check if the foreign object supports hasArrayElements message
  • PolygotProxy delegation to all the array related messages (probably done already)
  • getPolyglotCallType check to detect hasArrayElements support
  • Array builtin wrapper that delegates all its operations to Object using InteropLibrary
  • Conversion of getArrayElement values to Enso internally supported ones

That way we can have uniform, opaque support for polyglot arrays in Enso.
Everything that “ducks like an array” will be treated as array

  • All such objects will have Enso Array methods
  • No other methods will (need to) be exposed
  • Just one Enso Array type in the whole system
  • Yet there can be multiple XYZArray extends TruffleObject floating in the interpreter

From Enso to Polyglot

There are two types in Enso that represent a (polyglot) array:

  • Array - wrapper around (potentially mutable) foreign array
  • Vector - native Enso immutable array

Both these types may want to participate in polyglot interop with Java or other languages. To simplify conversion of Enso native data structures to other languages, both of these types shall be implemented as builtin types (just like EnsoDate) and both shall_ “quack like an array”_ - e.g. support hasArrayElements and getArrayElement messages.

Implementation note: Theoretically both Enso types (Array and Vector) could be implemented by the same TruffleObject subclass - they are not going to be mutable from outside, they’d rather have the same methods and they should look the same from a perspective of other languages. The only difference is that Array represents something that can potentially mutate, but Vector guarantees that its content is going to stay the same.

From Java to Enso without Copying

If there is a Java implementation of a table which represents a huge array - can it be used as Enso Array (or Vector) without any copying? Yes, enough to implement List or directly polyglot SDK ProxyArray - that way it is going to be perceived as hasArrayElements object - e.g. “duck like an array” and will be automatically represented as Enso Array.

Test Cases

Create an array in JavaScript, Python and return it to Enso. All methods of Array shall be available instantly.

Invoke a Java method that returns Object[] and use it from Enso. All methods of Array shall be available instantly.

Invoke a Java method that calls back and passes Object[] as an argument. Use the array from Enso. All methods of Array shall be available instantly.

Create an Enso Array with Date instances. Pass the array to Java method that accepts List<LocalDate> - verify dates are properly available.

Repeat the previous Array exercise with Vector. The result shall be the same.

Invoke a Java method that creates an array with all primitive types, String, date and another array of the same content. Make sure that it all gets converted to appropriate Enso types - e.g. Array of long and Text and another Array.

Create Java method that returns List - call it from Enso to get Array. Invoke length check and verify only the size() method was called - items of the array haven’t been touched.

Repeat the previous exercise with ProxyArray. Again, no elements shall be created.

Prior Work

Originally described at google doc.

Tasks:

  • Create a builtin type for Array
  • Make sure all its (core) methods operate via InteropLibrary
  • Implement conversions of everything that ducks like an array to such type
  • Make vector a builtin as well
  • Make sure Vector and Array share the same set of methods - reported as new user story
  • Convert values obtained from an Array to proper Enso types
  • Can a List/ProxyArray value be passed to Enso without copying?
  • Run & compare benchmark results

Comments:

Hubert Plociniczak reports a new STANDUP for yesterday (2022-08-22):

Progress: Started work towards improved Vector/Array. In the meantime fought with CI to get the interim #3628 PR in. It should be finished by 2022-08-24.

Next Day: Next day I will be working on the ##183000876 task. Pick up the pending PRs (waiting on review of time PR) and continue investigation into Arrays. (Enso Bot - Aug 23, 2022)

**Hubert Plociniczak** reports a new **STANDUP** for yesterday (2022-08-23):

Progress: Slowly progressing on builtin Vector. Addressed feedback for Date_Time PR. Slightly distracted by various random CI failures. It should be finished by 2022-08-24.

Next Day: Next day I will be working on the ##183000876 task. Hopefully merge Date_Time PR tomorrow. Prepare WIP for current work on Vector so that Jaroslav can pick it up. (Enso Bot - Aug 24, 2022)

**Hubert Plociniczak** reports a new **STANDUP** for yesterday (2022-08-24):

Progress: Good progress builtin Vector. Almost all test/Test passes. Fixing a few remaining ones will be straightforward. WIP PR is up for anyone to pick up. It should be finished by 2022-08-24.

Next Day: Next day I will be working on the ##183000876 task. Either continue on Vector/Array interop or whatever is next in line by then. (Enso Bot - Aug 24, 2022)

**Jaroslav Tulach** reports a new **🔴 DELAY** for yesterday (2022-08-29):

Summary: There is 7 days delay in implementation of the Truffle Interop Types: Array & Vectors (#183000876) task.
It will cause 7 days delay for the delivery of this weekly plan.

I've just took over the issue yesterday; moreover Marcin's current work has merge priority

Delay Cause: The issue was scheduled to be addressed this week

Possible solutions: Address the issues in the PR and integrate once Marcin's work is done (Enso Bot - Aug 30, 2022)

**Jaroslav Tulach** reports a new **🔴 DELAY** for yesterday (2022-08-29):

Summary: There is 7 days delay in implementation of the Truffle Interop Types: Array & Vectors (#183000876) task.
It will cause 7 days delay for the delivery of this weekly plan.

I've just took over the issue yesterday; moreover Marcin's current work has merge priority

Delay Cause: The issue was scheduled to be addressed this week

Possible solutions: Address the issues in the PR and integrate once Marcin's work is done (Enso Bot - Aug 30, 2022)

**Jaroslav Tulach** reports a new **🔴 DELAY** for yesterday (2022-08-29):

Summary: There is 12 days delay in implementation of the Truffle Interop Types: Array & Vectors (#183000876) task.
It will cause 12 days delay for the delivery of this weekly plan.

I've just took over the issue yesterday; moreover Marcin's current work has merge priority

Delay Cause: The issue was scheduled to be addressed this week

Possible solutions: Address the issues in the PR and integrate once Marcin's work is done (Enso Bot - Aug 30, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-08-29):

Progress: Merge of improved text visualization #3665.Array and Vector work: VectorLiteralsTest: 40cccac; Debugging Number.should_equals: c426639; Char conversion: 465a269; Debugging should capture traces correctly test; Merging with develop and removal of set_at; Fighting with should correctly propagate error through map: 8c6a1cc; Fixing tables: cf6a72a; Meetings, demos, lazy visualizations It should be finished by 2022-09-19.

Next Day: Continue Array & Vector work #3667; plan lazy visualizations (Enso Bot - Aug 30, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-08-30):

Progress: Array and Vector and lazy visualizations It should be finished by 2022-09-19.

Next Day: Fix CI gates for Array & Vector work #3667 (Enso Bot - Aug 31, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-08-31):

Progress: Merged Marcin's work with Array and Vector. Bugfixed everything (often case of: aa17c2b and 7df85d8). Meetings. It should be finished by 2022-09-19.

Next Day: Fix last five(?) failures in Table_Tests. Pass CI with Array and Vector work. (Enso Bot - Sep 1, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-09-01):

Progress: All CI checks passed for Vector and Array as of #3667 (comment) - requested review. Adding tests to cover more interop (Java, JavaScript, Python) aspects. It should be finished by 2022-09-19.

Next Day: More tests and fixes for the polyglot interop of Vector and Array (Enso Bot - Sep 2, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-09-02):

Progress: Enso Array methods rewritten to work on any array-like object - dispatch modified. Trying to compare Array and Vector in a Test suite. Defining Array.==. Trying coercion in Vector. It should be finished by 2022-09-19.

Next Day: Address the current ~10 failures in CI of Array & Vector (Enso Bot - Sep 3, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-09-05):

Progress: Proposed an umbrella for drop-downs PR; Reported slowness https://www.pivotaltracker.com/n/projects/2539304/stories/183176528 - possibly only caused by Vector & Array work: 542d85f; Updated parser branch to latest develop: #3611 It should be finished by 2022-09-19.

Next Day: Vector & Array CI cleanup and parser cleanup (Enso Bot - Sep 6, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-09-06):

Progress: Updating parser in the morning: parse type with data definitions: a5f74a6 & parsing export. Vector and Array performance, trying to fix CI. It should be finished by 2022-09-19.

Next Day: Ask for review Vector & Array. (Enso Bot - Sep 7, 2022)

Another task fixed:

Check behavior of copy/wrap over List - 8336c8e
Check the lazy behavior for ProxyArray - 981829c (jaroslavtulach - Sep 7, 2022)

The first picture shows _benchmark results produced by CI_ on Sep 8, 2022. Executed on [develop branch](https://github.com/enso-org/enso/actions/runs/3007625706) and against [PR-3667 branch](https://github.com/enso-org/enso/actions/runs/3010582834). Am I just dreaming or did we manage to speed up Enso in all the benchmarks?

As the CI measurements seem suspicious I've run the same measurements on my own computer over night. There is no improvement, but also no regression in the second picture. Good from Array & Vector point of work, not really good from the CI infrastructure point of view. CCing @murbanczyk .

I've just scheduled final CI test on Sep 12-13: develop vs. wip/hubert/array-vector-interop-183000876. These two measurement confirm the manual ones - no regression, no improvement.
(jaroslavtulach - Sep 7, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-09-07):

Progress: Test and fix for Atom.getMembers: #3689; Defensively copy in Vector.from_array: f628b28; Meeting about BigInteger: https://www.pivotaltracker.com/story/show/182962982/comments/233179278; Using cached interop for Vector: f0338c5; test for lazy List: 8336c8e; test for proxy array: 981829c; using from_polyglot_array: ce6aa5c; benchmarks results look very positive: https://www.pivotaltracker.com/story/show/183000876/comments/233194239 It should be finished by 2022-09-19.

Next Day: Address review comments for Vector & Array. (Enso Bot - Sep 7, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-09-08):

Progress: Proposing solution for broken IR caches: #3692; Meeting about Enso/Java StdLib interop: https://docs.google.com/document/d/1XOnkBJjZuiAhIwjtekSa8LjHtGLDHqGwyuEanei43wg; Pair debugging Truffle source code; Found (very useful) ReflectiveLibrary: f634036#diff-d784205295fc228b108d6ca365bf8371e6fa2337bd8f2c38e7a3bdac721f3a30R14; Some engine CI gates are green for Vector & Array: ef71c20 It should be finished by 2022-09-19.

Next Day: Write benchmarks for Vector & Array. (Enso Bot - Sep 9, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-09-09):

Progress: CI fighting: https://discord.com/channels/401396655599124480/1016968486754471966/1017642222558130186; but Broken IR caches fix integrated; toDisplayString for Vector: aaf33e3; Comparative benchmark for Vector&Array and fix to make it fast; Preparing presentation for GraalVM engineers for next week's GraalVM meetup - slides 12-19 It should be finished by 2022-09-19.

Next Day: Prepare for the GraalVM meetup (investigate Espresso, native-image). (Enso Bot - Sep 10, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-09-10):

Progress: Polishing Vector & Array: #3667 - it is green now. Reporting few TODOs into pivotal for the engine.Reported oracle/graal#4916 to GraalVM team It should be finished by 2022-09-19.

Next Day: Prepare for the GraalVM meetup (investigate Espresso, native-image). (Enso Bot - Sep 11, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-09-12):

Progress: Preparing for GraalVM meetup. Testing native image - backport of bugfix on GraalVM side needed: oracle/graal#4200; providing native image config files: b4292d3; PR to simplify hello world demo in Enso: #3696; Meetings. Listing technical debts in engine: https://discord.com/channels/401396655599124480/1018889551441313894/1018891614673973278 It should be finished by 2022-09-19.

Next Day: Prepare for the GraalVM meetup (investigate Espresso). (Enso Bot - Sep 13, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-09-13):

Progress: Finally merged vector & array: #3667;
Demo of using Espresso for Java interop: https://www.pivotaltracker.com/n/projects/2539304/stories/183260380;
Discussions about Espresso interop with GraalVM team, bugreport: oracle/graal#4939;
Discussing access to Java properties of Date: https://docs.google.com/document/d/1ErTnBtX52egVietPd8O-vS_pH_yAB968BTvpQZbFrYE It should be finished by 2022-09-19.

Next Day: Traveling to Zurich (Enso Bot - Sep 14, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-09-14):

Progress: Travel to Zurich; Reading emails; Preparing patch for GraalVM for discussion: oracle/graal#4916 It should be finished by 2022-09-19.

Next Day: GraalVM meetup, day 1 (Enso Bot - Sep 15, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-09-15):

Progress: GraalVM meetup, day 1 It should be finished by 2022-09-19.

Next Day: GraalVM meetup, day 2: Presentation about Enso (Enso Bot - Sep 16, 2022)

**Jaroslav Tulach** reports a new **STANDUP** for yesterday (2022-09-16):

Progress: GraalVM meetup, Enso presentation: https://twitter.com/wojciech_danilo/status/1570719810409103361?s=20&t=R35qiDsU40vQJhHnQ3U7CA; travel back to Prague It should be finished by 2022-09-19.

Next Day: Back to coding! (Enso Bot - Sep 17, 2022)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hubertp hubertp

@JaroslavTulach JaroslavTulach

Labels
-compiler -libs Libraries: New libraries to be implemented p-low Low priority x-new-feature Type: new feature request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hubertp @wdanilo @JaroslavTulach