Autovivification

[ilyash-b]

In previous episodes:

• [FR] Autovivification #549
• https://gist.github.com/ilyash-b/fd87050fd55e5d0de65a19cbee4b3ceb

[rdje]

Ilya,

To sum-up.

deep_set() will autovivify and set the provided value at path.
deep_get won't autovivify (stealth mode).

To have a read/check access with autovivification you're ok to add a new function whose name is left to be agreed upon.

You proposed

• deep_make(path)
• deep_ensure(path)

I would suggest

• deep_path(path), constructs path and assigns null, much like deep_set(path, null)
• deep_peek(path), like ninja mode but one that leave some clues behind
• deep_poke(path), like a gentle touch

Where path is made of a sequence of at least one of

• Int
• Str
• Arr of Int and Str

What do you think ?

[ilyash-b]

deep_path(path), constructs path and assigns null, much like deep_set(path, null)

Sounds too close to have both.

Having doubts about the necessity of deep_make/deep_ensure/deep_path etc. Sounds like it's the same as deep_set(x, path, []) and deep_set(x, path, {})?

[ilyash-b]

Where path is made of a sequence of at least one of

Currently, I'm proposing that path would be a single parameter to the deep_... methods. Yes, an array that is Repeat(AnyOf(Int, Str), 1..null) in our pattern matching parlance.

Edit: single parameter - to avoid interference with possible future methods in the deep_... multimethods.

[rdje]

Ok, deal :) if that's what it takes to have an autovivification API in NGS. 👍

[ilyash-b]

Reminder, mostly to myself: open issue - handle index type mismatch: using Int index in a Hash / using Str index in an Arr

[rdje]

Hi Ilya,

I just thought about something.

In my day work I rarely use JSON files. I can see some of them being use as inputs with some of the tools I use.

I noticed, please correct me if I am wrong, but what deep_set(...) helps creates is a hierarchically, deeply nested data structure made of Arrs and Hashes which to me should correspond to what a parsed JSON looks like.

Which means, unless I am wrong, there seem to be a 1-to-1 correspondance between a JSON file and what's get created using deep_set(...).

Am I wrong ?

-Richard

[rdje]

Hi Ilya,

I just thought about something.

In my day work I rarely use JSON files. I can see some of them being use as inputs with some of the tools I use.

I noticed, please correct me if I am wrong, but what deep_set(...) helps creates is a hierarchically, deeply nested data structure made of Arrs and Hashes which to me should correspond to what a parsed JSON looks like.

Which means, unless I am wrong, there seem to be a 1-to-1 correspondance between a JSON file and what's get created using deep_set(...).

Am I wrong ?

-Richard

We could even rename deep_set(...) by json_set(...) to make it more clear, no ?

[rdje]

Hi Ilya,
I just thought about something.
In my day work I rarely use JSON files. I can see some of them being use as inputs with some of the tools I use.
I noticed, please correct me if I am wrong, but what deep_set(...) helps creates is a hierarchically, deeply nested data structure made of Arrs and Hashes which to me should correspond to what a parsed JSON looks like.
Which means, unless I am wrong, there seem to be a 1-to-1 correspondance between a JSON file and what's get created using deep_set(...).
Am I wrong ?
-Richard

We could even rename deep_set(...) by json_set(...) to make it more clear, no ?

I wasn't mistaken, the deep data structure created by deep_set(...) is effectively an in-memory JSON or a deserialized JSON

• json format

Conversely, you might want to add a json_serialize() to eventually dump the deep data structure directly into a JSON file.

What do you think ?

[ilyash-b]

is effectively an in-memory JSON or a deserialized JSON

I look at it from another perspective. It's a data structure. In our case it does not contain anything that can't be serialized to JSON. Side note: use F JsonData(x:MyType) ... to convert your custom type to basic data structure (Arr/Hash/...) which in turn can be converted to JSON.

json_set - nope. We work with data structure in memory, and not doing anything related to JSON serialization. The same data structure can also be potentially serialized to YAML and other formats... and we don't call our functions yaml_set, etc.

More at https://ilya-sher.org/2016/09/05/json-vs-data-structure/

json_serialize

NGS already has:

• https://ngs-lang.org/doc/latest/generated/encode_json.html
• https://ngs-lang.org/doc/latest/generated/decode_json.html
• https://ngs-lang.org/doc/latest/generated/store.html
• https://ngs-lang.org/doc/latest/generated/fetch.html

and run-and-parse syntax which automatically decodes JSON

echo(``curl -s https://api.myip.com``.cc)

[rdje]

is effectively an in-memory JSON or a deserialized JSON

I look at it from another perspective. It's a data structure. In our case it does not contain anything that can't be serialized to JSON. Side note: use F JsonData(x:MyType) ... to convert your custom type to basic data structure (Arr/Hash/...) which in turn can be converted to JSON.

json_set - nope. We work with data structure in memory, and not doing anything related to JSON serialization. The same data structure can also be potentially serialized to YAML and other formats... and we don't call our functions yaml_set, etc.

More at https://ilya-sher.org/2016/09/05/json-vs-data-structure/

json_serialize

NGS already has:

• https://ngs-lang.org/doc/latest/generated/encode_json.html
• https://ngs-lang.org/doc/latest/generated/decode_json.html
• https://ngs-lang.org/doc/latest/generated/store.html
• https://ngs-lang.org/doc/latest/generated/fetch.html

and run-and-parse syntax which automatically decodes JSON

echo(``curl -s https://api.myip.com``.cc)

Ok I see you have everything covered then ! :-)

[rdje]

One other remark.

If we stick on solely using Str values to build the paths passed to deep_set(...) we end up with a hierarchical data structure made of only Hashes as intermediate nodes.

These deep Hash-only based data structures may be considered as a generalization of a normal single level Hash.

Then it becomes interesting to create a new function call, say, recurse that will take a DeepHash or simply a Hash as 1st argument and a 2-argument closure as its 2nd argument (Hence my earlier question about the trailing closure syntax).

recurse would then walk the deep Hash until reaching a value whose type is not a Hash, at which point it would call the closure with the path of that non-Hash value and the value itself.

A lot of useful use cases, scenarios could be derived from that.

Also we may go even one level further and replace the condition that triggers the invocation of our 1st closure, say, closure1, by another closure, closure2, that would take the current path of the level we are at as one argument and the corresponding value (Int, Arr, Hash, Any) as its 2nd argument.

If the returned value of closure2 is true, recurse would call closure1 with both the path and value provided to closure2 for the condition.

Anyway, whether or not we use the fact that the current value during the parkour is not a Hash or use a closure for the condition, closure or closure1 respectively, would get called with both the path and its associated value.

I think what you talked earlier about

• do {closure}
• with {closure-for-condition} do {closure}

could be useful with the recurse function.

So I think there is something interesting here to apply with {...} and/or do {...}

Like

• dh.recurse do {closure}
• dh.recurse with {closure-for-condition} do {closure}
• dh.recurse ({closure-for-condition}) do {closure}
  • or even dh.recurse (closure-for-condition) do {closure}

What do you think ?

[ilyash-b]

There is partial design and experimental implementation of something that is pretty close to what you describe.

See

• https://github.com/ngs-lang/ngs/wiki/Deep-Data-Structures-Manipulation-Design
• https://github.com/ngs-lang/ngs/blob/7cd41de3f4d8f8426376495c4c0ab19637e1fce8/lib/autoload/globals/tr.ngs , especially TEST lines towards the end

A lot of useful use cases, scenarios could be derived from that.

That's why it's in the language :)

dh.recurse do {closure}

won't work, must be dh.recurse() do {closure}

[rdje]

There is partial design and experimental implementation of something that is pretty close to what you describe.

See

• https://github.com/ngs-lang/ngs/wiki/Deep-Data-Structures-Manipulation-Design
• https://github.com/ngs-lang/ngs/blob/7cd41de3f4d8f8426376495c4c0ab19637e1fce8/lib/autoload/globals/tr.ngs , especially TEST lines towards the end

A lot of useful use cases, scenarios could be derived from that.

That's why it's in the language :)

dh.recurse do {closure}

won't work, must be dh.recurse() do {closure}

The descriptions at the links you provided are for deep data structure with intermediate nodes of any kinds, that is they are not necessarily composed of only Hashes at every step along the path.

Since a Hash-only deep data structure may be seen as a generalized Hash, some specific functions or behaviors may be derived from that.

At least, that is what I did in the past when actively working on such generaized Hashes with Perl5.

Anyway, I think you understand what I want to raise here.

I wanted also to point out that your with {...} and do {...} constructs could be put, fairly quickly, to good use if you consider Hash-based deep data structure as a special and interesting kind.

[rdje]

There is partial design and experimental implementation of something that is pretty close to what you describe.
See

• https://github.com/ngs-lang/ngs/wiki/Deep-Data-Structures-Manipulation-Design
• https://github.com/ngs-lang/ngs/blob/7cd41de3f4d8f8426376495c4c0ab19637e1fce8/lib/autoload/globals/tr.ngs , especially TEST lines towards the end

A lot of useful use cases, scenarios could be derived from that.

That's why it's in the language :)

dh.recurse do {closure}

won't work, must be dh.recurse() do {closure}

The descriptions at the links you provided are for deep data structure with intermediate nodes of any kinds, that is they are not necessarily composed of only Hashes at every step along the path.

Since a Hash-only deep data structure may be seen as a generalized Hash, some specific functions or behaviors may be derived from that.

At least, that is what I did in the past when actively working on such generaized Hashes with Perl5.

Anyway, I think you understand what I want to raise here.

I wanted also to point out that your with {...} and do {...} constructs could be put, fairly quickly, to good use if you consider Hash-based deep data structure as a special and interesting kind.

tr.ngs focuses on transformations, so that's already a pretty specfic task.

The algorithm that walks the structure could be reused as a basis for recurse.
The only goal of recurse would be to walk the structure and based on the outcome of some condition (expression or with {closure}) would call do {closure}.

The do {closure} could be used for transformation, that is acts as a filter --datain->[closure]--data out-> as your tr() or for reporting purposes or anything else.

The actual result of this walk through the deep data structure won't necessarily be a new data structure as with .tr but could be anything, up to do {closure}.

Bear in mind that my initial idea is to have recurse only walk on Hash-only deep data structures.

Normally it should be possible to rewrite the transformation part of tr.ngs as do {closure}'s called by a generalized version of recurse coupled with with {closure}'s as in

• dh.recurse with {condition-closure} do {closure}, with dh being a mix Arr/Hash deep data structure.

[ilyash-b]

tr.ngs focuses on transformations, so that's already a pretty specfic task.

It enables transformations, you are free to ignore the result and have side effects while walking

The descriptions at the links you provided are for deep data structure with intermediate nodes of any kinds, that is they are not necessarily composed of only Hashes at every step along the path.

I don't see how that precludes any interesting functionality that would be available if the whole data structure was just nested Hashes.

Bear in mind that my initial idea is to have recurse only walk on Hash-only deep data structures.

I am failing to see a benefit of such limitation

It is hard for me to processes the proposal. Would you like to create short summary of the whole API? Data types + methods?

[ilyash-b]

It is hard for me to processes the proposal. Would you like to create short summary of the whole API? Data types + methods?

( Let's leave out the do and with syntax for now for simplicity and consistency )

[rdje]

The idea of having a specific behaviour for a deep data structure solely based on Hash as intermediate nodes is that it simplifies the condition to call the closure attached to the recurse function.

Again, I see a Hash-based deep data structure as a generalisation of the 1-level Hash built-in objects.

The idea is to be able to walk the Hash-based deep data structure and trigger calls to the closure without being forced to provide any explicit call condition.

The default condition allowing recurse to call its attached closure would be as follows

• call the closure if the value of the node located at the current path is not of type Hash

Whenever that default condition is met for a particular node then recurse would call the closure with its path and value as arguments.

There is really nothing more to it.

There was a time when this specific use case was pivotal to the tools I wrote in Perl5.

Again, no matter how you want or plan to implement the walk into these deep data structures, it would help to have a default condition based on which to call the attached closure.

What I mean but default condition is a condition that doesn't need to be explicitly mentioned during the call to recurse.

[ilyash-b]

Let's discuss possible implementation - https://gist.github.com/ilyash-b/909461aee302de7f3286d1470863119f