Declarative DSL #2
Labels
Comments
|
I've been thinking about cfg mgmt in golang, and this DSL is where puppet & chef really shine. One of their most useful accidents (not present in ansible, terraform) is the ability to create resource-providers entirely in the DSL out of other resources. I'm of the opinion that this lack is causing a lot of reuse pain in the Ansible Galaxy. I'd love to help spike a plugin/DSL syntax using lua like the https://github.com/mozilla-services/heka project is doing. |
|
On Mon, Feb 1, 2016 at 1:49 AM, Joseph Anthony Pasquale Holsten <
https://github.com/Shopify/go-lua and examples: More discussion to come! |
|
Puppet and chef both have bad DSL's in my opinion. A less constricting way to do this is write a DSL that allows you to shell out (such as make) and just focuses on the 'resolving' graphs, this has many benefits. |
|
On Mon, Feb 1, 2016 at 3:17 AM, Nathan [email protected] wrote:
Could you elaborate on this comment please? I really agree about the DSL Thanks! |
I think this was in reference to the |
|
On Mon, Feb 1, 2016 at 8:47 AM, Erik Dalén [email protected] wrote:
If that is the correct interpretation, then it is absolutely a design |
|
Also, I realized that I have mentioned this in discussion, but maybe it's not clear in the blog post, but the current "yaml" definition files are merely placeholders for testing the engine before there is a proper DSL. It is absolutely temporary, and I can't wait to remove all of that parsing code. |
Yep, this.
<3 |
|
TL;DR: external process resource-providers require a trade-off between implementation complexity and smart dependency things @nwmcsween I'd love to do everything by having external tools make the changes. But most unix tools aren't designed to act idempotently. More so, a single tool often doen't do everything required. For example, a mere Ansible's external-plugin model gives the separation between the execution-coordinator and the resource-provider implementation. But it runs each resource a single time, which internally does a separate check-phase and apply-phase. Compare this to puppet, &c which do a complete check-phase in all resources before doing an apply-phase. I don't have a concrete example right now, but this causes the dependency system in ansible to be very weak. Terraform's external plugin model uses full-duplex communication between the coordinator and the resource provider. This allows a complete check-phase across all resources separate from the apply-phase. It also means the provider doesn't need to be run a separate check-process from the apply-process, even lets you have a single process for all resources using the same provider. The pain here is that you don't get to write plugins as standard unix tools naively reading from And I don't even want to think about parallel execution implications for resource providers. But that sounds especially unpleasant with ansible-style plugins. |
|
This is exactly the same as make, the onus is on the developer to make sure On Tue, Feb 2, 2016, 12:08 PM Joseph Anthony Pasquale Holsten <
|
|
I'm a long time user of Puppet DSL and I've come to really hate the constraints it imposes. It is improving, but everything has to be reengineered (we finally have 'for' loops!). I prefer the idea of leveraging an existing language and providing an interface for declaring resources. Basically, like Chef does with Ruby. Consider the proof that most Puppet modules implement Ruby code to augment the shortcomings of the DSL anyway. There's so much to start building in terms of language internals, but also writing a parser, testing framework, etc. Would it be worth it? I'd like to see a design where the user is writing Go/other code to declare resources, dependencies, etc. while taking advantage of Go's conditionals, operators, testing framework, quality tools, etc. But using an API interface which eases/enforces idempotency, extensibility, etc. Admittedly it creates opportunity for users to start breaking things with an unconstrained API... You'd also have to include the Go compiler in your toolset. Go is also not the most declarative DSL like language... Method chaining/LINQy stuff would be ideal IMHO. Ignoring broken syntax, something like: That's my thoughts though :) |
|
@cavaliercoder Good comments, here are some thoughts to help continue the discussion...
Cheers! |
|
LISP could be a candidate. It's easy to implement, it's easy to parse, it's supported out of the box by any editor. It's easy to make it look like a custom DSL while still being extensible. It's easy to keep it functional. The only downside is that some people don't like parentheses position. For example, if I take on of your example: (resources
(exec
((name "exec1")
(cmd "echo hello from exec1")
(timeout 0)
(watchcmd "sleep 10s")
(state 'present))
((name "exec2")
(cmd "echo hello from exec2")
(watchcmd "sleep 10s")
(state 'present))))
(edges
((name "e1")
from (exec "exec1")
to (exec "exec2")))Some parentheses could be removed for example: (exec (("exec1"
timeout 0
state 'present)))So, it looks like the YAML. No need to learn anything. If you want to use a loop, it's easy (but you have to learn the backquote): (resources
(exec
`(loop for i upto 10 collect
(list (concat "exec" (str i))
cmd (concat "echo hello " (str i))
timeout 0
state 'present))))If you a user want to introduce a new type of resource, they can write a function. If they want to extend the DSL, they can write a macro. |
|
@vincentbernat Thanks for your input! One probable requirement of the DSL will include the ability to make it reactive, and also to have the compiler/lexer written in golang. Do you know if either of these already exist in lisp? Having said that, I did consider lisp, but I'm not sure if it would be a good choice for mainstream users. Readability is an important property. Having said that, there's nothing to stop someone from writing a lisp frontend. |
|
For a more readable variant, you can look at Clojure syntax which have lists What you don't want is to transform YAML into a programming language (lke Ansible). For Go, unfortunately, I know there are many LISP implemented in Go but I have no real experience to point to a good one. What's great with LISP is that you can take a minimal implementation and works from here with macros. |
|
On Thu, Jul 7, 2016 at 6:05 AM, Vincent Bernat [email protected]
|
|
Unfortunately, I don't have time at the moment. Writing random comments in GitHub issue is easier than proposing patches. :) BTW, as an example of LISP as DSL, there is Riemann (http://riemann.io/, but it's written in Clojure, so, more natural choice). |
|
A full blown language would kill the idea of build system like DAG, it On Thu, Jul 7, 2016, 5:21 AM Vincent Bernat [email protected]
|
|
Hi, Just found your project, looks nice! :) I'm also working on a similar project, where I've just recently implemented a DSL based on Lua. You might want to look at it in case you decide to go with Lua. The project I'm working on is located at https://github.com/dnaeon/gru |
|
@dnaeon Funny enough someone send me a pointer to your project just the other day, although I haven't had a chance to dig into the code yet, sorry. I did look into using Lua, but for some design reasons unfortunately I don't think it would be a good choice, mainly because of a lack of safety constraints available in the language that we would require. The fact that you mention using it as a DSL, makes me think it's possibly a safe subset of the language that you're using? In any case, I'd love to chat about it more, and maybe you want to join forces and help out in mgmt? Can I persuade you to have a look at my intro article or video? https://ttboj.wordpress.com/2016/01/18/next-generation-configuration-mgmt/ I'm hanging out in the #mgmtconfig IRC channel on Freenode which you're more than welcome to join. Cheers and thanks for the message, |
|
Hey James, I'll check out the video and article, thanks for the links! This week I'm traveling most of the time, but once I get back from my vacation I'll ping you in IRC. Best regards, |
|
@dnaeon Sounds good. - Enjoy your vacation! Cheers, |
|
Jumping in here and I hope I get the context right. If you're going to invent a DSL for mgmt please consider publishing it as an open standard. There are so many OS config management tools and each has its own language, leading to lock-in. An open standard for future tools to use would be a great thing. |
|
@neilhwatson I personally have no objections to this, there's nothing proprietary about what we're building. The first step though is to get the code written so we have a minimally viable project. Can you help with some patches ? |
|
Hashicorp Config Language? I'm really interested in being involved in next-gen config management. As much as I love Chef, and others, they are all pretty terrible ecosystems. |
|
@shawncatz I'd be interested in what you find lacking in the chef or puppet ecosystems. I've noticed a much higher amount of modular code and code reuse in those ecosystems than in ansible, terraform and salt. I think one of the reasons is first class resource types. It requires a complete redesign and rewrite to convert an implementation from ansible task into ansible module, or terraform module into terraform provider. This makes it much harder to get a broad community of resource implementations, and makes it more expensive to prototype new approaches, and creates a hierarchy of contributors: "real" resource devs vs mere users. I say this as a contributor to HCL and a committer on terraform. I <3 it, but its architecture has serious drawbacks for long term growth. |
|
@shawncatz How can we help you get more involved? (Feel free to discuss this on IRC with us in #mgmtconfig on Freenode) @josephholsten and @shawncatz I looked briefly at HCL, and unfortunately I think it lacks some of the features we need, since it's much closer to a yaml than a full DSL. Thanks for the idea though! |
|
If we were looking for a fancy JSON/YAML/TOML alternative, https://github.com/vstakhov/libucl was the inspiration for HCL, and much nicer IM(NS)HO. I'm mostly in favor of Lua as a golang encore nation language, but if we want a LISP, guile is quite nice. |
|
As was suggested, I figured I'd like to the HCL lib here: https://github.com/hashicorp/hcl |
|
Hey @purpleidea, Just a thought about HCL - I've used HCL in the previous releases of Gru, and overall was pretty happy with it. The syntax is fine and configuration expressed in HCL looks nice. At some point you would want to introduce variable interpolations, add custom functions in the HCL configuration, etc. and would most likely end up with HIL as I did already. At the end I gave up on HCL and HIL, mainly because of the reasons listed in this issue. With that said, I'm not saying that HCL/HIL are bad, they just didn't cut it for me and for what I wanted to achieve. Anyways, thought I'd share this with you, in case you decice to go that way. P.S. Seems we can't get a hold on IRC because of the time difference :) I'll be joining in there, maybe I'll get a hold of you next time :) |
purpleidea
pushed a commit
that referenced
this issue
Jun 29, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Jun 29, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Jun 29, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Jun 30, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Jul 3, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Jul 15, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Jul 19, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Jul 27, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Jul 30, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 8, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 9, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 9, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 13, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 20, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 21, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 22, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 22, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 27, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 28, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 28, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 28, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 29, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 30, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Aug 31, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Sep 4, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Sep 4, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Sep 19, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Sep 19, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Sep 19, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
purpleidea
pushed a commit
that referenced
this issue
Sep 19, 2023
This is the 1st commit message:
lang: Split FuncValue into FuncValue and SimpleFn
Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.
Here is why this is a mistake:
The output of a function like
$f = fn(x) {
Shell(Sprintf("seq %d", x))
}
varies over time, while a single Value does not. Thus, code like
Map($f, list(1, 2))
would first produce the value list("1", "1"), but then it would
_not_ update to list("1", "2") when "seq 2" produces its second
line. That's because with the mistaken design, when Map receives a
new FuncValue or a new input list of N elements, Map calls the
function from Value to Value N times and produces a single output
list of N elements.
Here is why the corrected design is better:
Here's what happens with this new design when Map receives a new
FuncValue or a new input list of N elements.
First, Map constructs N item-input nodes, each of which extracts a
different entry from the list. Then, Map calls the function from
Vertex to Vertex N times, once for each item-input node, and thus
obtain N item-output nodes. Finally, Map constructs an
item-collecting node which constructs a list out of all of its
inputs, and Map connects the N item-output nodes to the
item-collecting node. This item-collecting node is the output of
Map.
The Vertex to Vertex function constructs and connects its own nodes;
in this case, it constructs an Sprintf node and connects the
item-input node to it, and then constructs a Shell node and connects
the Sprintf node to it, and then returns the Shell node as the
item-output node.
The two Shell node in this sub-graph emit a first value "1", which
propagates to the item-collecting node and causes it to output a
first value list("1", "1"). Then, the second Shell node emits a
second value "2", which propagates to the item-collecting node and
causes it to output a second value list("1", "2"), as desired.
Here is how this commit brings us closer to the above plan:
Changing FuncValue throughout the codebase is a big change. One of
the difficulties is that it is not just nodes which are emitting
FuncValues, there are also many other places in the code where
FuncValue is used to hold a golang function from Value to Value.
Some of those places now need to hold a golang function from Vertex
to Vertex, but other places still need to hold a golang function
from Vertex to Vertex.
Thus, as a first step, we need to split FuncValue into two types.
This commit splits the old FuncValue into two types:
1. The new FuncValue will hold a function from Vertex to Vertex.
FuncValue is a Value.
2. A new type named "SimpleFn" will hold a function from Value to
Value. SimpleFn is not a Value.
This commit replaces occurrences of the old FuncValue with one of
those two new types, as appropriate. This commit does not yet adapt
the surrounding code to make use of the new representation; that
will be done in future commits. I have annotated the missing parts
with the following panic message in order to make it easy to find
which parts still need to be implemented. The "..." part explains
what needs to be implemented.
panic("TODO [SimpleFn]: ...");
Here's where I need help:
One part of the code which is not clear to me are the parts which
use reflection. I don't understand the purpose of that code well
enough to explain what needs to be implemented. I have annotated
those "known unknown" parts of the remaining work with the following
panic message in order to make it easy to find which parts still
need more thinking and planning:
panic("TODO [SimpleFn] [Reflect]: ...");
This is the commit message #2:
lang: Add the core func graph Txn API
This will eventually let functions change the running graph via a
transaction API.
At the moment the core Lock and Unlock primitives aren't implemented.
This is the commit message #3:
lang: A reversible wrapper around Txn
This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.
This is the commit message #4:
lang: move FuncValue to its own package
This is the commit message #5:
lang: combine lang/func/structs and ast
Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.
This is the commit message #6:
lang: CallExpr must generate a subgraph
FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.
This is the commit message #7:
lang: ExprFunc does not need to store its value
This is the commit message #8:
spelling
This is the commit message #9:
convert between SimpleFn, FuncValue, and Func
This is the commit message #10:
no need for ExprCall.argVertices
it's the exact same thing as ExprCall.Args
This is the commit message #11:
ExprFunc.Func()'s three cases
This is the commit message #12:
FunctionFunc is no longer used
This is the commit message #13:
ExprBool.MergedGraph()
This is the commit message #14:
ExprIf.MergedGraph()
This is the commit message #15:
ExprFunc.MergedGraph()
This is the commit message #16:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.
This is the commit message #17:
lang: ast, interfaces: Add MergedGraph signature and implementation
This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.
This is the commit message #18:
Use MergedGraph signature and implementation
This puts it into play, but doesn't initialize the input args at all.
This is the commit message #19:
Restore ExprBool.Graph() and Func()
This reverts commit 3ea3845.
This is the commit message #20:
Restore ExprIf.Graph()
This reverts commit a62889e.
This is the commit message #21:
restore ExprFunc::Graph() and Func()
This is the commit message #22:
fix type errors
This is the commit message #23:
ExprCall.MergedGraph()
This is the commit message #24:
a more precise type for args
This is the commit message #25:
GraphTxn
This is the commit message #26:
ReversibleTxn.AddGraph
This is the commit message #27:
ExprFunc.MergedGraph()
This is the commit message #28:
ExprVar.MergedGraph()
This is the commit message #29:
sub-graph is spelled subgraph in this codebase
This is the commit message #30:
ExprFunc.mkFunc() was unused
This is the commit message #31:
CallFunc.Stream() should add Funcs to the graph, not Exprs
This is the commit message #32:
move ConstFunc to lang.funcs
This is the commit message #33:
move conversion functions to lang.funcs.simple
I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.
This is the commit message #34:
it is the responsibility of the function engine to call Init on the nodes
This is the commit message #35:
FuncValue.Call()
This is the commit message #36:
drop MergedGraph's unused txn parameter
This is the commit message #37:
move CallFunc to funcs.simple
This is the commit message #38:
Func from channel
This is the commit message #39:
ChannelBased{Source,Sink}Func
This is the commit message #40:
MapFunc
This is the commit message #41:
remove unused CallFunc field
This is the commit message #42:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #43:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #44:
extend environment with StmtProg's local variables
This is the commit message #45:
delete unused VarFunc
This is the commit message #46:
appease govet
This is the commit message #47:
add imported variables to the environment
This is the commit message #48:
add builtins to the environment
This is the commit message #49:
accumulate all the imported variables
previously, we were only keeping the imported variables from the last
import statement.
This is the commit message #50:
XXX: wip new function engine
This is the commit message #51:
comments
This is the commit message #52:
remove Engine.{Lock,Unlock} placeholders
This is the commit message #53:
f(...) support for ExprCall
This is the commit message #54:
[REVERT ME] remove unrelated files to make VS Code happy
This is the commit message #55:
don't recreate the subgraphInput
This is the commit message #56:
CallFunc now takes a single input, the FuncValue
Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.
This is the commit message #57:
GraphTxn must take a pointer in order to modify the graph
This is the commit message #58:
GraphTxn is unused
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Consider a DSL that is similar to make with explicitly set defaults to map to fqdn's (via import/whatever).
This way targets could be handled by an external program of some sort that 'schedules' hosts (e.g host with least latency to $someplace start http server).
The text was updated successfully, but these errors were encountered: