CS2 Discussion: Output: for-in/for-of

[rattrayalex]

CoffeeScript and ES6 have very different takes on these features. CoffeeScript's version is arguably more useful, but also more complicated – and non-standard.

What should we do?

[DomVinyard]

Non-standard yes, not sure I agree that it's more complicated. I'm not convinced that there is any compelling need to change these.

[carlsmith]

Personally, I think CoffeeScript made a mess of this. I would have opted for doing this like Python does. It's less complex, and more powerful.

[kirly-af]

Can you elaborate on how you would expect it to work ?

What do you mean Python loops are "more powerful" ? ?
Furthermore, this complexity (I guess you refer to the different for statements in CS) is the result of JS bad design.

The real issue is jashkenas/coffeescript#3832.

[carlsmith]

So, in Python, you only have one kind of for-loop, and it always iterates over a sequence. You can use it on strings, lists and sets.

$ for n in [1, 2, 3]: print n
1
2
3

You can also use it on a dictionary, where it will iterate over the keys.

$ for key in {"foo": 1, "bar": 2}: print key
foo
bar

If you want to iterate over the values, just call values to get a list of them:

$ for value in {"foo": 1, "bar": 2}.values(): print value
1
2

Python also supports unpacking sequences in assignment statements. It's a lot like CoffeeScript's version, but doesn't require any brackets.

$ x, y, z = [1, 2, 3]
$ print x + y + z
6

Python combines the two ideas, and supports unpacking a sequence inside of a for-loop.

$ sequence_of_tuples = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
$ for x, y, z in sequence_of_tuples: print x + y + z
6
15
24

While this doesn't seem that helpful at first, it is very powerful once you have some functions and methods that feed into it. For example, CoffeeScript has its magical fudge for getting the index:

$ (console.log index, char) for char, index in "abc"
0 a
1 b
2 c

In Python, there's no magic involved. You just use the enumerate function that takes a sequence, and returns an enumeration of the sequence, which is essentially a list of index-item tuples.

$ print list(enumerate("abc"))
[(0, 'a'), (1, 'b'), (2, 'c')]

So we can then do this to get the same effect as CoffeeScript, without any magic:

$ for index, character in enumerate("abc"): print index, character
0 a
1 b
2 c

The main advantage of not using magic is that we can use the same feature to iterate over anything we like, not just index-item tuples.

$ for key, value in {"foo": 1, "bar": 2}.items(): print key, value
foo 1
bar 2

$ colors = [(225, 0, 60), (0, 255, 100)]
$ for r, g, b in colors: use(r, g, b)

And so on. There's lots of ways to break a structure into a sequence of pairs or triples or n-length tuples.

Python doesn't need to distinguish between for-in and for-of. A for-loop is a for-loop. It also doesn't need magic to get the index. Python just combines its concept of a basic for-loop and its concept of unpacking a sequence, and everything else flows naturally from there.

Python actually takes iteration much further, but it's all possible because they got their for-loops right, and could build on them.

[carlsmith]

Doing things Pythonically would allow us to get rid of for-of loops altogether, and then just create methods or functions for turning whatever we have into whatever we need. In CoffeeScript, that'd look like this:

for key in hash.ownKeys()
for key in hash.allKeys()
for index in array.indexes()
for [item, index] in enumerate(array)
for [r, g, b] in elements.map (element) -> element.backgroundColorTuple()

Due to JavaScript's issues with defining methods on builtins, we'd probably use functions instead (they can be contained in a module these days), but you get the idea. You convert whatever structure you have into a sequence of tuples, and iterate over them.

You can use the same logic in list, dictionary and set comprehensions too.

foo = {
  key.toLowercase(): value * 2
  for key, value in someHash.items()
}

And generator expressions... Well, you get the point. Python nailed iteration.

If we supported unpacking a sequence inside a for-loop when the names are wrapped in brackets (for [key, value] in hash.items()), then we would have the option of using Python style for-loops without breaking backwards compatibility, assuming we want to keep something like the old behaviour as well. The square brackets are also consistent with CoffeeScript assignment syntax ([a, b, c] = [1, 2, 3]).

[JimPanic]

This is nice and a lot more intuitive. I support this a lot!

[kirly-af]

@carlsmith: I like the idea of unpacking sequences inside for loops.

Though I completely disagree we should introduce new functions and/or methods in the language like the enumerate() you suggest. I personally think CoffeeScript success is (partially) due its idiom It's just JavaScript.

Also, I don't consider the way CoffeeScript provides the index in for-in loops like magical.
The way I see it is that CoffeeScript treats Objects like Maps: a collection of sub-objects, iterable.

In the end, the only real difference we have with array-like values that are intended to work with for-in loops (ie. arrays and strings) is the key type (string vs number).

So to me the actual issue is the inconsistency we have between for-of and for-in:

(console.log key, val) for key, val of {'foo': 'bar'}
(console.log idx, val) for val, idx in ['foo', 'bar']

What would make the more sense to me would be to have only one for (I would choose for-in over for-of but this is not our concern here), and keep being consistent with the way we treat the index/key.

(console.log key, val) for key, val in {'foo': 'bar'}
(console.log idx, val) for idx, val in ['foo', 'bar']
(console.log idx, char) for idx, char in 'foobar'
(console.log count, item) for count, item in someIterrableObject # generator yield, es6 map, es6 set, etc.

Also, it would be easy to assume only value is required if we provide only one parameter to for (apologize I know parameter is not the proper term), we can assume only the value is requested.

(console.log key, val) for key, val in {'foo': 'bar'}
(console.log val) for val in {'foo': 'bar'}
(console.log idx, char) for idx, char in 'foobar'
(console.log char) for char in 'foobar'
# ... and so on

Now we have the main structure of our for loop: for <optionalKey>, value in someCollection.
Then, we could integrate the destructuring on both key and value:

(console.log foo, bar) for {foo: bar} in [{a: 1},{b: 2},{c: 3}]
(console.log thisIsA, mapKey) for {thisIsA: mapKey}, {andA: mapValue} in new Map(...) # maps keys can be any kind of values

This would require more efforts but in a perfect world, this is the way I would expect iteration to be.

[carlsmith]

Of course, there are different systems, and they each have there own merits, but Python is known for being especially good at iteration, so in an ideal world, I'd always opt for that approach. It has implications we didn't get into here as well. Python does a lot with iteration.

You don't have to add new functions like enum to the language. You just allow unpacking when more than one name inside brackets is used in a for-loop for [x, y] in coords and then people can add the rest of the functionality with library code and their own stuff. It's just sugar for for coord in coords then [x, y] = coord.

[kirly-af]

@carlsmith

Due to JavaScript's issues with defining methods on builtins, we'd probably use functions instead (they can be contained in a module these days), but you get the idea. You convert whatever structure you have into a sequence of tuples, and iterate over them.

That's the part of your post that confused me concerning the possible introduction of new functions.
I got your point about unpacking sequences and I completely agree this would be a great benefit to be able to use it in for loops as you demonstrated.

Though, my point was more about how to unify the different kind of CoffeeScript for loops, and so the inconsistency between for-in and for-of (ie. for val, index in vs for key, val of). Furthermore, we haven't talked yet about ES6 iterables. AFAIK those need to be integrated as neither for-in nor for-of loops can handle it.

Thus, a for-from form has been suggested to handle iterables in jashkenas/coffeescript#3832 (the whole issue is worth reading btw).
Should we introduce a new form ? Or should we unify it through a unique single for-in ?
I'd vote for the second option. The only issue is that it would involve if statements in the transpiled code, like suggested in that comment. I personally this would be better option.

Having 3 different kinds of for loops sounds crazy to me, there should be only one.

[DomVinyard]

I really like @kirly-af's example

(console.log key, val) for key, val in {'foo': 'bar'}
(console.log idx, val) for idx, val in ['foo', 'bar']

[carlsmith]

What would it compile to? We have to assume we don't know the type of the iterable, as it will often be an expression that evaluates to something that's defined elsewhere.

[carlsmith]

Having 3 different kinds of for loops sounds crazy to me, there should be only one.

Adding unpacking to loops doesn't create a new type of loop. It just makes the assignment in for-loops consistent with normal assignments. We could do what you're suggesting, and still allow unpacking in for-loops. We could have any kind or any number of for-loops, and still allow assignments to be unpacked within them.

I agree we should only have one kind of for-loop, but don't think it should have pseudo-unpacking, with only two names, and hardcode what gets unpacked based on the iterable's type. If we ever did create a dialect of CoffeeScript with one kind of for-loop, I'd much prefer Python's iteration, which handles all iterables, including instances of user-defined classes, and is far more flexible.

[carlsmith]

Sorry if that came across badly. I didn't mean to be rude. I appreciate you're trying to come up with something as close to CoffeeScript as possible, but sane as well, and in that context, what you have looks good.

I'm not certain, but I think CoffeeScript is inconsistent because it doesn't know the types, so it's difficult or impossible to cover every case correctly in the compiled code without injecting a function call for the type check, and they don't like to do that because of performance. Injecting a type check invocation wouldn't be too expensive as the iterable expression is only evaluated once (not per iteration, like while and until), so you probably could do what you're suggesting without any serious performance issues. Still, if you're changing the language enough to only have one kind of for-loop, why not just redo for-loops altogether?

[kirly-af]

Thanks @carlsmith for sharing your opinion, no worries I've taken no offense ! Sorry if that comment is going
to be long but I have many comments to answer.

Adding unpacking to loops doesn't create a new type of loop. It just makes the assignment in for-loops consistent with normal assignments.

Just to avoid misunderstandings, I confirm you I got your point here. Unpacking is a good thing with or without the new loop form/behavior.

What would it compile to?

Mostly inspired from this comment, I imagine something like:

foreach key, item in collection

would compile to:

for (var _iterator = collection,
  _i = 0,
  _it, _keys,
  _isArrayLike = Array.isArray(_iterator) || typeof _iterator === 'string',
  _isIterrable = !_isArrayLike && typeof collection[Symbol.iterator] === 'function',
  _iterator = _isIterrable ? _iterator[Symbol.iterator]() : _iterator;;
  ++_i)  {
  var _ref, _ref2;

  if (_isArrayLike) { // collection is array or string
    if (_i >= _iterator.length) break;
    _ref2 = _i;
    _ref = _iterator[_i];
  } else if (_isIterrable) {
    _it = _iterator.next();
    if (_it.done) break;
    _ref2 = _i;
    _ref = _it.value;
  } else { // object
    _keys = _keys || Object.keys(_iterator);
    if (_i >= _keys.length) break;
    _ref2 = _keys[_i];
    _ref = _iterator[_ref2];
  }

  var key = _ref2;
  var item = _ref;

  // loop body
}

This works for strings, arrays, objects and iterables.

Injecting a type check invocation wouldn't be too expensive as the iterable expression is only evaluated once (not per iteration, like while and until), so you probably could do what you're suggesting without any serious performance issues.

Indeed, here I keep the if statements inside the loop, and I don't even see any overhead on 100,000 items collections (every type of collections). It could be preferable to copy the loop in 3 different blocks but, referring to that comment:

Copying the entire loop body is not at all practical.

I don't know how reliable this comment is (though still babel author words).

We have to assume we don't know the type of the iterable, as it will often be an expression that evaluates to something that's defined elsewhere.

I miss your point. Could you provide an example of such case ?

I agree we should only have one kind of for-loop, but don't think it should have pseudo-unpacking, with only two names, and hardcode what gets unpacked based on the iterable's type.

I admit unpacking the index (I wouldn't use this term in this case) is questionable. Especially if we consider ES6 (Weak)Map, as the proposed foreach would require unpacking anyway to use it properly:
e.g.:

map = new Map [
  ['foo', 'bar']
  ['bar', ['foo']
]
(console.log count + ':', item) foreach count, item in map
# ouput: count: [key, value]
# 0: ['foo', 'bar']
# 1: ['bar', 'foo']

This is manageable and ideally should behave exactly like Objects.

If we ever did create a dialect of CoffeeScript with one kind of for-loop, I'd much prefer Python's iteration, which handles all iterables, including instances of user-defined classes, and is far more flexible.

As long as we have a way to handle iterables, we can handle user-defined classes (or more exactly objects). AFAIK, an iterable is just an object wich Symbol.iterator property yields iterators (ie. a generator).

Still, if you're changing the language enough to only have one kind of for-loop, why not just redo for-loops altogether

As I explain in #16, I changed my mind a little bit and I am now in favor of a new form. Of course the only reason to introduce a new kind of loop is to preserve the old behavior, without any possible overhead for those who would hate the feature (you know there would be MANY). I dislike for-from, and would suggest foreach-in.

[kirly-af]

I really love your idea about unpacking, but how would you get the index of arrays and strings inside the loop since we don't have any enumerate-like function in JS ? This is the only concern that makes me consider other options and index unpacking is the only option I can think about so far. I know most the time you don't care about the index, but there're some cases where it really matters.

Even though it doesn't handle objects ES6 for-of goal was to unify iteration. Since it doesn't provide a way to get the index (apart from Map of course), should we just inspire from it ?

[rattrayalex]

Wow, very impressive work there @kirly-af . Seems quite verbose output by coffeescript's standards but does seem to cover all bases I can think of.

One thing to keep in mind here is that there is array.entries() in ES6, which yields [index, item] tuples. But it only works on arrays (not even strings).

[kirly-af]

Seems quite verbose output

Indeed, it could be refactored. It was just to demonstrate how it could work.

One thing to keep in mind here is that there is array.entries() in ES6, which yields [index, item] tuples. But it only works on arrays (not even strings).

I wasn't aware of that method. It even makes us able to make this trick:
for (let [index, char] of Array.from('foobar').entries())

That's a little bit verbose, but works :). Furthermore Object.entries(), included in ES2017, is already supported by Chrome and Firefox (along Object.values()). We'll be able to do the following in JS:
for (let [key, val] of Object.entries(someObj))

I just discovered strings are iterable which means they made for-of generic enough. The more I think about all this, the more I think we could just keep it simple and just compile to ES6 for-of (which would require support for unpacking inside of for statements). The produced code will be cleaner, and there are very little benefits to base the behavior on the types:

• avoid doing Object.entries(obj) or Array.from(str).entries() (both ridiculously verbose)
• avoid the for-of overhead

The first benefit is obvious, but aligning with for-of would suppress all the magic around CS loops.
Concerning the second, as for-of and the Iterable interface are intended to be a big thing in JS, the engines will probably look forward to optimize it so I'm not sure it should not be our concern.

In the end my pov is:

• compiling to for-of would make only sense if we handle unpacking tuples inside our for loops.
• should we align for-in to for-of (basically we trust the future engines, what I think has been done on redux#344) or create a new form ?

This discussion looks to end up pretty much like cs#3882, so maybe, we should rather talk about it there ?

[carlsmith]

We have to assume we don't know the type of the iterable, as it will often be an expression that evaluates to something that's defined elsewhere.

I miss your point. Could you provide an example of such case ?

If the compiler is passed this source: (console.log x, y) for x, y in spam, it doesn't know the type of spam, but it's still valid CoffeeScript. The compiler assumes another script will create a global named spam first or this output will be concatenated into something else etc.

...how would you get the index of arrays and strings inside the loop since we don't have any enumerate-like function in JS ?

I would personally justify adding unpacking to for-loops as nice sugar that makes the language more consistent with itself, and people can incidentally write an enumerate function, or anything else they like from there. You're right though, that doesn't explain how people without an enumerate function will get the index. You can always desugar:

index = 0
until index is array.length
    item = array[index]
    index++

It's generally a good idea for CoffeeScript users to collect a set of generic helper functions they can add to the top of any file that could use them. Mostly, they're just one or two lines of code that help make the language a bit nicer. Adding an enumerate function would only be a couple more lines.

Honestly, I don't see any real problem with creating half a dozen helper functions in gists and encouraging people to improve them and add new ones and just point people to them. Even bundle super popular ones with the compiler, and add a flag to for injecting them into the top of files ;)

You made a lot of intelligent points @kirly-af and I've just been pretty flippant here. Sorry for that dude. I had hoped to dive into your answer more, but just don't have a lot of time right now. The misses wants to watch Dexter, so I had to wrap it up quick.

Thanks for all your hard work by the way. It's massively appreciated.

[rdeforest]

With mad props to @carlsmith and @kirly-af kirly-af, I'm hoping I can contribute to this discourse.

I'm a total n00b here, but I want to offer my answer to the "what would it compile to" question. Unfortunately, I've been so immersed in CS that I can only provide my answer in the form of translating CS to CS.

CS 6.0

    someCode = ->
      for idx, val in fn()
        block idx, val

CS 1.10

    someCode = ->
      _seq = fn()

      if _iter = _seq[Symbol.iterator]
        _i = 0

        while not (step = _iter.next).done
          block _i++, step.value

      else if _len = _seq.length
        for i in [0 .. _len]
          block i, _seq[i]

      else
        for key in Object.getOwnPropertyNames _seq
          block key, _seq[key]

Yes, it adds up to two branch tests, and triplicates the block but I'm certain folks can golf this down to something good enough for our purposes and I think the semantics fit well into the CS model.

And previous uses should still not surprise anyone:

CS 6

    someCode = ->
      for val in fn()
        block val

CS 1.10

    someCode = ->
      _seq = fn()

      if _iter = _seq[Symbol.iterator]
        while not (step = _iter.next).done
          block step.value

      else if _len = _seq.length
        for i in [0 .. _len]
          block _seq[i]

      else
        for key in Object.getOwnPropertyNames _seq
          block _seq[key]

[kirly-af]

Following my comment on #36, I really think we should just output for-in to for-of loops. That would be quite silly as it would mean CS outputs the exact opposite of JS. But that's the simplest way to do it.

I know there's a performance concern, but personnally I don't even use regular for loops in my JS anymore, probably many people do as well... and we're all fine. The performance might have been quite bad during ES6 early times, but I'm confident all modern engines will keep optimizing it as iterators and generators are now mainstream.

I don't think a slight performance hit is enough to introduce a new kind of for loops in the language (even thinking about it makes me sad). Let's preserve CS simplicity.

EDIT: looks like we're going for for-from in the end (see here).

[GeoffreyBooth]

This has been implemented.

[coffeescriptbot]

This issue was moved to jashkenas/coffeescript#4909