iterable[T]

[timotheecour]

• someIter(args) is now well typed: iterable[T] instead of untyped
• solves many long standing issues related to iterators
• sigmatch error messages show iterable[T] for iterables, which is more informative
• procs like toSeq, elementType etc can now be defined without untyped
• all (or almost all?) procs in sequtils can now be made more general, accepting also iterable inputs instead of openArray
• iterable arguments can now be used in UFCS/MCS, eg iota(3).toSeq now works

see tests

will fix these

nim-lang/RFCs#512
#9219
#13595
#14148
#16545
and sidestep #14778
and reduce cases where we hit #14827

note

this will make it much easier to implement online algorithms, eg summation functions should be usable in online fashion, more general than openArray · Issue nim-lang/Nim#288 · nim-lang/RFCs, avoiding issues related to untyped, and enabling UFCS/MCS

future work

• update spec
• update/simplify toSeq + all other templates that take an iterator (but this may require conditional compilation for bootstrap support)
• add a way to compute typeof(x, asIterable) which would return iterable[T] instead of T for iterables
• make or smarter so that it works with supertypes of typed, eg: seq or iterable
• support myiter is iterable
• investigate for loop invoked from generic procedure defined in another module cant find items iterator #11167; it seems like a different issue
• add test cases to close above issues
• update section on https://nim-lang.org/docs/manual.html#templates-limitations-of-the-method-call-syntax ; var info = something().toSeq now does work; and all templates that can take iterable instead of untyped also do work with MCS/UFCS. But again, untyped params are still problematic for MCS/UFCS and I can't think of a way to fix this because the problem is essentially ambiguous (outside of RFC: untyped(expr) for untyped params in overloaded routines timotheecour/Nim#630).

[konsumlamm]

What exactly is iterable[T]? Is it a builtin type class, like Ordinal[T]? If so, why use a lowercase name and not Iterable[T]?

And what exactly does iterable[T] encompass? All iterators with a "yield type" of T? Something else (seq[T], openArray[T], everything with an iter proc)?

Also, is there any previous discussion of this?

[timotheecour]

What exactly is iterable[T]? Is it a builtin type class, like Ordinal[T]? If so, why use a lowercase name and not Iterable[T]?

It's a type class; most other builtin type classes are lowercase.
This helps distinguish user or library defined types from builtin types that require compiler magic. Note that SomeFloat can be consider non-builtin even though it's in system, as it doesnt' require magic.

And what exactly does iterable[T] encompass? All iterators with a "yield type" of T? Something else (seq[T], openArray[T], everything with an iter proc)?

all calls to an iterator (inline or closure):

when true:
  template fn(a: iterable) = discard
  iterator iota(n: int): auto = (for i in 0..<n: yield i)
  fn(iota(3)) # ok
  fn(items(@[1,2])) # ok
  # fn(@[1,2]) # error

by design fn(@[1,2]) would give CT error:
here's the error msg:

t11921.nim(314, 5) Error: type mismatch: got <seq[int]>
but expected one of:
template fn(a: iterable) [template declared in t11921.nim(310, 12)]
  first type mismatch at position: 1
  required type for a: iterable [iterable declared in /Users/timothee/git_clone/nim/Nim_prs/lib/system.nim(128, 5)]
  but expression '@[1, 2]' is of type: seq[int] [sequence]

Also, is there any previous discussion of this?

I had an RFC in my nim fork (timotheecour#322)

[juancarlospaco]

changelog ?.

EDIT(timotheecour): done

[saem]

Apologies, if my questions are rather obvious, I'm trying to understand this:

• there are a number of skipTypes in various parts of sem that are required to get at the underlying type, my mind could be playing tricks on me but is the testing sufficient because it feels like there should be more updates for those?
• for iterable itself, shouldn't there be an underlying concept of iterable such that everything unifies, maybe I missed that?

[timotheecour]

there are a number of skipTypes in various parts of sem that are required to get at the underlying type, my mind could be playing tricks on me but is the testing sufficient because it feels like there should be more updates for those?

my understanding is these only concerned typed values, which iterable isn't part of; this doesn't constitute a bug though; this PR comes with tests, if we find something that doesn't work, we can always fix it in subsequent PR.

for iterable itself, shouldn't there be an underlying concept of iterable such that everything unifies, maybe I missed that?

this could be done in future work, but is not strictly necessary, and the issues mentioned in PR body can be fixed without such unification, which has its own can of worms. iterable[T] is in principle all you need, you can always pass a called iterator instead of passing a value that has an items defined for it, eg:

# instead of:
foo(@[1,2])
# use:
foo(items(@[1,2])

but as I showed in this PR, you can have an overload that take iterable and an overload that takes a typed value (possible taking a Itemable[T] concept which would check presence of a suitable items iterator defined on it).

[Araq]

How does this solution compare to the (to me) much more obvious tyIter (modelled like tyProc) solution?

[timotheecour]

tyProc is for a proc type, not for a called proc; if you model tyIter like tyProc, this would match an iterator type, not a called iterator, which is what we want here. eg:

iterator iota(n: int): int = (for i in 0..<n: yield i)
assert toSeq(iota(3)) == @[0,1,2]

after this PR, this can work with template toSeq[T](a: iterable[T]): seq[T] instead of requiring untyped param. Furthmore, MCS works. I'm not sure how tyIter would help, or at least this would need more details.

[konsumlamm]

Any update on this? I think this would be a very useful addition.

If I understand correctly, this should allow us to massively simplify enumerate by turning it into an iterator:

iterator enumerate[T](iter: iterable[T]): (int, T) =
  var i = 0
  for x in iter:
    yield (i, x)
    inc i

If this would not be possible, could this be achieved in a similar way?

[timotheecour]

@Araq PTAL; I will update spec in followup PR

@konsumlamm

Any update on this? I think this would be a very useful addition.

I've rebased to fix bitrot conflicts and fixed the case of proc fn(a: iterable[int]) = discard so that it errors in typeAllowedAux instead of later. This PR is ready to go again.

If I understand correctly, this should allow us to massively simplify enumerate by turning it into an iterator:

this requires #11992 (see tests/magics/tlambda_iterators.nim which has such examples) but would become more typesafe and easier to use with this PR (because of iterable instead of untyped). This currently works with #11992:

when true:
  import std/lambdas
  {.push experimental:"alias".}
  iterator enumerate(iter: aliassym): (int, char) =
    # `auto` or (int, typeof(iter)) will also be possible in future PRs
    var i = 0
    for x in iter:
      yield (i, x)
      inc i
  for i, ai in enumerate(lambdaIter {'a', 'b', 'c'}):
    echo (i, ai)

[Araq]

I will update spec in followup PR

Please do it in this PR, I now understand why and how you do it so I could write the spec myself. However, I want to see if my ideas agree with yours.

[konsumlamm]

@timotheecour

What about something like this?

template enumerate[T](iter: iterable[T]): iterable[(int, T)] =
  (iterator (): (int, T) =
    var i = 0
    for x in iter:
      yield (i, x)
      inc i)()

I experimented a bit, but couldn't get it to work. Also, when using untyped as return type in the above, i get an internal error on instantiation:

Error: internal error: expr(nkVarTuple); unknown node kind

EDIT: It seems like the error is unrelated to this PR, also the iterator seems to be transformed to a proc...

[timotheecour]

@Araq

Please do it in this PR, I now understand why and how you do it so I could write the spec myself. However, I want to see if my ideas agree with yours.

done, PTAL

@konsumlamm this should be defered to future work, again, see tests/magics/tlambda_iterators.nim from #11992

[Araq]

Problematic. See my final comment in the review.

[timotheecour]

replied in #17196 (comment)

[Araq]

The solution is quite good, however: Imagine a typed macro that takes someIter(args) and then does type introspection of the typed expression. Does that now produce the new tyIterator type? If so, that is a minor breaking change. If not, well maybe it should... :-)

[timotheecour]

Imagine a typed macro that takes someIter(args) and then does type introspection of the typed expression. Does that now produce the new tyIterator type?

macro fn(a: typed) = discard
fn(items(3..6))

produces, both before and after this PR:

Error: attempting to call routine: 'items'
found ...

Future work can decide whether typed should accept an iterable (it can make sense, given that typed also accepts statements, eg x.inc), or if we instead want to support typed or iterable.

If so, that is a minor breaking change

what do you mean? if/when we make typed accept iterable, it would turn a CT error into working code

[Araq]

produces, both before and after this PR

Thanks, was not aware. That answers my question.