Why don't array comprehensions desugar into map?

[samuela]

I discovered this fact while working with something like the following:

julia> da = distribute(1:10)
10-element DArray{Int64,1,UnitRange{Int64}}:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10

julia> [println((myid(), x)) for x in da];
(1,1)
(1,2)
(1,3)
(1,4)
(1,5)
(1,6)
(1,7)
(1,8)
(1,9)
(1,10)

julia> map((x) -> println((myid(), x)), da);
    From worker 2:  (2,1)
    From worker 3:  (3,6)
    From worker 2:  (2,2)
    From worker 2:  (2,3)
    From worker 2:  (2,4)
    From worker 2:  (2,5)
    From worker 3:  (3,7)
    From worker 3:  (3,8)
    From worker 3:  (3,9)
    From worker 3:  (3,10)

Which raises the obvious question: why don't array comprehensions simply desugar into maps? This smells of bad language design that apparently identical language features have subtly different semantics.

[tkelman]

Pretty much a duplicate of #670 (except that comprehension syntax was slightly different 2.5 years ago), very closely related to #8450. Map and array comprehensions are different right now due to inference and a number of other reasons.

I don't think we have a syntax for parallel array comprehension, so those are all getting executed on the master process in the first case. I'd call this an unexpected interaction of map (or comprehensions, pick one) and DArrays. Please do highlight things like this that surprise you (though the mailing list might be better for simple things like this at first, since there are a number of closely related issues already open that we will point you to), but the remarks on bad language design aren't a productive way to engage the community on your concerns.

[JeffBezanson]

One difference is that with multiple arguments, map zips them (as is the convention) while comprehensions iterate over their product and yield a multi-dimensional array.

[samuela]

Not meant as a jab; I understand that language design is tricky and that Julia hasn't yet hit v1.0. My point is that these types of issues are particularly critical when designing and implementing new languages, so they should be discussed thoroughly and implemented carefully. I see this as an issue because it suggests that something is weird in the way that array comprehensions are implemented. More precisely, I am concerned that features like array comprehension are given special status in the language. Needless to say, VIP features should be kept to a minimum.

I read @StefanKarpinski's comment on #670. The issue is quite old, so I'm not sure how much of what is said is still relevant but his comment raises a few questions:

• As noted, map's behavior is really gross. Julia's type system is an unorthodox beast that I don't fully understand yet but something clearly needs to change if a function as straightforward as map is at odds with the type system.
• Why hasn't this dynamic type inference magic been exposed yet?

I have loads of more general questions about the type system but my first post to the group has not been approved yet. If anyone here has the power over those sorts of things, please accept my post! I'll try not to spam!

[samuela]

@JeffBezanson Yes, technically I was thinking of flatMap.

[StefanKarpinski]

We should generally move to type-inference-independent versions of map and comprehensions, as discussed here: #7258.

[pao]

I have loads of more general questions about the type system but my first post to the group has not been approved yet.

If you posted to -dev, that's a smaller approval group, so it may take longer. If you posted to -users I don't see anything in the queue?

[StefanKarpinski]

Apparently Google Groups decided to just stop notifying me about pending members or messages on julia-dev. Sorry for the delay – I had no idea there was a backlog there :-(

[samuela]

Hmm, I posted to julia-users so I guess it just didn't send properly. I guess I'll try posting again later.

[JeffBezanson]

There are several issues about redesigning comprehensions, so I'm going to close this one as it's more just a discussion.