-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make unvetted size
throw an error for arrays with non-1 indexing
#17228
Conversation
# SafeIndices(). | ||
size( ::IndicesSafety, A::AbstractArray) = size(A) | ||
size( ::IndicesSafety, A::AbstractArray, d) = size(A,d) # fixme | ||
# size(s::IndicesSafety, A::AbstractArray, d) = d <= ndims(A) ? size(s, A)[d] : 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For some reason, uncommenting this line causes inference to bail during bootstrap. (Avoiding ndims
and using the parameter variant, A::AbstractArray{T,N}
, does not fix the problem either.)
As a consequence, people who write packages for unconventionally-indexed arrays have to specialize both size(s, A)
and size(s, A, d)
, which is a little unfortunate but not too terrible.
26d84cd
to
9faec4e
Compare
size( ::IndicesSafety, A) = size(A) | ||
size( ::IndicesSafety, A::AbstractArray, d) = size(A,d) # fixme | ||
# size(s::IndicesSafety, A::AbstractArray, d) = d <= ndims(A) ? size(s, A)[d] : 1 | ||
size{N}(s::IndicesSafety, A::AbstractArray, d1::Integer, d2::Integer, dx::Vararg{Integer, N}) = (size(s, A, d1), size(s, A, d2), ntuple(k->size(s, A, dx[k]), Val{N})...) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wrap this?
I'm not following when |
@@ -1417,7 +1447,7 @@ function mapslices(f, A::AbstractArray, dims::AbstractVector) | |||
idx[d] = Colon() | |||
end | |||
|
|||
r1 = f(view(A, idx...)) | |||
r1 = f(A[idx...]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not mentioned in commit message, maybe should have a specific test
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll split this out into a separate PR.
I'll modify the unmerged JuliaLang/www_old.julialang.org#386 when things settle out here. Bottom line is that neither is expected to be used in user-code; the idea is:
Internally within Base, |
@@ -577,6 +577,7 @@ export | |||
rot180, | |||
rotl90, | |||
rotr90, | |||
@safeindices, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
needs docstring inline and in rst, and probably a dedicated test of the macro?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will add the docstring. The test happens as a consequence of the new @test_throws
in offsetarray.jl
and the fact that the remaining tests work.
Test failure seems unrelated. I'm tempted to merge. I hope people realize this PR makes the unconventionally-indexed arrays brittle-by-design: in practice, this forces you to look at basically every "unvetted" function you pass them to. I've modified the big stuff (indexing, broadcasting, show) and a number of smaller routines, but Base and definitely packages will, in the early days, require changes. I think this is the best approach, however, since uncontrolled failures are much worse. |
s = op(s, f(Ai)) | ||
inds = linearindices(A) | ||
n = length(inds) | ||
@inbounds begin |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does this make an assumption that f(a)
is bounds-safe?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@inbounds
doesn't propagate, even with inlining (unless so marked). We had inbounds
previously for the n<16
case, but it missed several A[i]
accesses.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I clearly need to watch https://www.youtube.com/watch?v=l0icRnmqeYs a few more times
Okay, then I guess devdocs is better, but in the current state it'll be easy to accidentally use the wrong one in base. |
Are you suggesting I move the blog post to devdocs? I can do that. I'll also clarify the meaning of the two alternatives. |
I suppose that works. Would like to get out of the habit of merging things without accompanying documentation. |
9faec4e
to
67a2c0d
Compare
OK, other than moving the blog post I think I've addressed all the review comments---and frankly I think it's best to do that once #17137 merges. I'm having trouble building the docs locally: tim@diva:~/src/julia-0.5$ make -C doc html
make: Entering directory '/home/tim/src/julia-0.5/doc'
. ./../deps/build/julia-env/bin/activate && pip install sphinx==1.3.1 \
&& pip install -r /home/tim/src/julia-0.5/doc/requirements.txt
Traceback (most recent call last):
File "/home/tim/src/julia-0.5/deps/build/julia-env/bin/pip", line 7, in <module>
from pip import main
File "/home/tim/src/julia-0.5/deps/build/julia-env/local/lib/python2.7/site-packages/pip/__init__.py", line 14, in <module>
from pip.utils import get_installed_distributions, get_prog
File "/home/tim/src/julia-0.5/deps/build/julia-env/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 27, in <module>
from pip._vendor import pkg_resources
File "/home/tim/src/julia-0.5/deps/build/julia-env/local/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 36, in <module>
import plistlib
File "/usr/lib/python2.7/plistlib.py", line 62, in <module>
import datetime
ImportError: No module named datetime
Makefile:33: recipe for target '../deps/build/julia-env/bin/sphinx-build' failed
make: *** [../deps/build/julia-env/bin/sphinx-build] Error 1
make: Leaving directory '/home/tim/src/julia-0.5/doc'
tim@diva:~/src/julia-0.5$ locate datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/lib/python2.7/site-packages/numpy/core/tests/test_datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.10.2-py27_0/lib/python2.7/site-packages/numpy/core/tests/test_datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.10.2-py27_0/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.10.4-py27_0/lib/python2.7/site-packages/numpy/core/tests/test_datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.10.4-py27_0/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.10.4-py27_1/lib/python2.7/site-packages/numpy/core/tests/test_datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.10.4-py27_1/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.11.0-py27_0/lib/python2.7/site-packages/numpy/core/tests/test_datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.11.0-py27_0/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.11.0-py27_1/lib/python2.7/site-packages/numpy/core/tests/test_datetime.py
/home/tim/.julia/v0.4/Conda/deps/usr/pkgs/numpy-1.11.0-py27_1/lib/python2.7/site-packages/numpy/core/tests/test_datetime.pyc
/usr/lib/python2.7/dist-packages/numpy/core/tests/test_datetime.py
/usr/lib/python2.7/dist-packages/numpy/core/tests/test_datetime.pyc
/usr/lib/python2.7/dist-packages/zope/interface/common/idatetime.py
/usr/lib/python2.7/dist-packages/zope/interface/common/idatetime.pyc
/usr/lib/python2.7/dist-packages/zope/interface/common/tests/test_idatetime.py
/usr/lib/python2.7/dist-packages/zope/interface/common/tests/test_idatetime.pyc
/usr/lib/python3.5/datetime.py If the solution is not obvious, perhaps someone else can build the docs for me and push it to this branch? |
|
a4e50e5
to
a9be8ce
Compare
Thanks, @tkelman! I made a few more doc tweaks, and decided to make Before merging, we should check performance: @nanosoldier |
|
||
trailingsize(A, n) = trailingsize(UnsafeIndices(), A, n) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are you able to build the docs? this may make these 2 separate code blocks?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't (but thanks for noticing that I forgot make julia-genstdlib
).
3008bd7
to
542b088
Compare
@nanosoldier |
Nanosoldier seems to be taking a well-deserved holiday. |
Looks like there was a network issue while I was away, since the error in the server logs is @nanosoldier |
Ugh, master broke Nanosoldier's v0.5 version of JLD, which is why the build is failing. I'll update it and see if that helps. ref JuliaCI/BenchmarkTools.jl#15 |
@nanosoldier |
Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @jrevels |
|
||
# newindexer(shape, A) generates `keep` (for use by `newindex` above) | ||
# for a particular array `A`, given the broadcast_shape `shape` | ||
# Equivalent to map(==, indices(A), shape) (but see #17126) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also equivalent to indices(A) .== shape
or?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we don't support that syntax for tuples (but conceptually, yes)
All review comments addressed. |
_maybe_reshape(::LinearFast, A::AbstractArray, i) = A | ||
_maybe_reshape(::LinearSlow, A::AbstractVector, i) = A | ||
_maybe_reshape(::LinearSlow, A::AbstractArray, i) = _maybe_reshape(LinearSlow(), index_ndims(i), A) | ||
_maybe_reshape{N}(::LinearIndexing, ::NTuple{N}, A) = _= reshape(A, Val{N}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
= _=
typo?
82b79ff
to
f85f858
Compare
Set again (I hope!) |
I don't entirely get what these various |
_maybe_reshapeThis is extended from an older function from 51b4bc1, Since types are involved, it's important to do all this in a type-stable manner. So basically this implements the following logic in a way the compiler can follow: IF this is a LinearSlow array, AND the dimensionality of the indices doesn't match the dimensionality of the array, reshape the array to match the dimensionality of the indices. The computation of the dimensionality of the indices is actually a little tricky because of maybe_onetoThis is to handle the opposite case from partial linear indexing, when the user supplies an extra index (e.g., indexing a vector as |
Just an FYI that from my perspective this is merge-worthy and hence closes a 0.5.0 milestone. Alternatively, I can await more reviews and perhaps pile on my promise in https://github.com/JuliaLang/julia/pull/17283/files/c68a2cd9919cdf4a23243b313670ad8dc41a9c27#r71298118. |
let's do @nanosoldier |
Assuming the nanosoldier run looks good, I think you should go ahead and merge, @timholy. |
Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @jrevels |
More than good. |
Thanks @timholy, this approach looks good. Kudos especially for the excellent documentation. |
Hope you had a great vacation! And thanks for the guidance---as usual, your taste is exquisite, and I just wasn't brave enough to throw a basic function like |
For arrays that violate the assumption that indexing starts at 1, there's a risk that formerly-safe code could exit with nasty errors, see here. Discussion in #16260 led to a good suggestion for a migration strategy: during julia-0.5,
size
should throw an error by default for such "unconventional" arrays. This PR introduces the necessary infrastructure:IndicesSafety
trait that can be passed as an argument throughsize
and related functions@safeindices
that you can use to wrap functions that you've certified are safe for arrays with unconventional indices.Together with #17137, this closes #16973 and should be the end of array changes that I envision for 0.5. (It is expected that more functions will become
@safeindices
over 0.5.x releases as people increasingly use such unconventional arrays.)