reverse iteration for eachline

[stevengj]

This PR implements Iterators.Reverse iteration for the EachLine iterator, so that now e.g. last(eachline(io)) will work. (This seems to have been requested multiple times on discourse etcetera, and is nontrivial for users implement efficiently themselves. It also seems to be a pretty common question in Python, where the most common answers involve reading the whole file to get the last few lines., or executing an external tail program.)

For seekable streams, it works by reading the data from the end in 4kiB chunks. Performance seems comparable to forwards iteration.

For non-seekable streams, it falls back to reading the whole stream into memory. However, for last(eachline(...)) and take(Reverse(eachline(...)), n) it uses an algorithm (similar to how the Unix tail function works) that requires only bounded memory — it reads the file in a sequence of 4kiB chunks, but discards earlier chunks if they are not needed for the requested number of lines. it throws an exception when seekend or seek is called.

I noticed that first(r::Reverse) was calling last(r.itr), which fails for non-indexable iterators because last(itr) defaults to itr[end], so I changed it to use the default implementation (in abstractarrays.jl) which iterates once. Otherwise, you can't implement last(itr::MyType) by first(Iterators.Reverse(itr)). This is analogous to how last(itr, n) is already implemented. (It might be good to change the default last(itr) method, but that's a topic for another PR.)

[tkf]

I don't think non-seekable IO can satisfy the last interface since it is defined to be O(1)

julia/base/abstractarray.jl

Lines 456 to 460 in 4805d54

	last(coll)
	Get the last element of an ordered collection, if it can be computed in O(1) time. This is
	accomplished by calling [`lastindex`](@ref) to get the last index. Return the end
	point of an [`AbstractRange`](@ref) even if it is empty.

A similar discussion on length: #35947

[stevengj]

It would be easy to restrict this to seekable io and to throw an error otherwise, if desired.

Update: I've changed the PR to drop non-seekable support, since that seemed controversial. (I saved the non-seekable branch at https://github.com/stevengj/julia/tree/sgj/reverse-eachline-nonseekable.)

[stevengj]

Unrelated CI error on freebsd64:

Error in testset LinearAlgebra/lu:
Error During Test at none:1
  Got exception outside of a @test
  ProcessExitedException(5)

[stevengj]

Apparently unrelated error on freebsd64 (Error in testset LinearAlgebra/lapack from ProcessExitedException(6)) and unrelated error on win32:

Error in testset Test:
Test Failed at C:\buildbot\worker-tabularasa\tester_win32\build\share\julia\stdlib\v1.8\Test\test\runtests.jl:777
  Expression: orig == Random.default_rng()
   Evaluated: Xoshiro(0xc1fb896c346f54ba, 0x3f32061aaae1bb3a, 0x6256794da49331ff, 0x85db582666fa8259) == TaskLocalRNG()
Error in testset Test:
Test Failed at C:\buildbot\worker-tabularasa\tester_win32\build\share\julia\stdlib\v1.8\Test\test\runtests.jl:778
  Expression: rand(orig) == rand()
   Evaluated: 0.544086357518677 == 0.6916330293666764

This PR should be ready for review?

[stevengj]

Updated to handle CRLF line endings, with a test.

[tkf]

LGTM!

[stevengj]

(CI failures on freebsd64 and win32 are unrelated: ProcessExitedException in testset LinearAlgebra. Is there an issue tracking this? I'm seeing it on several PRs.)

[tkf]

Looks like JuliaLang/LinearAlgebra.jl#873?

[stevengj]

Rebased. Hopefully can be merged if green?