Should Base.BufferStream have content present after eof(stream)==true?

[ericphanson]

julia> io = Base.BufferStream()
BufferStream(bytes waiting=0, isopen=true)

julia> let
            t = @async begin
                for _ = 1:10
                    bytes = readavailable(io)
                    @show (String(bytes), eof(io), isopen(io))
                end
            end
            run(pipeline(`echo "hi"`, stdout=io))
            run(pipeline(`echo "bye"`, stdout=io))
        end
(String(bytes), eof(io), isopen(io)) = ("hi\n", true, false) # EOF is true here
(String(bytes), eof(io), isopen(io)) = ("bye\n", true, false) # we get more bytes out here
(String(bytes), eof(io), isopen(io)) = ("", true, false)
(String(bytes), eof(io), isopen(io)) = ("", true, false)
(String(bytes), eof(io), isopen(io)) = ("", true, false)
(String(bytes), eof(io), isopen(io)) = ("", true, false)
(String(bytes), eof(io), isopen(io)) = ("", true, false)
(String(bytes), eof(io), isopen(io)) = ("", true, false)
(String(bytes), eof(io), isopen(io)) = ("", true, false)
(String(bytes), eof(io), isopen(io)) = ("", true, false)
Process(`echo bye`, ProcessExited(0))

The docs for eof say:

Test whether an I/O stream is at end-of-file. If the stream is not yet exhausted, this function will block to wait for more data if necessary, and then return false. Therefore it is always safe to read one byte after seeing eof return false. eof will return false as long as buffered data is still available, even if the remote end of a connection is closed.

and that seems contradicted here, e.g. the stream does not seem "exhausted" but eof returns true. This example is related to #49233 (and JuliaLang/Pkg.jl#3430 (comment)) but Base.BufferStream seems to have particularly weird behavior in this case (compared to IOBuffer and Base.PipeBuffer) so I thought it was worth a separate issue.

xref #42424 which discusses other issues with Base.BufferStream

[vtjnash]

Sounds like a bug. The second run call should be throwing exceptions when it tries to operate on a closed object.

[pfitzseb]

Why does run close the buffer? Is that documented somewhere?

[vtjnash]

It copies the underlying stream and is supposed to be "indistinguishable" from it. Since the underlying IO gets closed, so this gets closed too. If you didn't want that, then you should have made sure the underlying stream doesn't do that either. Which is to say, make a Pipe() yourself, and then explicitly close it when you want to indicate that it is done. There is some helper functions (e.g. Base.setup_stdio) that can assist with this.

[ericphanson]

I’m trying to understand streams better-

It copies the underlying stream

here “it” is run?

the underlying IO gets closed

This is the underlying IO of the process being run?

make a Pipe() yourself

the difference here is Pipe is the Julia representation of an OS-level object, as opposed to a Julia object?

[pfitzseb]

It copies the underlying stream and is supposed to be "indistinguishable" from it.

Fair enough, but that is not documented anywhere afaict. It totally makes sense that the fd or I/O stream would be closed by run, but an IOBuffer doesn't look like an object at the level of abstraction to me.

It's also worth noting that this changed somewhat recently (Julia 1.7 did not close the IOBuffer).

[vtjnash]

Note that IOBuffer should NEVER be used here for process spawn output, in any version of Julia, but I have not re-implemented the error for that yet.