Generify BorrowedBuf and BorrowedCursor

[carbotaniuman]

Proposal

Problem statement

Rust does not provide great provisions for reading data into uninitialized buffers, and while BorrowedBuf and BorrowedCursor solve the problem for u8 in the context of I/O, generifying the struct would solve this problem for the entire ecosystem, instead of just for I/O.

Motivating examples or use cases

My main use case is for safe FFI wrappers, where there are a lot of libraries that take a pointer and a length that could be massaged to work with BorrowedBuf, either as a caller or for reimplementation in safe(r) Rust. Some examples include I/O that deals in c_char or even wchar due to legacy APIs.

https://docs.rs/rustix/latest/rustix/fs/fn.flistxattr.html

In addition, the general idea behind reading data into an uninitialized buffer is general enough that existing Rust code is bound to run into the issue, such as rayon.

rust-lang/rust#78485 (comment)

Solution sketch

The solution is to add a (potentially defaulted to u8) generic parameter after the existing lifetime parameters. The documentation of the struct may have to change to reflect it dealing in elements and not in bytes.

Alternatives

Another solution could be a BorrowedBuf being a typealias to a new struct that is generic, or creating an entirely new struct for the purpose of storing arbitrary items, as opposed to I/O. Creating a new struct very redundant in my opinion, and having a single way to talk about reading into uninitialized memory seems preferable.

Links and related work

This was discussed during the RFC and rejected as future work. I think that generifying the API to allow for progressive initialization would be useful for other many use cases, even in the presence of a separate Vec-like data structure that works off of uninitialized memory.

rust-lang/rfcs#2930 (comment)

What happens now?

This issue is part of the libs-api team API change proposal process. Once this issue is filed the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.

Possible responses

The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):

• We think this problem seems worth solving, and the standard library might be the right place to solve it.
• We think that this probably doesn't belong in the standard library.

Second, if there's a concrete solution:

• We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
• We're not sure this is the right solution, and the alternatives or other materials don't give us enough information to be sure about that. Here are some questions we have that aren't answered, or rough ideas about alternatives we'd want to see discussed.

[joshtriplett]

We talked about this in today's libs-api meeting.

BorrowedBuf is specifically a type for doing I/O. It tracks more than you might want for potential other applications, and we don't want to simultaneously try to be the most optimal type for I/O and also be a general type meant for other uses.

For use with i8 or c_char, that's a zero-cost safe conversion; you can use a BorrowedBuf that operates on u8 and then treat the result as i8 or c_char.

For use with other types, we're not closing the door on the future possibility of either adding a separate type or making BorrowedBuf an alias for that type, but we don't want to do that at this time.