Support operations on byte arrays

[joshlf]

Progress

• Add ByteArray as private type
• Use ByteArray internally to experiment
• Add public uses of ByteArray

Motivation

We want to be able to support operations on byte arrays, especially [u8; size_of::<T>()] for some zerocopy-compatible type T. E.g.:

trait FromBytes {
    fn from_bytes(bytes: [u8; size_of::<Self>()]) -> Self
        where Self: Sized;
    fn ref_from_bytes(bytes: &[u8; size_of::<Self>()]) -> Self
        where Self: Sized + Unaligned + Immutable;
}

trait AsBytes {
    fn into_bytes(self) -> [u8; size_of::<Self>()]
        where Self: Sized;
    fn as_bytes(&self) -> &[u8; size_of::<Self>()]
        where Self: Sized + Immutable;
}

let t: T = transmute!([0u8; size_of::<T>()]);

A lot of code both inside zerocopy and in user code currently has no way to reason about size equality on byte slices, and so ends up re-doing bounds checks. Consider this code from Fuchsia's packet crate:

fn take_obj_front<T>(&mut self) -> Option<Ref<B, T>>
where
    T: Unaligned,
{
    let bytes = self.take_front(mem::size_of::<T>())?;
    // new_unaligned only returns None if there aren't enough bytes
    Some(Ref::new_unaligned(bytes).unwrap())
}

In this code, self.take_front returns Option<&[u8]>, and so the type system can't reason about the returned byte slice satisfying bytes.len() == size_of::<T>().

As part of #1315, we'd like to support the general pattern of reading or writing objects to byte slices or fancier buffer types. Given support for byte arrays, we could write something like:

trait Buffer {
    fn take_bytes_front<const N: usize>(&mut self) -> Option<&[u8; N]>;

    fn take_obj_front<T: Unaligned>(&mut self) -> Option<&T> {
        let bytes = self.take_bytes_front::<{size_of::<T>()}>()?;
        Some(transmute_ref!(bytes))
    }
}

Stabilize size_of::<T>()

One approach we could take to accomplish this would be to stabilize size_of::<T>() for use in a type in a generic context (a special case of generic_const_exprs.

Polyfill

Another approach - that we can implement on our own without being blocked on Rust - is to add a polyfill type like the following

/// An array of `size_of::<T>()` bytes.
///
/// Since the `generic_const_exprs` feature is unstable, it is not possible
/// to use the type `[u8; size_of::<T>()]` in a context in which `T` is
/// generic. `ByteArray<T>` fills this gap.
///
/// # Layout
///
/// `ByteArray<T>` has the same layout and bit validity as `[u8; size_of::<T>()]`.
#[derive(FromBytes, Unaligned)]
#[repr(transparent)]
pub struct ByteArray<T>(
    // INVARIANT: All of the bytes of this field are initialized.
    Unalign<MaybeUninit<T>>,
);

impl<T> ByteArray<T> {
    // Not necessarily public. This is where we write the unsafe code that understands
    // that `size_of::<T>() == size_of::<ByteArray<T>>()` since the type system itself
    // isn't smart enough to understand that (at least when `T` is generic).
    fn as_t(&self) -> Ptr<'_, T, (invariant::Shared, invariant::Any, invariant::Initialized)> {
        let ptr = Ptr::from_ref(self);
        // SAFETY: TODO
        let ptr = unsafe { ptr.cast_unsized(|b| b as *mut T) };
        // SAFETY: By safety invariant on `ByteArray`, `ByteArray<T>` has the same bit validity
        // as `[u8; _]`, which requires its bytes all be initialized.
        unsafe { ptr.assume_initialized() }
    }
}

Using this polyfill, we could write the Buffer trait from the motivation section as:

trait Buffer {
    fn take_bytes_front<T>(&mut self) -> Option<&ByteArray<T>>;

    fn take_obj_front<T: Unaligned>(&mut self) -> Option<&T> {
        let bytes = self.take_bytes_front::<T>()?;
        Some(bytes.as_t())
    }
}

If we use a type which supports unsized types (Unalign doesn't), we could even make this more powerful than [u8; size_of::<T>()]. For T: Sized, ByteArray<T> would have the same layout as T, but for T: !Sized, it would have a layout closer to [u8]. It's unclear how an unsized version of this could be constructed, though, since the fat pointer would need to know the number of trailing slice elements in T, not the number of bytes.

Interior mutability

TODO: Explain why Stacked Borrows would require T: Immutable, but why we may not need that bound in practice (ie, we can "disable" interior mutability).
d a &ByteArray<T> to the same memory if T contained an UnsafeCell.*

This was originally prototyped (though never merged) here.

TODO: Is it possible to support T: ?Sized? MaybeUninit<T> requires T: Sized, and in general, unions don't support unsized types, so it's not possible to just manually implement a standin MaybeUninit that does support T: ?Sized.