std: Deprecate all std::os::*::raw types #31551

alexcrichton · 2016-02-11T01:08:55Z

This commit is an implementation of RFC 1415 which deprecates all types
in the std::os::*::raw modules.

Many of the types in these modules don't actually have a canonical platform
representation, for example the definition of stat on 32-bit Linux will change
depending on whether C code is compiled with LFS support or not. Unfortunately
the current types in std::os::*::raw are billed as "compatible with C", which
in light of this means it isn't really possible.

To make matters worse, platforms like Android sometimes define these types as
smaller than the way they're actually represented in the stat structure
itself. This means that when methods like DirEntry::ino are called on Android
the result may be truncated as we're tied to returning a ino_t type, not the
underlying type.

The commit here incorporates two backwards-compatible components:

Deprecate all raw types that aren't in std::os::raw
Expand the std::os::*::fs::MetadataExt trait on all platforms for method
accessors of all fields. The fields now returned widened types which are the
same across platforms (consistency across platforms is not required, however,
it's just convenient).

and two also backwards-incompatible components:

Change the definition of all std::os::*::raw type aliases to
correspond to the newly widened types that are being returned on each
platform.
Change the definition of std::os::*::raw::stat on Linux to match the LFS
definitions rather than the standard ones.

The breaking changes here will specifically break code that assumes that libc
and std agree on the definition of std::os::*::raw types, or that the std
types are faithful representations of the types in C. An audit has been
performed of crates.io to determine the fallout which was determined two be
minimal, with the two found cases of breakage having been fixed now.

Ok, so after all that, we're finally able to support LFS on Linux! This commit
then simultaneously starts using stat64 and friends on Linux to ensure that we
can open >4GB files on 32-bit Linux. Yay!

Closes #28978
Closes #30050
Closes #31549

rust-highfive · 2016-02-11T01:09:07Z

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

cuviper · 2016-02-11T07:51:13Z

we can open >4GB files

Yay! Although the limit was actually 2GB, since off_t was i32.

This highlights to me that you've made the "universal" type off_t = u64, changing the sign -- is that just so MetadataExt::size() -> u64 can still be assigned to a deprecated off_t?

alexcrichton · 2016-02-11T17:18:51Z

@cuviper oh wow, even better than I thought!

The change for off_t was primarily done for the std::os::unix::fs::MetadataExt trait. Today it returns off_t, and code which looks like:

fn foo(amt: off_t) { ... }

foo(metadata.size());

would break otherwise. We in theory want to return u64 from the size method (both there and across all platforms), so the trait is indeed changing to return u64 and to prevent some breakage the type definition is changing as well.

Does that makes sense? I tried to word the deprecation message to indicate "these type definitions are known to not be correct" to ensure that this was conveyed.

cuviper · 2016-02-11T18:21:53Z

Right, your explanation is basically what I suspected. Makes sense.

My only concern would be if someone also tried to use negative off_t for something, like posix's common -1 for error/invalid results. That may be more subtle - do you think your audit would have revealed this?

alexcrichton · 2016-02-11T18:36:31Z

I think it would have, yeah. I found only a very few locations that use std::os::unix::raw at all, and anything I did find was just a intermediate cast or something that goes directly to a u64 anyway. We'll always have a window during nightly/beta, however, to weed out any other possible breakage, and we'll still have a chance to revert this if it comes to that.

cuviper · 2016-02-11T18:52:17Z

OK. The easy fallback in this case is just i64, which ought to suffice since the platform itself does have to support signed offsets, conflated with file sizes.

brson · 2016-02-11T18:56:17Z

@bors r+

bors · 2016-02-11T18:56:19Z

📌 Commit 9de5ff3 has been approved by brson

bors · 2016-02-13T06:44:23Z

⌛ Testing commit 9de5ff3 with merge d8c3443...

bors · 2016-02-13T06:47:28Z

💔 Test failed - auto-linux-32-opt

alexcrichton · 2016-02-13T09:01:10Z

@bors: r=brson 3a35cbb

bors · 2016-02-13T10:08:07Z

⌛ Testing commit 3a35cbb with merge 4906b4c...

bors · 2016-02-13T10:23:48Z

💔 Test failed - auto-mac-64-opt

alexcrichton · 2016-02-13T18:50:51Z

@bors: r=brson 2991b04

bors · 2016-02-13T18:53:30Z

⌛ Testing commit 2991b04 with merge 564ac80...

bors · 2016-02-13T19:23:35Z

💔 Test failed - auto-linux-64-x-android-t

alexcrichton · 2016-02-13T20:49:23Z

@bors: r=brson ede1e29

bors · 2016-02-13T21:10:05Z

⌛ Testing commit ede1e29 with merge 2905cd9...

bors · 2016-02-13T21:43:27Z

💔 Test failed - auto-mac-ios-opt

This commit is an implementation of [RFC 1415][rfc] which deprecates all types in the `std::os::*::raw` modules. [rfc]: https://github.com/rust-lang/rfcs/blob/master/text/1415-trim-std-os.md Many of the types in these modules don't actually have a canonical platform representation, for example the definition of `stat` on 32-bit Linux will change depending on whether C code is compiled with LFS support or not. Unfortunately the current types in `std::os::*::raw` are billed as "compatible with C", which in light of this means it isn't really possible. To make matters worse, platforms like Android sometimes define these types as *smaller* than the way they're actually represented in the `stat` structure itself. This means that when methods like `DirEntry::ino` are called on Android the result may be truncated as we're tied to returning a `ino_t` type, not the underlying type. The commit here incorporates two backwards-compatible components: * Deprecate all `raw` types that aren't in `std::os::raw` * Expand the `std::os::*::fs::MetadataExt` trait on all platforms for method accessors of all fields. The fields now returned widened types which are the same across platforms (consistency across platforms is not required, however, it's just convenient). and two also backwards-incompatible components: * Change the definition of all `std::os::*::raw` type aliases to correspond to the newly widened types that are being returned on each platform. * Change the definition of `std::os::*::raw::stat` on Linux to match the LFS definitions rather than the standard ones. The breaking changes here will specifically break code that assumes that `libc` and `std` agree on the definition of `std::os::*::raw` types, or that the `std` types are faithful representations of the types in C. An [audit] has been performed of crates.io to determine the fallout which was determined two be minimal, with the two found cases of breakage having been fixed now. [audit]: rust-lang/rfcs#1415 (comment) --- Ok, so after all that, we're finally able to support LFS on Linux! This commit then simultaneously starts using `stat64` and friends on Linux to ensure that we can open >4GB files on 32-bit Linux. Yay! Closes rust-lang#28978 Closes rust-lang#30050 Closes rust-lang#31549

alexcrichton · 2016-02-13T22:43:11Z

@bors: r=brson aa23c98

bors · 2016-02-13T23:30:36Z

⌛ Testing commit aa23c98 with merge 7589f86...

bors · 2016-02-13T23:37:09Z

⛄ The build was interrupted to prioritize another pull request.

bors · 2016-02-14T02:17:38Z

⌛ Testing commit aa23c98 with merge 09395bb...

This commit is an implementation of [RFC 1415][rfc] which deprecates all types in the `std::os::*::raw` modules. [rfc]: https://github.com/rust-lang/rfcs/blob/master/text/1415-trim-std-os.md Many of the types in these modules don't actually have a canonical platform representation, for example the definition of `stat` on 32-bit Linux will change depending on whether C code is compiled with LFS support or not. Unfortunately the current types in `std::os::*::raw` are billed as "compatible with C", which in light of this means it isn't really possible. To make matters worse, platforms like Android sometimes define these types as *smaller* than the way they're actually represented in the `stat` structure itself. This means that when methods like `DirEntry::ino` are called on Android the result may be truncated as we're tied to returning a `ino_t` type, not the underlying type. The commit here incorporates two backwards-compatible components: * Deprecate all `raw` types that aren't in `std::os::raw` * Expand the `std::os::*::fs::MetadataExt` trait on all platforms for method accessors of all fields. The fields now returned widened types which are the same across platforms (consistency across platforms is not required, however, it's just convenient). and two also backwards-incompatible components: * Change the definition of all `std::os::*::raw` type aliases to correspond to the newly widened types that are being returned on each platform. * Change the definition of `std::os::*::raw::stat` on Linux to match the LFS definitions rather than the standard ones. The breaking changes here will specifically break code that assumes that `libc` and `std` agree on the definition of `std::os::*::raw` types, or that the `std` types are faithful representations of the types in C. An [audit] has been performed of crates.io to determine the fallout which was determined two be minimal, with the two found cases of breakage having been fixed now. [audit]: rust-lang/rfcs#1415 (comment) --- Ok, so after all that, we're finally able to support LFS on Linux! This commit then simultaneously starts using `stat64` and friends on Linux to ensure that we can open >4GB files on 32-bit Linux. Yay! Closes #28978 Closes #30050 Closes #31549

bors · 2016-02-14T05:16:08Z

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-mac-ios-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-opt

- Successful merges: #31551, #31581, #31614, #31626, #31632, #31642 - Failed merges:

@alexcrichton

This follows the pattern already used for stat functions from rust-lang#31551. Now `ftruncate`, `lseek`, and `readdir_r` use their explicit 64-bit variants for LFS support, using wider `off_t` and `dirent` types. This also updates to `open64`, which uses no different types but implies the `O_LARGEFILE` flag. Non-Linux platforms just map their normal functions to the 64-bit names. r? @alexcrichton

@alexcrichton

This follows the pattern already used for stat functions from #31551. Now `ftruncate`, `lseek`, and `readdir_r` use their explicit 64-bit variants for LFS support, using wider `off_t` and `dirent` types. This also updates to `open64`, which uses no different types but implies the `O_LARGEFILE` flag. Non-Linux platforms just map their normal functions to the 64-bit names. r? @alexcrichton

Fixes #108. MetadataExt now returns direct numeric types rather than platform-specific ones, so we need to adjust the functions that use these to have the new types. I've just aliased the types to specific ones so the rest of the code remains the same (file.rs is the only place that uses this) The RFC that changed this is here: rust-lang/rust#31551

rust-highfive assigned brson Feb 11, 2016

alexcrichton force-pushed the deprecate-std-os-raw branch from b2a2534 to a97bc1e Compare February 11, 2016 01:10

ogham mentioned this pull request Feb 11, 2016

[2 in 1] datetime/exa issues rust-datetime/datetime#14

Closed

alexcrichton force-pushed the deprecate-std-os-raw branch from a97bc1e to 9de5ff3 Compare February 11, 2016 17:15

brson added the relnotes Marks issues that should be documented in the release notes of the next release. label Feb 11, 2016

alexcrichton force-pushed the deprecate-std-os-raw branch from 9de5ff3 to 3a35cbb Compare February 13, 2016 09:00

alexcrichton force-pushed the deprecate-std-os-raw branch from 3a35cbb to 2991b04 Compare February 13, 2016 18:50

alexcrichton force-pushed the deprecate-std-os-raw branch from 2991b04 to ede1e29 Compare February 13, 2016 20:49

alexcrichton force-pushed the deprecate-std-os-raw branch from ede1e29 to aa23c98 Compare February 13, 2016 22:42

Manishearth mentioned this pull request Feb 14, 2016

Rollup of 6 pull requests #31646

Merged

bors merged commit aa23c98 into rust-lang:master Feb 14, 2016

llogiq mentioned this pull request Feb 14, 2016

TWIR 118's COTW + interesting PRs rust-lang/this-week-in-rust#170

Closed

MagaTailor mentioned this pull request Feb 15, 2016

32-bit file size problem ogham/exa#86

Closed

cuviper mentioned this pull request Feb 15, 2016

std: use more LFS functions on Linux #31668

Merged

alexcrichton deleted the deprecate-std-os-raw branch February 15, 2016 19:05

bors added a commit that referenced this pull request Feb 15, 2016

Auto merge of #31646 - Manishearth:rollup, r=Manishearth

81c7024

- Successful merges: #31551, #31581, #31614, #31626, #31632, #31642 - Failed merges:

bors added a commit that referenced this pull request Feb 15, 2016

Auto merge of #31646 - Manishearth:rollup, r=Manishearth

13675a5

- Successful merges: #31551, #31581, #31614, #31626, #31632, #31642 - Failed merges:

malbarbo mentioned this pull request May 4, 2017

Fix stat definition for android aarch64 #41752

Closed

cuviper mentioned this pull request Jul 27, 2017

Fix some raw types in std to match what's in libc #43499

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

std: Deprecate all std::os::*::raw types #31551

std: Deprecate all std::os::*::raw types #31551

alexcrichton commented Feb 11, 2016

rust-highfive commented Feb 11, 2016

cuviper commented Feb 11, 2016

alexcrichton commented Feb 11, 2016

cuviper commented Feb 11, 2016

alexcrichton commented Feb 11, 2016

cuviper commented Feb 11, 2016

brson commented Feb 11, 2016

bors commented Feb 11, 2016

bors commented Feb 13, 2016

bors commented Feb 13, 2016

alexcrichton commented Feb 13, 2016

bors commented Feb 13, 2016

bors commented Feb 13, 2016

alexcrichton commented Feb 13, 2016

bors commented Feb 13, 2016

bors commented Feb 13, 2016

alexcrichton commented Feb 13, 2016

bors commented Feb 13, 2016

bors commented Feb 13, 2016

alexcrichton commented Feb 13, 2016

bors commented Feb 13, 2016

bors commented Feb 13, 2016

bors commented Feb 14, 2016

bors commented Feb 14, 2016

std: Deprecate all std::os::*::raw types #31551

std: Deprecate all std::os::*::raw types #31551

Conversation

alexcrichton commented Feb 11, 2016

rust-highfive commented Feb 11, 2016

cuviper commented Feb 11, 2016

alexcrichton commented Feb 11, 2016

cuviper commented Feb 11, 2016

alexcrichton commented Feb 11, 2016

cuviper commented Feb 11, 2016

brson commented Feb 11, 2016

bors commented Feb 11, 2016

bors commented Feb 13, 2016

bors commented Feb 13, 2016

alexcrichton commented Feb 13, 2016

bors commented Feb 13, 2016

bors commented Feb 13, 2016

alexcrichton commented Feb 13, 2016

bors commented Feb 13, 2016

bors commented Feb 13, 2016

alexcrichton commented Feb 13, 2016

bors commented Feb 13, 2016

bors commented Feb 13, 2016

alexcrichton commented Feb 13, 2016

bors commented Feb 13, 2016

bors commented Feb 13, 2016

bors commented Feb 14, 2016

bors commented Feb 14, 2016