Skip to content
This repository has been archived by the owner on Jul 4, 2023. It is now read-only.

Obsolete El Capitan System Integrity Protection (SIP) Instructions #45387

Closed
ilovezfs opened this issue Oct 27, 2015 · 33 comments
Closed

Obsolete El Capitan System Integrity Protection (SIP) Instructions #45387

ilovezfs opened this issue Oct 27, 2015 · 33 comments
Assignees
Labels

Comments

@ilovezfs
Copy link
Contributor

The Homebrew installer (https://raw.githubusercontent.com/Homebrew/install/master/install) currently does the following check:

"abort <<-EOABORT if !File.directory?(HOMEBREW_PREFIX) && macos_version >= "10.11"
OS X 10.11 & above require an additional step to write into #{HOMEBREW_PREFIX}.
Please follow the steps described here and then run the install script again:
https://git.io/vnC7w
EOABORT"

While these directions may have been correct for some range of the El Capitan developer betas, they seem to be incorrect now.

I removed the /usr/local dir, upgraded 10.11.0 -> 10.11.1, and had no trouble running sudo mkdir /usr/local && sudo chflags norestricted /usr/local && sudo chown -R $(whoami):admin /usr/local

There was no need to disable SIP.

@MikeMcQuaid
Copy link
Member

CC @DomT4 who would be interested here.

@DomT4
Copy link
Member

DomT4 commented Oct 27, 2015

As far as I know Apple didn't change this, but they may have snuck in the change in the way Apple is prone to.

@MikeMcQuaid
Copy link
Member

@DomT4 Let's investigate. Would be awesome if we can remove these scary warnings.

@mchccc
Copy link

mchccc commented Oct 28, 2015

This "serious bug" from a couple of weeks ago seems to state otherwise, and then some:
https://openradar.appspot.com/23093676

@ilovezfs
Copy link
Contributor Author

@mikedld @DomT4 OK, I investigated further.

I believe this is going to come down to the ontogeny of the particular system.

The difference turns out to be in what I will call the "compatibility paths file."

/System/Library/Sandbox/Compatibility.bundle/Contents/Resources/paths

It is insufficient for /usr/local to be whitelisted only in /System/Library/Sanbox/rootless.conf. It must also be whitelisted in the compatibility paths file. Whether that should or should not be necessary (i.e., whether this should be considered a more general bug with the functioning of rootless.conf) is unclear, but that is the difference.

A clean install of 10.11 (15A284) has /usr/local in the compatibility paths file (version 12 of the bundle according to its Info.plist), and still has /usr/local in this file after the upgrade to 10.11.1. By deliberately going in and removing /usr/local from the compatibility paths file, I can "infect" an otherwise healthy system with the "bug."

A Developer Beta 8 VM that was upgraded stepwise from Developer Beta 1 (15A178w) -> Developer Beta 2 (15A204h) -> Developer Beta 3 (15A216g) -> Developer Beta 4 (15A226f) -> Developer Beta 5 (15A235d) -> Developer Beta 8 (15A279b) does not have /usr/local in Compatibility.bundle/Contents/Resources/paths (version 10 of the bundle according to its Info.plist) and exhibits the symptoms of the bug.

One option would be to actually check for "/usr/local" in the compatibility paths file and only issue the "scary warnings" if it's missing, since that is not the normal state of affairs on a clean install. I have not checked a 10.10.5->10.11.0 or 10.10.5->10.11.1 upgrade path to see if they are affected.

You can tell if you're affected by running

grep 'usr/local' /System/Library/Sandbox/Compatibility.bundle/Contents/Resources/paths

If you don't see /usr/local, you should add it or wait for Apple to add it for you the next time they upgrade the whole Sandbox or at least the Compatibility.bundle within it. Or perhaps they will remove it from the compatibility paths file altogether if they decide to change how SIP handles non-existent directories whitelisted in rootless.conf but not in the compatibility paths.

It is good news that version 12 of the bundle has /usr/local and version 10 does not in terms of the intent that it signals on Apple's part with respect to /usr/local.

@mikedld
Copy link
Contributor

mikedld commented Oct 28, 2015

@ilovezfs wrong person highlight ;) Thanks for keeping me in the loop :))

@ilovezfs
Copy link
Contributor Author

@mikedld hehe Sorry about that.
@MikeMcQuaid

@ilovezfs
Copy link
Contributor Author

@MikeMcQuaid @DomT4 @cicuz

The scary message can indeed go bye-bye or at least be revised.

The mystery was solved by https://swscan.apple.com/content/catalogs/others/index-10.11-1.sucatalog

http://swcdn.apple.com/content/downloads/53/38/031-40358/y7sn2l7yvfxb73myuiufqin8fp2sqyetvl/SystemIntegrityProtectionConfig.pkg installs com.apple.pkg.SystemIntegrityProtectionConfig.14U2076, which is Compatibility.bundle version 12.

Neither version 10 nor 11 contained /usr/local in /System/Library/Sandbox/Compatibility.bundle/Contents/Resources/paths whereas 12 does.

So users whose /System/Library/Sandbox/Compatibility.bundle/Contents/Resources/paths lacks /usr/local (or, equivalently, whose Compatibility.bundle version is less than 12) should be directed to install that package (or later). This probably happens automatically if you have "Install system data files and security updates" enabled in your App Store preferences.

/usr/bin/curl -O http://swcdn.apple.com/content/downloads/53/38/031-40358/y7sn2l7yvfxb73myuiufqin8fp2sqyetvl/SystemIntegrityProtectionConfig.pkg

Here's the relevant catalog entry:

            <key>031-40358</key>
            <dict>
                <key>ServerMetadataURL</key>
                <string>http://swcdn.apple.com/content/downloads/53/38/031-40358/y7sn2l7yvfxb73myuiufqin8fp2sqyetvl/SystemIntegrityProtectionConfig.smd</string>
                <key>Packages</key>
                <array>
                    <dict>
                        <key>Digest</key>
                        <string>7ea4c74931632d0a874cf046c1d5444c160acb5b</string>
                        <key>MetadataURL</key>
                        <string>https://swdist.apple.com/content/downloads/53/38/031-40358/y7sn2l7yvfxb73myuiufqin8fp2sqyetvl/SystemIntegrityProtectionConfig.pkm</string>
                        <key>URL</key>
                        <string>http://swcdn.apple.com/content/downloads/53/38/031-40358/y7sn2l7yvfxb73myuiufqin8fp2sqyetvl/SystemIntegrityProtectionConfig.pkg</string>
                        <key>Size</key>
                        <integer>11628</integer>
                    </dict>
                </array>
                <key>PostDate</key>
                <date>2015-10-21T17:33:00Z</date>
                <key>Distributions</key>
                <dict>
                    <key>English</key>
                    <string>https://swdist.apple.com/content/downloads/53/38/031-40358/y7sn2l7yvfxb73myuiufqin8fp2sqyetvl/031-40358.English.dist</string>
                </dict>
            </dict>

@ilovezfs
Copy link
Contributor Author

Confirmed with a user who still had version Compatibility.bundle version 11. He had "Install system data files and security updates" unchecked.

@DomT4
Copy link
Member

DomT4 commented Oct 28, 2015

I'll reply here later. Travelling today; not ignoring you intentionally. Thanks for doing some research on the changes!

Sent from my iPhone

On 28 Oct 2015, at 16:02, ilovezfs [email protected] wrote:

@MikeMcQuaid @DomT4 @cicuz

The scary message can indeed go bye-bye or at least be revised.

The mystery was solved by https://swscan.apple.com/content/catalogs/others/index-10.11-1.sucatalog

http://swcdn.apple.com/content/downloads/53/38/031-40358/y7sn2l7yvfxb73myuiufqin8fp2sqyetvl/SystemIntegrityProtectionConfig.pkg installs com.apple.pkg.SystemIntegrityProtectionConfig.14U2076, which is Compatibility.bundle version 12.

Neither version 10 nor 11 contained /usr/local in /System/Library/Sandbox/Compatibility.bundle/Contents/Resources/paths whereas 12 does.

So users whose /System/Library/Sandbox/Compatibility.bundle/Contents/Resources/paths lacks /usr/local (or, equivalently, whose Compatibility.bundle version is less than 12) should be directed to install that package (or later). This probably happens automatically if you have "Install system data files and security updates" enabled in your App Store preferences.

/usr/bin/curl -O http://swcdn.apple.com/content/downloads/53/38/031-40358/y7sn2l7yvfxb73myuiufqin8fp2sqyetvl/SystemIntegrityProtectionConfig.pkg
Here's the relevant catalog entry:

        <key>031-40358</key>
        <dict>
            <key>ServerMetadataURL</key>
            <string>http://swcdn.apple.com/content/downloads/53/38/031-40358/y7sn2l7yvfxb73myuiufqin8fp2sqyetvl/SystemIntegrityProtectionConfig.smd</string>
            <key>Packages</key>
            <array>
                <dict>
                    <key>Digest</key>
                    <string>7ea4c74931632d0a874cf046c1d5444c160acb5b</string>
                    <key>MetadataURL</key>
                    <string>https://swdist.apple.com/content/downloads/53/38/031-40358/y7sn2l7yvfxb73myuiufqin8fp2sqyetvl/SystemIntegrityProtectionConfig.pkm</string>
                    <key>URL</key>
                    <string>http://swcdn.apple.com/content/downloads/53/38/031-40358/y7sn2l7yvfxb73myuiufqin8fp2sqyetvl/SystemIntegrityProtectionConfig.pkg</string>
                    <key>Size</key>
                    <integer>11628</integer>
                </dict>
            </array>
            <key>PostDate</key>
            <date>2015-10-21T17:33:00Z</date>
            <key>Distributions</key>
            <dict>
                <key>English</key>
                <string>https://swdist.apple.com/content/downloads/53/38/031-40358/y7sn2l7yvfxb73myuiufqin8fp2sqyetvl/031-40358.English.dist</string>
            </dict>
        </dict>


Reply to this email directly or view it on GitHub.

@ilovezfs
Copy link
Contributor Author

@DomT4 Didn't imagine you were, and you're welcome!

@MikeMcQuaid
Copy link
Member

Yeh thanks for this @ilovezfs, we can definitely tweak some of the checks (perhaps inspect this actual file)

@DomT4 DomT4 added the 10.11 label Oct 29, 2015
@DomT4 DomT4 self-assigned this Oct 29, 2015
@stovak
Copy link

stovak commented Oct 29, 2015

One thing to note... Java now installs as an internet plugin and the java_home is INSIDE THE PLUGIN PACKAGE. If you have something that needs java (e.g. selenium), you're going to have to change the invocation from /usr/bin/java to /Library/Internet Plug-ins/JavaAppletPlugin.plugin/Contents/Home/bin/java

@DomT4
Copy link
Member

DomT4 commented Oct 29, 2015

We work out the Java in a fairly special way rather than necessarily just poking at /usr/bin/java.

@stovak
Copy link

stovak commented Oct 29, 2015

https://github.com/Homebrew/homebrew/blob/master/Library/Formula/selenium-server-standalone.rb#L29

Selenium doesn't use this to determine the path to use in the launcher.

  1. install java
  2. try to launch selenium, get a dialogue that java is not installed
  3. goto step "1"

@DomT4
Copy link
Member

DomT4 commented Oct 29, 2015

You mean the plist? That could be an issue. Is it still the case regardless of Java, i.e. if you install Oracle's Java SE or SDK?

@stovak
Copy link

stovak commented Oct 29, 2015

Yes, the plist. I installed Oracle's basic Java SE. Not sure about the SDK.

@DomT4
Copy link
Member

DomT4 commented Oct 29, 2015

Interesting. Guess I'll have to find some time to spin up yet another clean El Cap VM and see what's going on.

@stovak
Copy link

stovak commented Oct 29, 2015

Also, installing the SE edition JAVA_HOME was not set for the command line so your solution would not work unless i'm missing something, which I very well may be.

@CamJN
Copy link
Contributor

CamJN commented Oct 30, 2015

Just an FYI: I have "Install system data files and security updates" enabled in my App Store preferences, but that update does not show up. My system's on 10.11.1 (15B42)

@ilovezfs
Copy link
Contributor Author

@CamJN How about the remainder of the App Store preferences? Automatically check for updates and automatically download updates? Probably not needed, but worth noting their values.

I'm not sure what schedule "Install system data files and security updates" runs on, or if it's a push-based model. Perhaps someone else knows off the top of their head.

@CamJN
Copy link
Contributor

CamJN commented Oct 30, 2015

Everything is on, and I used softwareupdate on the cli to check manually. I also reset my update server just in case. That update just isn't showing up.

@ilovezfs
Copy link
Contributor Author

"but that update does not show up" ... where are you looking? Did you check your packages database itself? I don't think this one would necessarily show in the App Store itself.

bash-3.2$ pkgutil --pkgs | grep Integrity
com.apple.pkg.SystemIntegrityProtectionConfig.14U2076
bash-3.2$ pkgutil --pkg-info com.apple.pkg.SystemIntegrityProtectionConfig.14U2076
package-id: com.apple.pkg.SystemIntegrityProtectionConfig.14U2076
version: 1.0.0.0.1.1444769888
volume: /
location: /
install-time: 1445476605
bash-3.2$ 

1445476605 is 1:16:45 am UTC | Thursday, October 22, 2015

@JMoVS
Copy link
Contributor

JMoVS commented Oct 30, 2015

running

--pkgs | grep Integrity

results in

com.apple.pkg.SystemIntegrityProtectionConfig.14U2076

on 10.11.1 which never saw a dev build. Just used csrutil with the kext-dev, but that's it

@ilovezfs
Copy link
Contributor Author

I'm back to wondering if there's an issue specific to machines that were/are configured to use the beta and/or dev program seeds. @CamJN was/is the machine ever set up that way?

@CamJN
Copy link
Contributor

CamJN commented Oct 30, 2015

pkgutil --pkgs | grep Integrity is blank. I never used this box for the beta or dev programs.

@DomT4
Copy link
Member

DomT4 commented Oct 30, 2015

I'm back to wondering if there's an issue specific to machines that were/are configured to use the beta and/or dev program seeds.

I run the betas. Have both:

com.apple.pkg.SystemIntegrityProtectionConfig.14U2068
com.apple.pkg.SystemIntegrityProtectionConfig.14U2076

@ilovezfs
Copy link
Contributor Author

@CamJN Weird.

"Everything is on, and I used softwareupdate on the cli to check manually. I also reset my update server just in case. That update just isn't showing up."

I wonder if others who don't have the update are running their own update servers.

@CamJN
Copy link
Contributor

CamJN commented Oct 30, 2015

I'm not running my own update server, however I reset the value just in case b/c I couldn't find a way to check what it was set to.

@ilovezfs
Copy link
Contributor Author

@CamJN What's the output of

 grep Integrity /var/log/install.log 

@CamJN
Copy link
Contributor

CamJN commented Oct 31, 2015

Oct  1 18:09:37 MacBook-Pro OSInstaller[443]: System Defense: Hole-punching System Integrity Protection due to Migration Data override for /System/Library/PrivateFrameworks/iTunesAccess.framework
Oct  1 18:09:37 MacBook-Pro OSInstaller[443]: System Defense: Hole-punching System Integrity Protection due to Migration Data override for /System/Library/PrivateFrameworks/CoreFP.framework
Oct  1 18:09:37 MacBook-Pro OSInstaller[443]: System Defense: Hole-punching System Integrity Protection due to Migration Data override for /System/Library/PrivateFrameworks/CoreADI.framework
Oct  1 18:09:37 MacBook-Pro OSInstaller[443]: System Defense: Hole-punching System Integrity Protection due to Migration Data override for /System/Library/PrivateFrameworks/AirTrafficHost.framework
Oct  1 18:09:37 MacBook-Pro OSInstaller[443]: System Defense: Hole-punching System Integrity Protection due to Migration Data override for /System/Library/PrivateFrameworks/MobileDevice.framework
Oct  1 18:09:37 MacBook-Pro OSInstaller[443]: System Defense: Hole-punching System Integrity Protection due to Migration Data override for /System/Library/PrivateFrameworks/DeviceLink.framework
Oct  1 18:09:37 MacBook-Pro OSInstaller[443]: System Defense: Hole-punching System Defense due to System Integrity Protection override for /System/Library/Extensions/JMicronATA.kext
Oct  1 18:09:37 MacBook-Pro OSInstaller[443]: System Defense: Hole-punching System Defense due to System Integrity Protection override for /System/Library/Speech
Oct  1 18:09:37 MacBook-Pro OSInstaller[443]: System Defense: Hole-punching System Integrity Protection due to Migration Data override for /System/Library/DirectoryServices
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
Oct 22 15:58:27 SECUR-T DownloadService[14999]: Finished downloading package SystemIntegrityProtectionConfig.pkg to file:///var/folders/zz/zyxvpxvq6csfxvn_n00000s0000068/C/com.apple.SoftwareUpdate/CFNetworkDownload_7TJUbp.tmp (error (null)) from peer: (null)
    031-40358    | System Integrity Protection Configuration Data 12.0
Oct 23 22:32:41 SECUR-T softwareupdated[1172]: SoftwareUpdate: Added foreground transaction [0x2] for System Integrity Protection Configuration Data-12.0
Oct 23 22:32:41 SECUR-T com.apple.SoftwareUpdateConfigData[33048]: SoftwareUpdateConfigData: 031-40358 failed install (Error Domain=SUErrorDomain Code=301 "The update has been deleted since being downloaded from the Apple Software Update server." UserInfo={SUErrorUpdateTitle=System Integrity Protection Configuration Data, SUErrorUpdateProductKey=031-40358, NSLocalizedDescription=The update has been deleted since being downloaded from the Apple Software Update server.})
Oct 23 22:32:41 SECUR-T com.apple.SoftwareUpdateConfigData[33048]: SoftwareUpdateConfigData: failed with Error Domain=SUErrorDomain Code=301 "The update has been deleted since being downloaded from the Apple Software Update server." UserInfo={SUErrorUpdateTitle=System Integrity Protection Configuration Data, SUErrorUpdateProductKey=031-40358, NSLocalizedDescription=The update has been deleted since being downloaded from the Apple Software Update server.}
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
Oct 28 16:17:02 SECUR-T softwareupdated[1143]: SoftwareUpdate: Added foreground transaction [0x2] for System Integrity Protection Configuration Data-12.0
Oct 28 16:17:02 SECUR-T com.apple.SoftwareUpdateConfigData[22670]: SoftwareUpdateConfigData: 031-40358 failed install (Error Domain=SUErrorDomain Code=301 "The update has been deleted since being downloaded from the Apple Software Update server." UserInfo={SUErrorUpdateTitle=System Integrity Protection Configuration Data, SUErrorUpdateProductKey=031-40358, NSLocalizedDescription=The update has been deleted since being downloaded from the Apple Software Update server.})
Oct 28 16:17:02 SECUR-T com.apple.SoftwareUpdateConfigData[22670]: SoftwareUpdateConfigData: failed with Error Domain=SUErrorDomain Code=301 "The update has been deleted since being downloaded from the Apple Software Update server." UserInfo={SUErrorUpdateTitle=System Integrity Protection Configuration Data, SUErrorUpdateProductKey=031-40358, NSLocalizedDescription=The update has been deleted since being downloaded from the Apple Software Update server.}
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0
    031-40358    | System Integrity Protection Configuration Data 12.0

So it looks like the install can fail and OS X won't retry it. That's not great...

@ilovezfs
Copy link
Contributor Author

It looks like it tried twice. Once on 10/23 and once on 10/28. So maybe it tries every 5 days or something like that. The question is why did it fail with the same error both times: "The update has been deleted since being downloaded from the Apple Software Update server."

It's possible more of /var/log/install.log from 10/28 could shed some light, if you don't already know what the reason could be.

grep 'Oct 28' /var/log/install.log

You should probably paste output into a gist or pastebin.

@CamJN
Copy link
Contributor

CamJN commented Oct 31, 2015

https://gist.github.com/CamJN/ac9d784091adb9918a77#file-oct-28_install-log

Looks like the download didn't finish and therefore got deleted.

MikeMcQuaid added a commit to Homebrew/install that referenced this issue Nov 1, 2015
These seem to not apply for everyone on 10.11.0 any more (as explained
in Homebrew/legacy-homebrew#45387).
MikeMcQuaid added a commit to Homebrew/install that referenced this issue Nov 1, 2015
These seem to not apply for everyone on 10.11.0 any more (as explained
in Homebrew/legacy-homebrew#45387).
@Homebrew Homebrew locked and limited conversation to collaborators Jul 10, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

8 participants