Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try all frameworks paths regardless of file exist check. #1216

Merged
merged 1 commit into from
Jul 3, 2020

Conversation

dkocher
Copy link
Contributor

@dkocher dkocher commented Jun 28, 2020

New in macOS Big Sur 11 beta, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)

Fix #1215.

Signed-off-by: David Kocher [email protected]

@matthiasblaesing
Copy link
Member

Could you please have a look at the result of the CI build? https://travis-ci.org/github/java-native-access/jna/jobs/702961796

The change introduces unittest failures.

@msm
Copy link

msm commented Jun 29, 2020

Could you please have a look at the result of the CI build? https://travis-ci.org/github/java-native-access/jna/jobs/702961796

The change introduces unittest failures.

(not the OP but took a look at their commit)

It looks like the failing test is essentially using comparing new File(..).exists() logic with NativeLibrary.matchFramework(..)'s behavior:

public void testMatchFramework() {
        if (!Platform.isMac()) {
            return;
        }
        final String[][] MAPPINGS = {
            // Depending on the system, /Library/Frameworks may or may not
            // have anything in it.
            { "QtCore", expected("/Library/Frameworks/QtCore.framework/QtCore") },
            { "Adobe AIR", expected("/Library/Frameworks/Adobe AIR.framework/Adobe AIR") },

            { "QuickTime", expected("/System/Library/Frameworks/QuickTime.framework/QuickTime") },
            { "QuickTime.framework/Versions/Current/QuickTime", expected("/System/Library/Frameworks/QuickTime.framework/Versions/Current/QuickTime") },
        };
        for (int i=0;i < MAPPINGS.length;i++) {
            assertEquals("Wrong framework mapping", MAPPINGS[i][1], NativeLibrary.matchFramework(MAPPINGS[i][0]));
        }
    }

Previously, this made sense since the matchFramework(..) method was designed to look for potential library paths that exist.

By definition, this won't work anymore with this new change's strategy - per the Apple release note, system-provided libraries won't necessarily exist anymore, yet they will be magically available if you dlopen() them.

In other words, unless there is a reason not to try to dlopen(..) paths that may not exist, I think this specific unit test should be removed as the behavior it is trying to test is no longer something we want.

@dbwiddis
Copy link
Contributor

It seems the logic here tries to solve all problems at once, but probably handles more cases than it should.

  • I generally avoid premature optimization, but we're incurring multiple filesystem accesses here without a good reason. If the file is found (which should only happen on 10.x), why do we not just return it there instead of adding to the list and continuing to (unnecessarily) add more to the path?
  • We assume if the file is not found that we must be on macOS 11 and load all the paths. But that's not appropriate if we're on 10.x. We can easily check System.getProperty("os.version") without incurring an (attempted) filesystem read, and react appropriately
    • If we know we're on macOS 10.x why do anything other than what we currently do (find the file and return it)?
    • If we know we're on macOS 11.x why try to find the file, instead of going straight to the new behavior?

Previously, this made sense

By definition, this won't work anymore

It still does make sense and works for 10.x. Switching on os.version can also be applied to the testcase. I'd favor keeping the existing test with a conditional if we're on 10.x, and (if possible/feasible) adding a new appropriate test if we're on 11.x.

@msm
Copy link

msm commented Jun 30, 2020

  • If we know we're on macOS 10.x why do anything other than what we currently do (find the file and return it)?
  • If we know we're on macOS 11.x why try to find the file, instead of going straight to the new behavior?

So, I may be missing something obvious here, but one question: this new behavior on macOS 11.x is for "system-provided" libraries. Previously, we poked around and attempted to find them directly on the file system. Now, they magically come back if you ask for them, presumably by just their name, not with a fake path?

What if, when System.getProperty("os.version") indicates that we are on macOS 11.x, we simply attempt to dlopen() the library name at the end of the chain if we didn't find the library on the filesystem?

E.g., instead of changing the logic to try all framework paths regardless of file presence, instead just do a "blind" dlopen(..) as a latch-ditch attempt if we haven't found anything more specific on the filesystem (with the "old" logic) and are running on macOS 11.x?

Copy link
Contributor

@dbwiddis dbwiddis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more comments on the code, but I think we still have an open discussion on the overall logic.

};
for (int i=0;i < MAPPINGS.length;i++) {
assertEquals("Wrong framework mapping", MAPPINGS[i][1], NativeLibrary.matchFramework(MAPPINGS[i][0]));
if (!System.getProperty("os.version").startsWith("10")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can probably simplify by including this in the previous conditional. Also for the comment rather than "10.11" should say either "macOS 11" or "11.x".

libraryPath = matchFramework(libraryName);
if (libraryPath != null) {
for(String frameworkName : matchFramework(libraryName)) {
libraryName = frameworkName;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm missing why this assignment is needed.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm missing why this assignment is needed.

In fact, this causes a problem for the later attempt to find a library in an embedded jar. By assigning the various attempts to libraryPath (e.g., mutating that variable) we end up trying to find an embedded jar in ${os-prefix}/libraryPath with whatever the last "guess" value was for libraryPath.

Removing that assignment and instead passing frameworkName to the Native.open(frameworkName, openFlags) call fixes this issue by not mangling libraryPath for the later attempt to open from the JAR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But that change was backed out from the final commit. Here's the actual changes.

}
}
return null;
return paths.toArray(new String[paths.size()]);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor point: the [0] version of toArray is slightly more efficient, and we went through and changed them all back in #1060 so best to be consistent here.

}
// Depending on the system, /Library/Frameworks may or may not have anything in it.
assertEquals("Wrong framework mapping", "/Library/Frameworks/QtCore.framework/QtCore",
NativeLibrary.matchFramework("QtCore")[2]);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not understanding why the array index 2 here and the next two entries, but 0 for QuickTime?

};
for (int i=0;i < MAPPINGS.length;i++) {
assertEquals("Wrong framework mapping", MAPPINGS[i][1], NativeLibrary.matchFramework(MAPPINGS[i][0]));
if (!System.getProperty("os.version").startsWith("10")) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: System.getProperty("os.version") currently returns 10.16 for developer beta 1. My understanding is this will eventually become 11.00 (or possibly 11.0?) in a subsequent build, and may already be reporting that on the Apple DTK ARM build. But for the time being this check on just startsWith("10") won't work on DP1

Copy link
Contributor

@dbwiddis dbwiddis Jun 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

System.getProperty("os.version").compareTo("10.15") > 0 should work.

(EDIT: or not, given I'm currently on 10.15.5. Comparing >= 10.16 should work!)

System.getProperty("os.version").compareTo("10.16") >= 0 should work.

@dkocher
Copy link
Contributor Author

dkocher commented Jun 30, 2020

  • If we know we're on macOS 10.x why do anything other than what we currently do (find the file and return it)?
  • If we know we're on macOS 11.x why try to find the file, instead of going straight to the new behavior?

So, I may be missing something obvious here, but one question: this new behavior on macOS 11.x is for "system-provided" libraries. Previously, we poked around and attempted to find them directly on the file system. Now, they magically come back if you ask for them, presumably by just their name, not with a fake path?

What if, when System.getProperty("os.version") indicates that we are on macOS 11.x, we simply attempt to dlopen() the library name at the end of the chain if we didn't find the library on the filesystem?

E.g., instead of changing the logic to try all framework paths regardless of file presence, instead just do a "blind" dlopen(..) as a latch-ditch attempt if we haven't found anything more specific on the filesystem (with the "old" logic) and are running on macOS 11.x?

Probably makes sense to simplify this way and only do the various path concatenation when running on earlier versions. Will do the System.getProperty("os.version").compareTo("10.16") >= 0 which should be future proof.

@dkocher
Copy link
Contributor Author

dkocher commented Jun 30, 2020

Let me know if I should sqash the commits.

@dbwiddis
Copy link
Contributor

Let me know if I should sqash the commits.

Generally before merging, yes. For complex changes, I don't mind seeing individual commits during review so we can see what has changed in each commit.

You can squash when you add a changelog entry. :-)

@dbwiddis
Copy link
Contributor

This looks good to me. I'll leave it for anyone else to comment.

@dbwiddis
Copy link
Contributor

dbwiddis commented Jul 1, 2020

Merged in 65085c1

@msm
Copy link

msm commented Jul 2, 2020

Welp, I may have been trying to be too clever too quickly with my previous suggestion to do a "blind" dlopen(..) as a latch-ditch attempt on macOS 11.

@dkocher were you able to successfully access a native framework library, e.g. Carbon, with this version of code?

dlopen() fails for me if I pass it simply Carbon or libCarbon by itself, on macOS 11 DP1. That works fine from previous versions of macOS, and that is how other popular libraries consume JNA currently.

When I build JNA using your first attempt that went through the previous logic of coming up with possible paths to try to open, but didn't do the file-exists-check, that did work, and allow me to invoke Carbon APIs as expected.

Is my experience different from yours? Are you able to load a system framework with the currently-merged code in JNA?

@dbwiddis dbwiddis reopened this Jul 2, 2020
@dkocher
Copy link
Contributor Author

dkocher commented Jul 2, 2020

@dkocher were you able to successfully access a native framework library, e.g. Carbon, with this version of code?

I mean to have this verified with loading Foundation.framework on macOS 11 DP1. I will double check I have indeed tested the latest revision.

@dkocher
Copy link
Contributor Author

dkocher commented Jul 2, 2020

@dkocher were you able to successfully access a native framework library, e.g. Carbon, with this version of code?

dlopen() fails for me if I pass it simply Carbon or libCarbon by itself, on macOS 11 DP1. That works fine from previous versions of macOS, and that is how other popular libraries consume JNA currently.

I forgot to install the latest build into my local Maven repository and thus tested with a previous build :( I can confirm loading with Foundation or Foundation.framework fails (neither on macOS 10.15 or 11DP1). As I have quoted initially one has to pass the full path to the framework like /System/Library/Frameworks/Foundation.framework/Foundation.

Thus I propose to revert to a variant of my initial patch.

… coverage.

> Check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache.

Signed-off-by: David Kocher <[email protected]>
@dbwiddis
Copy link
Contributor

dbwiddis commented Jul 2, 2020

I'll wait until @msm tests this latest version before merging again. Thanks for updating.

@msm
Copy link

msm commented Jul 3, 2020

I have tested the updated fix for loading frameworks on macOS 11. I am happy to report it is working now for both system frameworks and loading embedded libraries in jars.

@dkocher Thanks for updating!

@dbwiddis Would it be possible to work towards releasing a 5.6 or similar version in the near future? There are a few other projects that have issues reported against them that this would fix:

gmethvin/directory-watcher#52
https://youtrack.jetbrains.com/issue/IDEA-244962?p=JBR-2535

@dbwiddis
Copy link
Contributor

dbwiddis commented Jul 3, 2020

OK, I'll get this merged (again!). Regarding a 5.6.0 release, I suggest starting a thread on the mailing list.

@dbwiddis dbwiddis merged commit e6c6b37 into java-native-access:master Jul 3, 2020
rgm added a commit to rgm/figwheel-main that referenced this pull request Jul 7, 2020
Changes to the latest beta version of macOS broke the mechanism that
we're using to observe filesystem change events. It's more than just
filesystem observation that has broken: any attempt to run a
hot-reloading workflow crashes the Figwheel process with an uncaught
exception at startup. (Note that build-once still works, since it
doesn't try to access filesystem events).

The underlying hawk library does have an implementation of a
less-efficient polling watcher. Figwheel can use this to provide
hot-reloading even when native events aren't available.

This commit wraps the `hawk.core/watch!` function to catch any exception
that's thrown, and re-try with the same options *except* that it
explicitly starts a polling watcher.

See also

- gjoseph/BarbaryWatchService#13
- java-native-access/jna#1216

Part of bhauman#253.
@dbwiddis
Copy link
Contributor

dbwiddis commented Jul 7, 2020

@dkocher , @msm Would you be able to test the 5.6.0 snapshot on the macOS 11 beta? See the mailing list thread for links.

rgm added a commit to rgm/BarbaryWatchService that referenced this pull request Aug 4, 2020
Mac OS 11 (aka. "Big Sur") made some changes to the way that the Carbon
library is located, causing JNA calls to fail. This has been fixed in
JNA 5.6.0; see java-native-access/jna#1216

Since Barbary is implemented on JNA, this is causing downstream projects
that rely on file watching to crash on JVM start when used on Mac OS 11.

Examples:
- bhauman/figwheel-main#253
- thheller/shadow-cljs#767
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Failure loading frameworks on macOS 11
4 participants