Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[native-image] Invoking MethodHandle in clojure.lang.Reflector does not work #2214

Closed
borkdude opened this issue Mar 3, 2020 · 5 comments
Closed
Assignees
Labels

Comments

@borkdude
Copy link

borkdude commented Mar 3, 2020

Describe GraalVM and your environment :

  • GraalVM version or commit id if built from source: 20.0.0
  • CE or EE: CE
  • JDK version: java11
  • OS and OS Version: macOS Mojave
  • Architecture: amd64
  • The output of java -Xinternalversion:
java -Xinternalversion
OpenJDK 64-Bit Server VM (11.0.6+9-jvmci-20.0-b02) for bsd-amd64 JRE (11.0.6+9-jvmci-20.0-b02), built on Jan 20 2020 14:32:51 by "graal1" with gcc 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)

Describe the issue

Compiling Clojure code that uses clojure.lang.Reflector does not work with java11 due to the invocation of a MethodHandle that cannot be reduced to a constant at compile time. The code works on java 8 where it does not use the MethodHandle invocation.

Code snippet or code repository that reproduces the issue

reflection.json:

[{
  "name": "java.io.File",
  "allPublicMethods": true
}]

Reflector.java:

import java.lang.invoke.MethodHandle;
import java.lang.invoke.MethodHandles;
import java.lang.invoke.MethodType;
import java.lang.reflect.Constructor;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.lang.reflect.Modifier;
import java.util.*;
import java.util.stream.Collectors;

public class Reflector{

    private static final MethodHandle CAN_ACCESS_PRED;

    // Java 8 is oldest JDK supported
    private static boolean isJava8() {
	return System.getProperty("java.vm.specification.version").equals("1.8");
    }

    static {
	MethodHandle pred = null;
	try {
            if (! isJava8())
                pred = MethodHandles.lookup().findVirtual(Method.class, "canAccess", MethodType.methodType(boolean.class, Object.class));
	} catch (Throwable t) {
            // Util.sneakyThrow(t); // commented out, not important for repro
	}
	CAN_ACCESS_PRED = pred;
    }

    private static boolean canAccess(Method m, Object target) {
	if (CAN_ACCESS_PRED != null) {
            // JDK9+ use j.l.r.AccessibleObject::canAccess, which respects module rules
            try {
                return (boolean) CAN_ACCESS_PRED.invoke(m, target);
            } catch (Throwable t) {
                // throw Util.sneakyThrow(t); // commented out, not important for repro
            }
	} else {
            // JDK 8
            return true;
	}
        return true; // had to add this to make the compiler happy...
    }

    public static void main(String[] args) {
        Method method = java.io.File.class.getMethods()[0];
        boolean hasAccess = canAccess(method, new java.io.File("."));
        System.out.println(System.getProperty("java.version"));
        System.out.println(System.getProperty("java.vm.version"));
        System.out.println("Can access method: " + hasAccess);
    }
}

Steps to reproduce the issue
Please include both build steps as well as run steps

$ $GRAALVM_HOME/bin/javac Reflector.java
$ $GRAALVM_HOME/bin/native-image \
  -H:Name=reflector-test \
  --no-server \
  --no-fallback \
  --initialize-at-build-time \
  -H:+ReportExceptionStackTraces \
  -H:ReflectionConfigurationFiles=reflection.json \
  Reflector

Expected behavior
I'm not sure what should happen yet. Since this issue is likely to come up often when Clojure projects will migrate to java11 builds of GraalVM, I think it's good to discuss the options that either GraalVM or Clojure could offer to solve this issue.

Note: it seems that adding the option --report-unsupported-elements-at-runtime does seem to help with this issue:

$ ./reflector-test
11.0.6
GraalVM 20.0.0 CE
Can access method: true

Full output of compilation

$ $GRAALVM_HOME/bin/native-image \
  -H:Name=reflector-test \
  --no-server \
  --no-fallback \
  --initialize-at-build-time \
  -H:+ReportExceptionStackTraces \
  Reflector
[reflector-test:60867]    classlist:   1,269.64 ms,  1.20 GB
[reflector-test:60867]        (cap):     841.73 ms,  1.20 GB
[reflector-test:60867]        setup:   1,967.33 ms,  1.20 GB
[reflector-test:60867]   (typeflow):   4,149.33 ms,  1.47 GB
[reflector-test:60867]    (objects):   5,129.85 ms,  1.47 GB
[reflector-test:60867]   (features):     165.64 ms,  1.47 GB
[reflector-test:60867]     analysis:   9,609.12 ms,  1.47 GB
Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Invoke with MethodHandle argument could not be reduced to at most a single call or single field access. The method handle must be a compile time constant, e.g., be loaded from a `static final` field. Method that contains the method handle invocation: java.lang.invoke.Invokers$Holder.invoke_MT(Object, Object, Object, Object)
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The error is then reported at run time when the invoke is executed.
Detailed message:
Trace:
	at parsing Reflector.canAccess(Reflector.java:35)
Call path from entry point to Reflector.canAccess(Method, Object):
	at Reflector.canAccess(Reflector.java:32)
	at Reflector.main(Reflector.java:48)
	at com.oracle.svm.core.JavaMainWrapper.runCore(JavaMainWrapper.java:151)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:186)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)

com.oracle.svm.core.util.UserError$UserException: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Invoke with MethodHandle argument could not be reduced to at most a single call or single field access. The method handle must be a compile time constant, e.g., be loaded from a `static final` field. Method that contains the method handle invocation: java.lang.invoke.Invokers$Holder.invoke_MT(Object, Object, Object, Object)
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The error is then reported at run time when the invoke is executed.
Detailed message:
Trace:
	at parsing Reflector.canAccess(Reflector.java:35)
Call path from entry point to Reflector.canAccess(Method, Object):
	at Reflector.canAccess(Reflector.java:32)
	at Reflector.main(Reflector.java:48)
	at com.oracle.svm.core.JavaMainWrapper.runCore(JavaMainWrapper.java:151)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:186)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)

	at com.oracle.svm.core.util.UserError.abort(UserError.java:79)
	at com.oracle.svm.hosted.FallbackFeature.reportAsFallback(FallbackFeature.java:221)
	at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:741)
	at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:530)
	at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:445)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Invoke with MethodHandle argument could not be reduced to at most a single call or single field access. The method handle must be a compile time constant, e.g., be loaded from a `static final` field. Method that contains the method handle invocation: java.lang.invoke.Invokers$Holder.invoke_MT(Object, Object, Object, Object)
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The error is then reported at run time when the invoke is executed.
Detailed message:
Trace:
	at parsing Reflector.canAccess(Reflector.java:35)
Call path from entry point to Reflector.canAccess(Method, Object):
	at Reflector.canAccess(Reflector.java:32)
	at Reflector.main(Reflector.java:48)
	at com.oracle.svm.core.JavaMainWrapper.runCore(JavaMainWrapper.java:151)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:186)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)

	at com.oracle.graal.pointsto.constraints.UnsupportedFeatures.report(UnsupportedFeatures.java:126)
	at com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:738)
	... 8 more
Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Invoke with MethodHandle argument could not be reduced to at most a single call or single field access. The method handle must be a compile time constant, e.g., be loaded from a `static final` field. Method that contains the method handle invocation: java.lang.invoke.Invokers$Holder.invoke_MT(Object, Object, Object, Object)
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The error is then reported at run time when the invoke is executed.
	at com.oracle.svm.hosted.phases.IntrinsifyMethodHandlesInvocationPlugin.reportUnsupportedFeature(IntrinsifyMethodHandlesInvocationPlugin.java:602)
	at com.oracle.svm.hosted.phases.IntrinsifyMethodHandlesInvocationPlugin.processInvokeWithMethodHandle(IntrinsifyMethodHandlesInvocationPlugin.java:500)
	at com.oracle.svm.hosted.phases.IntrinsifyMethodHandlesInvocationPlugin.handleInvoke(IntrinsifyMethodHandlesInvocationPlugin.java:220)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.tryNodePluginForInvocation(BytecodeParser.java:2217)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.appendInvoke(BytecodeParser.java:1901)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genDynamicInvokeHelper(BytecodeParser.java:1790)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genInvokeVirtual(BytecodeParser.java:1738)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genInvokeVirtual(BytecodeParser.java:1723)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBytecode(BytecodeParser.java:5286)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3397)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBlock(BytecodeParser.java:3204)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.build(BytecodeParser.java:1085)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.buildRootMethod(BytecodeParser.java:979)
	at jdk.internal.vm.compiler/org.graalvm.compiler.java.GraphBuilderPhase$Instance.run(GraphBuilderPhase.java:84)
	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.run(Phase.java:49)
	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:197)
	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:42)
	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:38)
	at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:221)
	at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:340)
	at com.oracle.graal.pointsto.flow.MethodTypeFlow.doParse(MethodTypeFlow.java:310)
	at com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureParsed(MethodTypeFlow.java:300)
	at com.oracle.graal.pointsto.flow.MethodTypeFlow.addContext(MethodTypeFlow.java:107)
	at com.oracle.graal.pointsto.flow.StaticInvokeTypeFlow.update(InvokeTypeFlow.java:346)
	at com.oracle.graal.pointsto.BigBang$2.run(BigBang.java:511)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:171)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
	... 5 more
Error: Image build request failed with exit status 1
@borkdude borkdude added the bug label Mar 3, 2020
@eginez
Copy link
Contributor

eginez commented Mar 4, 2020

@borkdude MethodHandles are currently not supported in native-image. This has been documented in https://github.com/oracle/graal/blob/master/substratevm/LIMITATIONS.md#invokedynamic-bytecode-and-method-handles. We are however looking to address this issue in the future

@borkdude
Copy link
Author

borkdude commented Mar 4, 2020

@eginez In the above case using the flag --report-unsupported-elements-at-runtime seems to fix the issue, but I know of Clojure projects for which that flag doesn't fix their issue and the reported error still shows up.

Maybe there could be something like a MethodHandle config where the user can give a hint to what the MethodHandle is going to be at runtime. E.g. when compiling the above with JDK11 the MethodHandle is always going to be canAccess of class Method (this is getting pretty meta).

Clojure uses this to be compatible with both JDK8 and JDK11+. Here is the link to the original code: https://github.com/clojure/clojure/blob/f5403e9c666f3281fdb880cb2c21303c273eed2d/src/jvm/clojure/lang/Reflector.java#L27-L57

@borkdude
Copy link
Author

borkdude commented Mar 9, 2020

After learning about substitutions in issue #2136, I found out that this fixes the clojure.lang.Reflector issue with java11:

import com.oracle.svm.core.annotate.Substitute;
import com.oracle.svm.core.annotate.TargetElement;
import com.oracle.svm.core.annotate.TargetClass;
import com.oracle.svm.core.jdk.JDK11OrLater;

import java.lang.reflect.Method;

public final class ReflectorSubstitutions {
}

@TargetClass(className = "clojure.lang.Reflector")
final class Target_clojure_lang_Reflector {

    @Substitute
    @TargetElement(onlyWith = JDK11OrLater.class)
    private static boolean canAccess(Method m, Object target) {
        // JDK9+ use j.l.r.AccessibleObject::canAccess, which respects module rules
        try {
            return (boolean) m.canAccess(target);
        } catch (Throwable t) {
            // throw Util.sneakyThrow(t);
            return false;
        }
    }
}

@borkdude
Copy link
Author

borkdude commented Mar 9, 2020

Also available as library now: https://github.com/borkdude/clj-reflector-graal-java11-fix

sergey-latacora added a commit to latacora/native-cogaws that referenced this issue Sep 21, 2020
relevant issues:
oracle/graal#2214
oracle/graal#955

full exception (fixed):

{:cognitect.anomalies/category :cognitect.anomalies/fault, :cognitect.aws.client/throwable #error {
 :cause Invoke with MethodHandle argument could not be reduced to at most a single call or single field access. The method handle must be a compile time constant, e.g., be loaded from a `static final` field. Method that contains the method handle invocation: java.lang.invoke.Invokers$Holder.invoke_MT(Object, Object, Object, Object)
 :via
 [{:type com.oracle.svm.core.jdk.UnsupportedFeatureError
   :message Invoke with MethodHandle argument could not be reduced to at most a single call or single field access. The method handle must be a compile time constant, e.g., be loaded from a `static final` field. Method that contains the method handle invocation: java.lang.invoke.Invokers$Holder.invoke_MT(Object, Object, Object, Object)
   :at [com.oracle.svm.core.util.VMError unsupportedFeature VMError.java 86]}]
 :trace
 [[com.oracle.svm.core.util.VMError unsupportedFeature VMError.java 86]
  [clojure.lang.Reflector canAccess Reflector.java 49]
  [clojure.lang.Reflector toAccessibleSuperMethod Reflector.java 84]
  [clojure.lang.Reflector lambda$invokeInstanceMethod$0 Reflector.java 99]
  [java.util.stream.ReferencePipeline$3$1 accept ReferencePipeline.java 195]
  [java.util.ArrayList$ArrayListSpliterator forEachRemaining ArrayList.java 1655]
  [java.util.stream.AbstractPipeline copyInto AbstractPipeline.java 484]
  [java.util.stream.AbstractPipeline wrapAndCopyInto AbstractPipeline.java 474]
  [java.util.stream.ReduceOps$ReduceOp evaluateSequential ReduceOps.java 913]
  [java.util.stream.AbstractPipeline evaluate AbstractPipeline.java 234]
  [java.util.stream.ReferencePipeline collect ReferencePipeline.java 578]
  [clojure.lang.Reflector invokeInstanceMethod Reflector.java 101]
  [cognitect.aws.protocols.rest$serialize_uri$fn__12861 invoke rest.clj 29]
  [clojure.string$replace_by invokeStatic string.clj 69]
  [clojure.string$replace invokeStatic string.clj 106]
  [cognitect.aws.protocols.rest$serialize_uri invokeStatic rest.clj 22]
  [cognitect.aws.protocols.rest$serialize_uri invoke rest.clj 22]
  [clojure.core$update invokeStatic core.clj 6202]
  [cognitect.aws.protocols.rest$build_http_request invokeStatic rest.clj 174]
  [cognitect.aws.protocols.rest_xml$fn__12989 invokeStatic rest_xml.clj 11]
  [cognitect.aws.protocols.rest_xml$fn__12989 invoke rest_xml.clj 11]
  [clojure.lang.MultiFn invoke MultiFn.java 234]
  [cognitect.aws.client$send_request$fn__12660$state_machine__7879__auto____12687$fn__12689$fn__12703 invoke client.clj 99]
  [cognitect.aws.client$send_request$fn__12660$state_machine__7879__auto____12687$fn__12689 invoke client.clj 96]
  [cognitect.aws.client$send_request$fn__12660$state_machine__7879__auto____12687 invoke client.clj 84]
  [clojure.core.async.impl.ioc_macros$run_state_machine invokeStatic ioc_macros.clj 978]
  [clojure.core.async.impl.ioc_macros$run_state_machine_wrapped invokeStatic ioc_macros.clj 980]
  [cognitect.aws.client$send_request$fn__12660 invoke client.clj 84]
  [clojure.lang.AFn run AFn.java 22]
  [java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java 1128]
  [java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java 628]
  [clojure.core.async.impl.concurrent$counted_thread_factory$reify__3362$fn__3363 invoke concurrent.clj 29]
  [clojure.lang.AFn run AFn.java 22]
  [java.lang.Thread run Thread.java 834]
  [com.oracle.svm.core.thread.JavaThreads threadStartRoutine JavaThreads.java 517]
  [com.oracle.svm.core.posix.thread.PosixJavaThreads pthreadStartRoutine PosixJavaThreads.java 192]]}}
@fernando-valdez fernando-valdez self-assigned this Jan 19, 2024
@fernando-valdez
Copy link
Member

This issue has been fixed in later releases. Closing this ticket

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants