Support both GraalVM 19.2.1 and 19.3.1 #6574

gwenneg · 2020-01-16T01:14:43Z

gwenneg · 2020-01-16T01:20:38Z

JNI is currently enabled no matter which GraalVM version is used. Should we try to disable it when possible? I don't know if this can have an impact on the native image size with 19.2.1.

I'd like to run some tests again before this is merged, I didn't have enough time to test everything I wanted tonight.

gwenneg · 2020-01-16T01:39:25Z

core/deployment/src/main/java/io/quarkus/deployment/steps/NativeImageAutoFeatureStep.java

+            // The following line is required to throw an exception and make sure we load the 19.2.1 class when needed.
+            workaroundTryBlock.invokeVirtualMethod(
+                    ofMethod(Class.class, "getDeclaredMethod", Method.class, String.class, Class[].class), locClass,
+                    workaroundTryBlock.load("addBundleToCache"),
+                    workaroundTryBlock.marshalAsArray(Class.class, workaroundTryBlock.loadClass(String.class)));


I tried to find a simpler expression to force the exception throw here, but I didn't find anything that worked. I'm sure this expression could be much shorter...

I think it's just fine. You are just trying to make sure the method exists right? It should be perfectly OK and in any case it should be temporary

Actually, I only need to check if the com.oracle.svm.core.jdk.LocalizationFeature class is available, I'm not really interested in the method itself. But invoking a simple getName from LocalizationFeature wasn't enough to force an exception throw. That's not really a problem since this code is only a copy of the original code a few lines below and it is also temporary, but that's still not pretty :)

geoand · 2020-01-16T07:19:41Z

Something seems to be wrong with CI:

2020-01-16T07:06:26.4612036Z [ERROR] 'dependencies.dependency.version' for org.graalvm.nativeimage:svm:jar is missing. @ line 25, column 21

gsmet · 2020-01-16T07:42:44Z

It would be nice to force JNI only for 19.3. We don't need it to be as pretty as the patch you worked on.

Something like: if graalVMVersion.isPresent() && !graalVMVersion.contains(" 19.2.") enableJni = true; in NativeImageBuildStep.

We can refine with your other patch once we only support GraalVM 19.3.

geoand · 2020-01-16T07:46:50Z

It would be nice to force JNI only for 19.3. We don't need it to be as pretty as the patch you worked on.

Something like: if graalVMVersion.isPresent() && !graalVMVersion.contains(" 19.2.") enableJni = true; in NativeImageBuildStep.

We can refine with your other patch once we only support GraalVM 19.3.

It also means that the default value in NativeConfig needs to be false

gwenneg · 2020-01-16T08:04:39Z

Something seems to be wrong with CI:

2020-01-16T07:06:26.4612036Z [ERROR] 'dependencies.dependency.version' for org.graalvm.nativeimage:svm:jar is missing. @ line 25, column 21

I've seen that, but that's a weird issue, I don't know where it is coming from. There isn't a single org.graalvm.nativeimage left in any of the Quarkus pom.xml...

geoand · 2020-01-16T10:17:25Z

Yeah it is weird, especially given the fact that the PR built for me locally...

gwenneg · 2020-01-16T10:20:44Z

Thanks for checking! Everything is working fine on my computer too.

I'll update the PR with the JNI changes in a couple hours, we'll see if CI is happier then.

geoand · 2020-01-16T10:21:09Z

Great, thanks a lot!

gwenneg · 2020-01-16T13:30:32Z

JNI is now disabled by default with GraalVM 19.2.1 and enabled by default with GraalVM 19.3.1. All the enableJni build parameters that were removed from the integration tests pom.xml files in #5358 have been restored in this PR. I will update #6306 to remove them again once this PR is merged.

CI is still unhappy with the PR... @gsmet have you already seen the following kind of error before?

[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] 'dependencies.dependency.version' for org.graalvm.nativeimage:svm:jar is missing. @ line 25, column 21
 @ 
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project io.quarkus:quarkus-narayana-stm:999-SNAPSHOT (D:\a\1\s\extensions\narayana-stm\runtime\pom.xml) has 1 error
[ERROR]     'dependencies.dependency.version' for org.graalvm.nativeimage:svm:jar is missing. @ line 25, column 21
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException

There's absolutely no trace of org.graalvm.nativeimage in the pom.xml mentioned in the error message...

gsmet · 2020-01-16T14:31:43Z

You have to rebase and fix the artifact in the narayana-stm extension. I did it but I can't force push to your branch so you'll have to do it yourself :).

CI applies your PR on top of master.

gsmet

I added a few comments.

The last commit need to be squashed into the one already changing the artifacts.

core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildStep.java

integration-tests/jpa-without-entity/pom.xml

integration-tests/vertx-http/pom.xml

extensions/jdbc/jdbc-h2/runtime/src/main/java/io/quarkus/jdbc/h2/runtime/graal/Engine.java

gwenneg · 2020-01-17T06:55:30Z

There's a small issue visible in #6594 that needs to be fixed before this PR is merged:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GraalVM Version 19.3.1 CE
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] docker run -v /home/vsts/work/1/s/integration-tests/artemis-core/target/quarkus-integration-test-artemis-core-999-SNAPSHOT-native-image-source-jar:/project:z --user 1001:117 --rm quay.io/quarkus/ubi-quarkus-native-image:19.3.1-java8 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar quarkus-integration-test-artemis-core-999-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -J-Xmx6g -H:+AddAllCharsets -H:EnableURLProtocols=http -H:+JNI --no-server -H:-UseServiceLoaderFeature -H:+StackTrace quarkus-integration-test-artemis-core-999-SNAPSHOT-runner
[quarkus-integration-test-artemis-core-999-SNAPSHOT-runner:19]    classlist:  15,374.36 ms
[quarkus-integration-test-artemis-core-999-SNAPSHOT-runner:19]        (cap):   1,375.41 ms
[quarkus-integration-test-artemis-core-999-SNAPSHOT-runner:19]        setup:   3,503.42 ms
06:32:24,524 INFO  [org.jbo.threads] JBoss Threads version 3.0.0.Final
[quarkus-integration-test-artemis-core-999-SNAPSHOT-runner:19]     analysis:  52,563.75 ms
Fatal error: com.oracle.graal.pointsto.util.AnalysisError$ParsingError: Error encountered while parsing com.oracle.svm.reflect.ClassLoader_defineClass_c6c343b4d6dc22ca64eb2d8503b13ac9c340dcb3.invoke(java.lang.Object, java.lang.Object[]) 
Parsing context:
	parsing java.lang.reflect.Method.invoke(Method.java:498)
	parsing javax.enterprise.util.AnnotationLiteral.invoke(AnnotationLiteral.java:288)
	parsing javax.enterprise.util.AnnotationLiteral.getMemberValue(AnnotationLiteral.java:276)
	parsing javax.enterprise.util.AnnotationLiteral.hashCode(AnnotationLiteral.java:246)
	parsing java.util.concurrent.CopyOnWriteArrayList.hashCode(CopyOnWriteArrayList.java:1062)
	parsing java.util.AbstractList.hashCode(AbstractList.java:541)
	parsing org.graalvm.collections.EconomicMapImpl.getHashIndex(EconomicMapImpl.java:414)
	parsing org.graalvm.collections.EconomicMapImpl.putHashEntry(EconomicMapImpl.java:577)
	parsing org.graalvm.collections.EconomicMapImpl.put(EconomicMapImpl.java:451)
	parsing org.graalvm.collections.EconomicMap.putAll(EconomicMap.java:84)
	parsing org.graalvm.collections.EconomicMapImpl.<init>(EconomicMapImpl.java:183)
	parsing org.graalvm.collections.EconomicMapImpl.create(EconomicMapImpl.java:158)
	parsing org.graalvm.collections.EconomicMap.create(EconomicMap.java:171)
	parsing org.graalvm.compiler.options.ModifiableOptionValues.update(ModifiableOptionValues.java:93)
	parsing com.oracle.svm.core.option.RuntimeOptionParser.parse(RuntimeOptionParser.java:164)
	parsing com.oracle.svm.core.option.RuntimeOptionParser.parseAndConsumeAllOptions(RuntimeOptionParser.java:77)
	parsing com.oracle.svm.core.JavaMainWrapper.runCore(JavaMainWrapper.java:132)
	parsing com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:186)
	parsing com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)

	at com.oracle.graal.pointsto.util.AnalysisError.parsingError(AnalysisError.java:138)
	at com.oracle.graal.pointsto.flow.MethodTypeFlow.doParse(MethodTypeFlow.java:327)
	at com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureParsed(MethodTypeFlow.java:300)
	at com.oracle.graal.pointsto.flow.MethodTypeFlow.addContext(MethodTypeFlow.java:107)
	at com.oracle.graal.pointsto.DefaultAnalysisPolicy$DefaultVirtualInvokeTypeFlow.onObservedUpdate(DefaultAnalysisPolicy.java:191)
	at com.oracle.graal.pointsto.flow.TypeFlow.notifyObservers(TypeFlow.java:343)
	at com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:385)
	at com.oracle.graal.pointsto.BigBang$2.run(BigBang.java:511)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:171)
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: com.oracle.svm.hosted.substitute.DeletedElementException: Unsupported method java.lang.ClassLoader.defineClass(String, byte[], int, int) is reachable: The declaring class of this element has been substituted, but this element is not present in the substitution class
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The unsupported element is then reported at run time when it is accessed the first time.
	at com.oracle.svm.hosted.substitute.AnnotationSubstitutionProcessor.lookup(AnnotationSubstitutionProcessor.java:183)
	at com.oracle.graal.pointsto.infrastructure.SubstitutionProcessor$ChainedSubstitutionProcessor.lookup(SubstitutionProcessor.java:128)
	at com.oracle.graal.pointsto.infrastructure.SubstitutionProcessor$ChainedSubstitutionProcessor.lookup(SubstitutionProcessor.java:128)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookupAllowUnresolved(AnalysisUniverse.java:397)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookup(AnalysisUniverse.java:377)
	at com.oracle.graal.pointsto.meta.AnalysisUniverse.lookup(AnalysisUniverse.java:75)
	at com.oracle.graal.pointsto.infrastructure.UniverseMetaAccess.lookupJavaMethod(UniverseMetaAccess.java:93)
	at com.oracle.graal.pointsto.meta.AnalysisMetaAccess.lookupJavaMethod(AnalysisMetaAccess.java:66)
	at com.oracle.graal.pointsto.meta.AnalysisMetaAccess.lookupJavaMethod(AnalysisMetaAccess.java:39)
	at com.oracle.svm.reflect.hosted.ReflectionSubstitutionType$ReflectiveInvokeMethod.buildGraph(ReflectionSubstitutionType.java:511)
	at com.oracle.graal.pointsto.meta.AnalysisMethod.buildGraph(AnalysisMethod.java:319)
	at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:185)
	at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:340)
	at com.oracle.graal.pointsto.flow.MethodTypeFlow.doParse(MethodTypeFlow.java:310)
	... 12 more

geoand · 2020-01-17T06:59:32Z

As discussed with @gsmet, once this is in we should setup some simple GitHub Actions pipeline to ensure that things work properly for both GraalVM versions

geoand · 2020-01-17T07:08:35Z

@gwenneg that seems similar to what #6597 fixes. Mind giving it a try?

gwenneg · 2020-01-17T07:14:53Z

That's the same one indeed. Both this PR and #6594 are editable, feel free to rebase both on master once #6597 is merged if you want quick results. I won't be able to push anything before the end of the day (which just started for me).

gsmet · 2020-01-17T07:26:56Z

Let's merge that one and make progress from there. Additional 19.3.1 issues can be fixed later, they are independent from this PR.

geoand · 2020-01-17T07:40:53Z

@gwenneg will do, thanks

ozkanpakdil · 2020-01-22T07:43:00Z

Sorry to bother. I tried to use 1.2.0.CR1. I have small maven project which uses quarkus. and I just wanted to compile it with graalvm 19.3 but it did not work
<properties> <maven.compiler.parameters>true</maven.compiler.parameters> <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> <compiler-plugin.version>3.8.1</compiler-plugin.version> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <quarkus-plugin.version>1.1.1.Final</quarkus-plugin.version> <quarkus.platform.artifact-id>quarkus-universe-bom</quarkus.platform.artifact-id> <quarkus.platform.group-id>io.quarkus</quarkus.platform.group-id> <quarkus.platform.version>1.2.0.CR1</quarkus.platform.version> <surefire-plugin.version>2.22.2</surefire-plugin.version> </properties>

I have changed quarkus.platform.version and ran mvn install just to see if build works but failed. with
[ERROR] The project com.mascix:quarkus-command-runner:1.0-SNAPSHOT (D:\Projects\quarkus-command-runner\pom.xml) has 7 errors [ERROR] Non-resolvable import POM: Could not find artifact io.quarkus:quarkus-universe-bom:pom:1.2.0.CR1 in central (https://repo.maven.apache.org/maven2) @ line 23, column 25 -> [Help 2]

if you dont mind can you tell me how to test new CRs ? or better just point me to an example repo where you test CR1 :) I can copy from there

geoand · 2020-01-22T07:44:46Z

The quarkus-universe-bom hasn't been released yet. However you can replace it with quarkus-bom and everything should be OK

ozkanpakdil · 2020-01-23T07:19:20Z

thank you very much and here is result
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] C:\Program Files\GraalVM\graalvm-ce-java11-19.3.0\bin\native-image.cmd -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar quarkus-command-runner-1.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:+AddAllCharsets -H:EnableURLProtocols=http -H:-JNI -H:-UseServiceLoaderFeature -H:+StackTrace quarkus-command-runner-1.0-SNAPSHOT-runner Warning: the '=' character in program arguments is not fully supported. Make sure that command line arguments using it are wrapped in double quotes. Example: "--vm.Dfoo=bar"

Error: Unknown arguments: 100, io.quarkus.vertx.core.runtime.VertxLogDelegateFactory, true, DISABLED, 1, com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime, 1, 0, http, quarkus-command-runner-1.0-SNAPSHOT-runner`
probably there is already a bug/issue for it somewhere 😃 good day

geoand · 2020-01-23T07:21:43Z

@ozkanpakdil Maybe you can ask on the Quarkus Zulip chat?

gwenneg · 2020-01-23T16:01:08Z

@ozkanpakdil My answer won't be directly related to your issue, but please note that GraalVM 19.3.0 is not supported by Quarkus 1.2.0-CR1. You need to use either GraalVM 19.2.1 or 19.3.1. You should also know that the GraalVM support for the Windows platform is still experimental as explained here. Furthermore, the JDK 11 support was introduced very recently in GraalVM, it was experimental with 19.3.0 and I don't know what the official status is with 19.3.1.

If you still can't generate your native image with GraalVM 19.3.1, please open a new GitHub issue from here. It will be easier for us to help you that way. Thanks!

bradcupit · 2020-01-23T21:06:19Z

@ozkanpakdil: I got it working with a combination of these changes:

    <quarkus-plugin.version>1.2.0.Final</quarkus-plugin.version>
    <quarkus.platform.artifact-id>quarkus-bom</quarkus.platform.artifact-id>
    <quarkus.platform.version>1.2.0.CR1</quarkus.platform.version>

quarkus-plugin.version is the maven plugin and without upgrading it I would get this error:

Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.IllegalStateException: Unsupported version of GraalVM detected: GraalVM Version 19.3.1 CE. Please use GraalVM 19.2.1.

Use quarkus-bom instead of quarkus-universe-bom (thanks @geoand!) since quarkus-universe-bom 1.2.0.Final hasn't been released

Set quarkus.platform.version to 1.2.0.CR1 works. 1.2.0.Final gets close to working but failed with the below error so I had to use 1.2.0.CR1 instead:

... Could not transfer artifact io.quarkus:quarkus-core:jar:1.2.0.Final ...

I'm assuming the 1.2.0.Final release is imminent, perhaps even in progress as we speak, and that's why we're having to cobble a few things together for the moment.

gwenneg requested a review from gsmet January 16, 2020 01:20

gwenneg commented Jan 16, 2020

View reviewed changes

geoand mentioned this pull request Jan 16, 2020

Double GraalVM compatibility effort #6483

Closed

7 tasks

gwenneg force-pushed the issue-6483-support-graalvm-19.2.1-and-19.3.1 branch from 50abff4 to dc2ff88 Compare January 16, 2020 11:02

gwenneg marked this pull request as ready for review January 16, 2020 13:32

gwenneg force-pushed the issue-6483-support-graalvm-19.2.1-and-19.3.1 branch 2 times, most recently from 2621c83 to d68ff6a Compare January 16, 2020 14:47

gsmet requested changes Jan 16, 2020

View reviewed changes

gwenneg commented Jan 16, 2020

View reviewed changes

extensions/jdbc/jdbc-h2/runtime/src/main/java/io/quarkus/jdbc/h2/runtime/graal/Engine.java Show resolved Hide resolved

gwenneg added 6 commits January 16, 2020 20:56

Support both GraalVM 19.2.1 and 19.3.1

33ea19d

Fix quarkus-integration-test-infinispan-embedded in native mode

27af95c

Disable quarkus-integration-test-jsch in native mode temporarily

9b74e0e

Remove useless project.version in quarkus-narayana-stm-deployment

5cbefa6

Disable JNI by default with GraalVM 19.2.1

dea977d

Fix H2 native DeletedElementException

2d68b72

gwenneg force-pushed the issue-6483-support-graalvm-19.2.1-and-19.3.1 branch from 147e8e4 to 2d68b72 Compare January 16, 2020 20:03

gwenneg mentioned this pull request Jan 16, 2020

[WIP] Support both GraalVM 19.2.1 and 19.3.1 - Duplicate targeting GraalVM 19.3.1-java8 #6594

Closed

gsmet approved these changes Jan 17, 2020

View reviewed changes

gsmet merged commit 8d0b94e into quarkusio:master Jan 17, 2020

gwenneg mentioned this pull request Jan 17, 2020

Forbid native image generation from GraalVM 19.3.0.2 #6604

Merged

gwenneg added this to the 1.2.0 milestone Jan 17, 2020

gwenneg deleted the issue-6483-support-graalvm-19.2.1-and-19.3.1 branch January 17, 2020 11:40

gsmet added the release/noteworthy-feature label Jan 20, 2020

This was referenced Jan 20, 2020

RSS regression with 1.1.0.CR1 and GraalVM 19.3.0 #6136

Closed

Increase in native image RSS after upgrade to ce 19.3.0 oracle/graal#1984

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support both GraalVM 19.2.1 and 19.3.1 #6574

Support both GraalVM 19.2.1 and 19.3.1 #6574

gwenneg commented Jan 16, 2020

gwenneg commented Jan 16, 2020

gwenneg Jan 16, 2020

geoand Jan 16, 2020

gwenneg Jan 16, 2020

geoand commented Jan 16, 2020

gsmet commented Jan 16, 2020

geoand commented Jan 16, 2020

gwenneg commented Jan 16, 2020 •

edited

Loading

geoand commented Jan 16, 2020 •

edited

Loading

gwenneg commented Jan 16, 2020

geoand commented Jan 16, 2020

gwenneg commented Jan 16, 2020

gsmet commented Jan 16, 2020

gsmet left a comment

gwenneg commented Jan 17, 2020

geoand commented Jan 17, 2020 •

edited

Loading

geoand commented Jan 17, 2020 •

edited

Loading

gwenneg commented Jan 17, 2020

gsmet commented Jan 17, 2020

geoand commented Jan 17, 2020

ozkanpakdil commented Jan 22, 2020

geoand commented Jan 22, 2020

ozkanpakdil commented Jan 23, 2020 •

edited

Loading

geoand commented Jan 23, 2020

gwenneg commented Jan 23, 2020

bradcupit commented Jan 23, 2020 •

edited

Loading

Support both GraalVM 19.2.1 and 19.3.1 #6574

Support both GraalVM 19.2.1 and 19.3.1 #6574

Conversation

gwenneg commented Jan 16, 2020

gwenneg commented Jan 16, 2020

gwenneg Jan 16, 2020

Choose a reason for hiding this comment

geoand Jan 16, 2020

Choose a reason for hiding this comment

gwenneg Jan 16, 2020

Choose a reason for hiding this comment

geoand commented Jan 16, 2020

gsmet commented Jan 16, 2020

geoand commented Jan 16, 2020

gwenneg commented Jan 16, 2020 • edited Loading

geoand commented Jan 16, 2020 • edited Loading

gwenneg commented Jan 16, 2020

geoand commented Jan 16, 2020

gwenneg commented Jan 16, 2020

gsmet commented Jan 16, 2020

gsmet left a comment

Choose a reason for hiding this comment

gwenneg commented Jan 17, 2020

geoand commented Jan 17, 2020 • edited Loading

geoand commented Jan 17, 2020 • edited Loading

gwenneg commented Jan 17, 2020

gsmet commented Jan 17, 2020

geoand commented Jan 17, 2020

ozkanpakdil commented Jan 22, 2020

geoand commented Jan 22, 2020

ozkanpakdil commented Jan 23, 2020 • edited Loading

geoand commented Jan 23, 2020

gwenneg commented Jan 23, 2020

bradcupit commented Jan 23, 2020 • edited Loading

gwenneg commented Jan 16, 2020 •

edited

Loading

geoand commented Jan 16, 2020 •

edited

Loading

geoand commented Jan 17, 2020 •

edited

Loading

geoand commented Jan 17, 2020 •

edited

Loading

ozkanpakdil commented Jan 23, 2020 •

edited

Loading

bradcupit commented Jan 23, 2020 •

edited

Loading