VS Code cannot access maven dependencies

[tetchel]

After creating a new appsody Java project in Codewind for VS Code:

Project build error: Non-resolvable parent POM for dev.appsody.starter.java-microprofile:starter-app:1.0-SNAPSHOT: Failure to find dev.appsody:java-microprofile:pom:0.2.2 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced and 'parent.relativePath' points at wrong local POM

This causes numerous IDE features to not work such as intellisense and hitting debug breakpoints.

If you create an appsody project of the same type outside of codewind and then creating a new Codewind project of the same type, the new project will work. I expect that disabling/re-enabling would fix it too.

It sounds like the difference maker is that Codewind places the maven artifacts under codewind-workspace/.extensions/appsodyExtension/.m2 while normally they are placed under ~/.m2 which is where VS Code is looking for them.

[tetchel]

I am not sure of the eclipse behaviour.

[makandre]

This is related to appsody/stacks#127. There are some issues with the m2 initialization that needs to be ironed out.

[jgwest]

@makandre @sghung Probably want to mark this as a bug and either hot/stopship (depending on how spicy we think this is 🌶 )

[sghung]

The appsody team had made changes to fix this. Both spring and the microprofile now mount the maven cache location and files get copied there during the init/run. As a result, the dependencies would be copied to the maven cache and be made known to the IDE.

[keithchong]

On Eclipse, the workaround is:

1. From the New Codewind Project Wizard, select the Appsody Java Microprofile template.
2. Finish with the wizard
3. Once the project shows up in the Codewind Explorer view, rIght click on the project in the and select Show Log Files->Show All
4. If your .m2 cache is 'empty' or this is the first time creating an Appsody Microprofile project, you will see that the dependencies are downloaded. You will also see:
   [Container] Installing parent dev.appsody:java-microprofile
5. Wait until this cache is completed. (You can wait until the Liberty defaultServer is started or when the Project status is "Running")
6. Right click on the Project from the Project Explorer and select Maven->Update Project...
7. Accept the defaults and click OK
   ---> The project will be configured and the error "Project build error: Non-resolvable parent POM " should disappear.

If the m2 cache is already set up, then this workaround might not be needed - Any subsequent new projects will be configured properly.

[jgwest]

I'm coming to this as an Appsody-outsider, so feel free to poke holes in my logic 😃 .

The general problem of sharing the Maven cache

In Maven, dependency/plugin jar files are stored in an .m2 cache on your machine(/vm/container). You can find them under (user home dir)/.m2/repository.

All JAR files that are downloaded by Maven are stored in this directory, including external dependencies referenced as <dependency>...</dependency> in the pom.xml. Also stored in this directory are Maven plugins (for example, when you build with Liberty, it downloads the liberty-maven-plugin, which is Liberty-specific extension code that runs during build).

Unfortunately, this .m2 cache is not safe to concurrently read/write from: it is only ever correct for a single process to read or write data to this directory at a time (ie not 'thread-safe'). This is true whether it is a user running mvn package, or it's the Eclipse IDE's m2e plugins, or its Maven inside Appsody containers.
(Citation regarding unsafety: Steven located the relevant Maven bug, https://issues.apache.org/jira/browse/MNG-4706, and if you follow that it goes to an even older item, https://issues.apache.org/jira/browse/MNG-2802 from 2008.)

Fortunately, this is rarely an issue in practice, because the user manually controls when a build takes place, and can ensure that only one process is active at a time (or they delegate this responsibility to their IDE).

Specific to Appsody

My assumption (correct if wrong) is that in order to attempt to solve this bug, Appsody and this PR are bind mounting the user's .m2 directory (BTW this won't work on enterprise Windows) into the container's equivalent directory, using something like docker -v(path to local m2):(container internal maven repo path) (...).
(EDIT: Erin S helpfully clarified that the .m2 bind was not specific to this fix, but generally for reasons like tool compatibility, avoiding the need multiple dependency downloads from the internet, etc)

This means we have introduced the possibility for two different processes (Appsody, and the user's IDE) to concurrently read/write from this directory.

This would likely be fine if Appsody only had manually-triggered builds (eg no "auto-rebuild"), but Appsody listens for source file changes within the container and automatically initiates a Maven build when one is detected.

So now both Appsody, and the user's IDE, are simultaneously listening for changes to source code files (or build artifacts), and both will automatically start a Maven build -- using that shared cache dir -- once a change is detected:

1. User changes a file, MyServlet.java
2. Eclipse detects the file change, begins a mvn package, which starts downloading JARs into the .m2 dir; at the exact same moment, Appsody also detects the same MyServlet.java file change and ALSO initiates a separate mvn package process, which starts downloading the same JARs into the same .m2 dir.
3. Both processes simultaneously download JARs on top of each other, potentially corrupting the downloaded dependency JARs

• IMHO mostly likely to be hit here is the 129MB Liberty openliberty-runtime-19.0.0.X.zip file, which due to size will take relatively long to download, and thus both Maven processes will hit it at roughly the same time.
• I can imagine both processes will start this download simultaneously and both stream bytes to eg (user home)\.m2\repository\io\openliberty\openliberty-runtime\19.0.0.7\openliberty-runtime-19.0.0.7.zip), corrupting each other's downloads.

When and how often will it happen?

I think the burden of proof that this is safe is on the implementor of the Appsody PR above (rather than me, attempting a disproof of safety), but here are the scenarios where we are most likely to see issues:

1. Initial source file changes of a project, and/or initial build

• Most of the Maven dependency/plugin downloads are front loaded when the project is first built, and/or the first bit of source code is modified.

2. Invocations of mvn commands can cause arbitrary plugin download

• When you first call a mvn command, the required Maven plugins to run that command must be downloaded. These will tend to be front loaded (eg the first time mvn clean is called, the first time mvn package is called, etc), but OTOH I sometimes see redownloads of these plugins for no obvious reason.

3. Changes to dependencies in pom.xml

• Probably the most likely of the three to cause problems: when the user adds/updates a dependency in the pom.xml, both Appsody and the IDE will detect this file change and attempt to download the required dependency JARs at the same time.
• This may occur with either the new/updated dependency itself, or with any transitive dependencies that are required by this new/updated dependency.

As for how often, it's tough to model w/o experimentation, but it would depend on how often those 3 above events occur, and then it would depend on network I/O download speed (slower == more likely to occur) and disk I/O speed (likewise).

What is the end result?

As per the Maven bugs, when there are two concurrent writes to a dependency, it can corrupt those dependency JARs in the cache:

"It seems that access to local Maven repository is not concurrent-safe that is multiple Mavens running in parallel may damage contents of local Maven repository. "
(https://issues.apache.org/jira/browse/MNG-2802)

"Since X is not available locally, both can start download of X. Both will download X and both will try to write the file for X to disk.
On Windows platforms, one of them can fail with file locking problems. On Unixes the contents of the written file can be non-deterministic
(https://issues.apache.org/jira/browse/MNG-4706)

You will also get problems on concurrent write-read, but the above concurrent double-write problem is the most serious, as this leaves broken JARs in the cache, which will fail builds (perhaps permanently, with no obvious cause) and provide no obvious solution on how to fix.

During the inception of Microclimate, while investigating how to speed up container builds, we specifically avoided sharing the .m2 cache via bind mount, between the host and the container for these reasons (and have preserved this design decision with Eclipse Codewind).

It would be REALLY GREAT if this were concurrency safe, but inconveniently from our previous investigations, it is not. 😞

[makandre]

Thank you for the insights @jgwest. This is good information and a compelling reason we can take back to appsody team to push for moving the maven setup to the appsody init step.

During our discussions for this problem, the idea of introducing a 2nd-stage init was floated around, which would run a containerized operation to setup up the parent pom (with m2 mounted, so much like what it is now done during appsody run). I realize this does not completely avoid the concurrency problems you outlined but I think it would at least reduce the likelihood of it happening, since init is a one time action done during project creation, whereas run is long running.

Unfortunately we didn't get this 2nd-stage init, but hopefully appsody is still open to providing that, especially given the info you provided (or if we can get to running appsody outside of a containerized environment, then a 2nd-stage init might not be needed)

[keithchong]

Steps for Codewind for VS Code:

1. After creating an Appsody Java Microprofile project ensure that the codewind-workspace is opened
2. Right click on the codewind-workspace folder and select Add Folder to Workspace...
3. Choose the project folder and click Add

Other points to consider:
If you get unresolved parent pom issues, check first the .m2 cache. You may need to clear the cache prior to doing the above steps.

To Debug:

1. Right click on the project and select Restart in Debug Mode
   -> You may get one or both of the following error messages:
   Failed to attach to remote debuggee VM
   or
   Failed to attach debugger
   There is an issue for this: [Appsody debug scenario documentation] Attach Debugger must be run a second time for microprofile project extensions codewind#239
2. Right click on the project and select Attach Debugger

[sghung]

Closing as this has been documented.