Use maven assembly plugin instead of shade plugin

[eduramiba]

• Framework version: 1.0
• Implementations: Jersey / Spring / Spring Boot / Spark

Since using maven shade plugin to create a fat jar is known to be problematic because it can overwrite files by mixing jars (https://product.hubspot.com/blog/the-fault-in-our-jars-why-we-stopped-building-fat-jars), we currently use with success a safer way to deploy lambdas with dependencies: a single zip with all jars in a lib folder (as documented in https://docs.aws.amazon.com/en_en/lambda/latest/dg/create-deployment-pkg-zip-java.html)

Example code of how we do it in maven:

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <configuration>
            <finalName>${theapp.brandingToken}-${project.version}</finalName>
            <archive>
                <manifest>
                    <addClasspath>true</addClasspath>
                    <classpathPrefix>lib/</classpathPrefix>
                    <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
                    <addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
                </manifest>
            </archive>
        </configuration>
    </plugin>
    
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <executions>
            <execution>
                <id>copy-dependencies</id>
                <phase>package</phase>
                <goals>
                    <goal>copy-dependencies</goal>
                </goals>
                <configuration>
                    <outputDirectory>${project.build.directory}/lib</outputDirectory>
                    <includeScope>runtime</includeScope>
                </configuration>
            </execution>
        </executions>
    </plugin>
    
    <!-- Assembler to produce ditribution zip -->
    <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <executions>
            <execution>
                <id>zip-assembly</id>
                <phase>package</phase>
                <goals>
                    <goal>single</goal>
                </goals>
                <configuration>
                    <finalName>${theapp.brandingToken}-${project.version}</finalName>
                    <descriptors>
                        <descriptor>src/assembly/bin.xml</descriptor>
                    </descriptors>
                    <attach>false</attach>
                </configuration>
            </execution>
        </executions>
    </plugin>

And bin.xml file:

<assembly>
    <id>aws-lambda-package</id>
    <formats>
        <format>zip</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <fileSets>
        <fileSet>
            <directory>${project.build.directory}${file.separator}lib</directory>
            <outputDirectory>lib</outputDirectory>
            <useDefaultExcludes>true</useDefaultExcludes>
        </fileSet>
    </fileSets>
    <files>
        <file>
            <source>${project.build.directory}${file.separator}${theapp.brandingToken}-${project.version}.jar</source>
            <outputDirectory>lib</outputDirectory>
        </file>
    </files>
</assembly>

Thanks,
Eduardo

[sapessi]

Thanks for the feedback @eduramiba - I've been wondering whether it's time to build a Lambda-specific maven plugin to package an application and slim down the output jar as much as possible. I'll keep this open and think it through. I'll definitely look into adding the assembly plugin as an alternative to our samples and archetypes.

[humank]

Good to know this way to pack it. Thanks @eduramiba !

[Juchar]

@eduramiba Thank you! How would you exclude the embedded tomcat in this case?

[eduramiba]

@Juchar I guess you can use the goal options, https://maven.apache.org/plugins/maven-dependency-plugin/copy-dependencies-mojo.html

But if your dependency has provided scope, it will not be copied as includeScope is runtime.

[eduramiba]

Also, the jar manifest entries are not actually necessary in lambda runtime since it will load all jars in lib folder.

[Juchar]

@eduramiba Ok, thank you.

I changed you approach a little bit and it is also working (you only need to configure the assembly plugin this way):

<plugin>
  <artifactId>maven-assembly-plugin</artifactId>
  <executions>
    <execution>
    <id>zip-assembly</id>
    <phase>package</phase>
    <goals>
      <goal>single</goal>
    </goals>
    <configuration>
      <finalName>${project.artifactId}</finalName>
      <descriptors>
        <descriptor>src/assembly/bin.xml</descriptor>
      </descriptors>
      <archive>
          <manifest>
          <addClasspath>true</addClasspath>
          <classpathPrefix>lib/</classpathPrefix>
          <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
          <addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
        </manifest>
      </archive>
      <attach>false</attach>
    </configuration>
    </execution>
  </executions>
</plugin>

And the bin.xml:

<assembly>
  <id>aws-lambda-package</id>
  <formats>
    <format>zip</format>
  </formats>
  <includeBaseDirectory>false</includeBaseDirectory>
  <fileSets>
    <fileSet>
      <directory>${project.build.directory}${file.separator}classes</directory>
      <outputDirectory/>
    </fileSet>
  </fileSets>
  <dependencySets>
    <dependencySet>
      <excludes>
        <exclude>org.springframework.boot:spring-boot-devtools</exclude>
        <exclude>org.apache.tomcat.embed:*</exclude>
      </excludes>
      <useProjectArtifact>false</useProjectArtifact>
      <outputDirectory>lib</outputDirectory>
    </dependencySet>
  </dependencySets>
</assembly>

[eugenevd]

I've tried both the suggested examples provided.
The first results in my own code (Handler) to be added to it's own jar, and then along with the other dependecy jars, is put into "lib/" directly inside the zip.
lib/myHandler.jar
lib/dep1.jar

The second example puts all the dependencies in lib/ again, but at the root of the zip the directory structure;
eg:
com/company/Handler.class
lib/dep1.jar

Either way, I get
Class not found: com.company.Handler: java.lang.ClassNotFoundException

I've tried both ways to define the handler:

com.company.Handler
com.company.Handler::handleRequest

What am I missing?

Background
Using the maven shade plugin has been the only way that I could get the deployment to work, but after adding the bouncycastle dependency, I got error:
Error creating shaded jar: Invalid signature file digest for Manifest main attributes

There are suggestions to remove the signatures:

<configuration>
    <filters>
        <filter>
            <artifact>*:*</artifact>
            <excludes>
                <exclude>META-INF/*.SF</exclude>
                <exclude>META-INF/*.DSA</exclude>
                <exclude>META-INF/*.RSA</exclude>
            </excludes>
        </filter>
    </filters>
    <!-- Additional configuration. -->
</configuration>

but surely these will be rejected.
So my conclusion was to find a way to package all the dependency jars as is, hence the suggestions by earlier posts here.

[sapessi]

Hi @eugenevd - I'm on the road right now but I'll try to replicate your issue with the Shade plugin and see if we can find a fix on Thursday. As for the assembly plugin questions, I'll let @eduramiba and @Juchar chime in since they clearly know more than I do on the topic.

[Juchar]

Hi @eugenevd - the assembly configuration I mention above will produce a zip file following these conventions.

For me it worked out of the box. Maybe you could show us the sam.yml, possibly your error is somewhere else?

This is the relevant part from mine:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: My Description
Resources:
  MyServiceFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: com.my.function.StreamLambdaHandler::handleRequest
      Runtime: java8
      CodeUri: target/my-function-aws-lambda-package.zip
      MemorySize: 768
      Policies: AWSLambdaBasicExecutionRole
      Timeout: 30
      Events:
        GetResource:
          Type: Api
          Properties:
            Path: /{proxy+}
            Method: GET

Outputs:
  MyFunctionAPI:
    Description: URL for application
    Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/entity'
    Export:
      Name: MyFunctionAPI

[eugenevd]

To illustrate my situation, I've forked this project to https://github.com/eugenevd/aws-serverless-java-container and used the springboot pet-store sample as a base.

Therein two branches, one each for each of the examples posted by @eduramiba (branch issue133-comment302648748) and @Juchar (branch issue133-comment399079415)
master contains no changes except for a readme file.

Branch: master
No changes, uses mvn-shade and works as expected (except the issues when adding signed jars like bouncy-castle)

Branch: issue133-comment302648748
This results in a ZIP file that has the dependency jars in lib/ and the handler code in a .jar also in lib/
The result is ClassNotFoundException

Branch: issue133-comment399079415
This results in a ZIP file that has the dependency jars still in lib/ but with the code (as .class files) in the root of the zip.
The result is still ClassNotFoundException

Comments
I've made sure in each case to update sam.yaml to point to the correct zip , as each branch produces a zip file with a different name (switching branches leave behind the zip from the previous build)

packagedeployrun.sh - executes all the steps as outlined by https://github.com/awslabs/aws-serverless-java-container/wiki#deploying-the-sample-applications

To double check, after a deployment, I downloaded the zip files uploaded to the bucket and compared their contents with the "mvn package" produced zip files. These were the same as expected.

--
@Juchar - you asked to see the sam.yaml - see each of those branches.
And yes, the link you posted to the convention is what I've worked with too without success.

@sapessi No worries. Just to reiterate though; I'm trying to NOT use the shade plugin. If you have a signed dependency jar, these are invalidated when re-arranging that content the way shade does. Which is why I'm after what this issues' examples is about ...

[Juchar]

@eugenevd Puh, I checked out the branch and created the zip. After having a look everything looks fine to me. I have to admit that I ran out of ideas, from my point of view it should just work out.

[eugenevd]

@Juchar So you're also getting errors? The examples posted here by yourself and @eduramiba ; is there a sample where they're working somewhere?

[Juchar]

@eugenevd I did just package it, not deploy it. For my lambda in my project the configuration I provided here works fine.

[eugenevd]

I was hoping to find a working example that I can look at, to compare with, to see where it is that I'm going wrong...

[jrd281]

Did you get this resolved? I've got an issue where if I build it on my local machine, everything works. But if I build it on CodeBuild, it doesn't work.

[legezam]

@eugenevd , I had a similar issue and solved it with the below settings:
pom.xml:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <executions>
                    <execution>
                        <id>copy-dependencies</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.build.directory}/lib</outputDirectory>
                            <includeScope>runtime</includeScope>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

            <!-- Assembler to produce ditribution zip -->
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <executions>
                    <execution>
                        <id>zip-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <configuration>
                            <finalName>${theapp.brandingToken}-${project.version}</finalName>
                            <descriptors>
                                <descriptor>src/assembly/bin.xml</descriptor>
                            </descriptors>
                            <attach>false</attach>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

bin.xml:

<assembly>
    <id>aws-lambda-package</id>
    <formats>
        <format>zip</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <fileSets>
        <fileSet>
            <directory>${project.build.directory}${file.separator}lib</directory>
            <outputDirectory>lib</outputDirectory>
            <useDefaultExcludes>true</useDefaultExcludes>
        </fileSet>
        <fileSet>
            <directory>${project.build.directory}${file.separator}classes</directory>
            <outputDirectory>/</outputDirectory>
        </fileSet>
    </fileSets>

</assembly>

This works for me using sam local start-api. Update: works even on aws.
Two changes i made:
1, removed the jar plugin execution as the aws tutorial says we don't need a jar at all.
2, modified the assembly to copy the classes directory as a whole to the zip root.

[sapessi]

Resolving in preparation for 1.3 release.