Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: Records with the same package as a data pipeline cannot be used as inbound/outbound types #152

Closed
ewilkins-csi opened this issue Jun 14, 2024 · 1 comment
Closed

BUG: Records with the same package as a data pipeline cannot be used as inbound/outbound types #152

ewilkins-csi opened this issue Jun 14, 2024 · 1 comment
Assignees
@jaebchoi
Labels
bug Something isn't working
Milestone
1.10.0

Comments

@ewilkins-csi
Copy link
Contributor

ewilkins-csi commented Jun 14, 2024 •
edited by jaebchoi
Loading

Description

When the package name of a record matches the package name of a Spark data pipeline, the record cannot be used as an inbound or outbound record type because a NPE is thrown. It's unclear if this would happen with PySpark pipelines.

Steps to Reproduce

  1. Create an aissemble project
  2. Add a record MetaModel with the package org.test
  3. Add a data-delivery-spark pipeline MetaModel with the package org.test that sets the inbound or outbound recordType of a step to the record created in step 2
  4. Run the build and a NPE is thrown during generate-sources

Expected Behavior

The build should successfully generate sources via Fermenter.

Actual Behavior

A NullPointerException is thrown.

Definition of Done

  • Reproduce issue as mentioned in Step to Reproduce section in above.
  • Fix Null pointer Exception when Record package matches with Pipeline Package when specifying inbound or outbound Record Type.
  • Run Integration test and archetype test to make sure successful run.
  • Follow Step to Reproduce and Verify There's no NullPointerException Thrown.

Testing Steps

  • Pull latest dev branch and build using mvn clean install
  • Create an aissemble project using following command
mvn archetype:generate -U -DarchetypeGroupId=com.boozallen.aissemble \
  -DarchetypeArtifactId=foundation-archetype \
  -DarchetypeVersion=1.10.0-SNAPSHOT\
  -DgroupId=com.boozallen.aissemble.test \
  -DartifactId=issue-152\
  -DprojectGitUrl=git.example.com/issue-152\
  -DprojectName="issue-152”
  • Create a record MetaModel with the package called org.test.aissemble.data.profiling (Contact James Choi for example Record)
  • Create Dictonary MetaModel (doesn't have to match any package and Contact James Choi for example Dictonary)
  • Create a data-delivery-spark pipeline MetaModel with the package org.test.aissemble.data.profiling that sets the inbound or outbound recordType of a step to the record created in above (Contact James Choi for example pipeline)
  • cd into downstream project.
  • Add a data delivery spark pipeline, record and dictionary json files to an aissemble project by saving attached json into following respectively.
{downstream-project}/{downstream-project}-pipeline-models/src/main/resources/pipelines
{downstream-project}/{downstream-project}-pipeline-models/src/main/resources/records
{downstream-project}/{downstream-project}-pipeline-models/src/main/resources/dictionary
  • Run the mvn clean install and run all manual actions
  • Verified downstream project is built successfully.

Additional Context

  • Version: aiSSEMBLE 1.7.0
  • Build output:
10:36:59 [INFO] --- fermenter-mda:2.10.3:generate-sources (generate-sources) @ spark-pipeline ---
10:36:59 [INFO] Loading targets from: jar:file:/Users/ewilkins/.m2/repository/com/boozallen/aissemble/foundation-mda/1.7.0/foundation-mda-1.7.0.jar!/targets.json
10:36:59 [INFO] Loading targets from: jar:file:/Users/ewilkins/.m2/repository/org/technologybrewery/fermenter/brett/brett-mda/2.8.0/brett-mda-2.8.0.jar!/targets.json
10:36:59 [INFO] Loading profiles from: jar:file:/Users/ewilkins/.m2/repository/com/boozallen/aissemble/foundation-mda/1.7.0/foundation-mda-1.7.0.jar!/profiles.json
10:36:59 [INFO] Loading profiles from: jar:file:/Users/ewilkins/.m2/repository/org/technologybrewery/fermenter/brett/brett-mda/2.8.0/brett-mda-2.8.0.jar!/profiles.json
10:36:59 [INFO] Loading families from: jar:file:/Users/ewilkins/.m2/repository/com/boozallen/aissemble/foundation-mda/1.7.0/foundation-mda-1.7.0.jar!/families.json
10:36:59 [INFO] Loading types from: jar:file:/Users/ewilkins/.m2/repository/com/boozallen/aissemble/foundation-mda/1.7.0/foundation-mda-1.7.0.jar!/types.json
10:36:59 [INFO] Checking for local types at: /Users/ewilkins/dev/aiops/init-tests/oneoff/test-170-upgrade/test-170-upgrade-pipelines/spark-pipeline/src/main/resources/types.json...
10:36:59 [INFO] START: loading metamodel repository implementation: com.boozallen.aiops.mda.metamodel.AIOpsModelInstanceRepostory ...
10:36:59 [WARNING] deployment-config.json was not found.  Proceeding with defaults--
10:36:59 [INFO] Loaded 0 Dictionary(s)
10:36:59 [WARNING] /Users/ewilkins/dev/aiops/init-tests/oneoff/test-170-upgrade/test-170-upgrade-pipelines/spark-pipeline/src/main/resources/composites is not a valid directory!
10:36:59 [INFO] Loaded 0 Record(s)
10:36:59 [INFO] Loaded 0 Pipeline(s)
10:36:59 [INFO] Metamodel instances for artifactId 'spark-pipeline' have been loaded - 2ms
10:36:59 [INFO] Loaded 1 Dictionary(s)
10:36:59 [WARNING] /Users/ewilkins/dev/aiops/init-tests/oneoff/test-170-upgrade/test-170-upgrade-pipelines/spark-pipeline/src/main/resources/composites is not a valid directory!
10:36:59 [INFO] Loaded 1 Record(s)
10:36:59 [INFO] Loaded 3 Pipeline(s)
10:36:59 [INFO] Metamodel instances for artifactId 'test-170-upgrade-pipeline-models' have been loaded - 16ms
10:36:59 [INFO] COMPLETE: metamodel repository loading in 18 ms
10:36:59 [INFO] START: validating metamodel repository...
10:36:59 [INFO] COMPLETE: validation of metamodel repository in 1ms
10:36:59 [INFO] Generating code for profile 'data-delivery-spark-pipeline'
10:36:59 [ERROR] Error while performing source generation
org.technologybrewery.fermenter.mda.generator.GenerationException: Unable to generate file
    at org.technologybrewery.fermenter.mda.generator.AbstractGenerator.generateFile (AbstractGenerator.java:84)
    at com.boozallen.aiops.mda.generator.TargetedPipelineStepJavaGenerator.generate (TargetedPipelineStepJavaGenerator.java:53)
    at org.technologybrewery.fermenter.mda.GenerateSourcesHelper.performSourceGeneration (GenerateSourcesHelper.java:290)
    at org.technologybrewery.fermenter.mda.GenerateSourcesMojo.execute (GenerateSourcesMojo.java:182)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: org.apache.velocity.exception.MethodInvocationException: Invocation of method 'getBaseImports' in  class com.boozallen.aiops.mda.metamodel.element.java.JavaStep threw exception java.lang.NullPointerException at templates/data-delivery-spark/synchronous.processor.base.java.vm[line 3, column 28]
    at org.apache.velocity.runtime.parser.node.ASTIdentifier.execute (ASTIdentifier.java:246)
    at org.apache.velocity.runtime.parser.node.ASTReference.execute (ASTReference.java:368)
    at org.apache.velocity.runtime.parser.node.ASTReference.value (ASTReference.java:704)
    at org.apache.velocity.runtime.directive.Foreach.render (Foreach.java:237)
    at org.apache.velocity.runtime.parser.node.ASTDirective.render (ASTDirective.java:304)
    at org.apache.velocity.runtime.parser.node.SimpleNode.render (SimpleNode.java:439)
    at org.apache.velocity.Template.merge (Template.java:358)
    at org.apache.velocity.Template.merge (Template.java:262)
    at org.technologybrewery.fermenter.mda.generator.AbstractGenerator.generateFile (AbstractGenerator.java:67)
    at com.boozallen.aiops.mda.generator.TargetedPipelineStepJavaGenerator.generate (TargetedPipelineStepJavaGenerator.java:53)
    at org.technologybrewery.fermenter.mda.GenerateSourcesHelper.performSourceGeneration (GenerateSourcesHelper.java:290)
    at org.technologybrewery.fermenter.mda.GenerateSourcesMojo.execute (GenerateSourcesMojo.java:182)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: java.lang.NullPointerException
    at com.boozallen.aiops.mda.metamodel.element.java.JavaStep.addRecordImports (JavaStep.java:456)
    at com.boozallen.aiops.mda.metamodel.element.java.JavaStep.deriveImplType (JavaStep.java:384)
    at com.boozallen.aiops.mda.metamodel.element.java.JavaStep.getImplInboundType (JavaStep.java:327)
    at com.boozallen.aiops.mda.metamodel.element.java.JavaStep.getBaseInboundType (JavaStep.java:231)
    at com.boozallen.aiops.mda.metamodel.element.java.JavaStep.getBaseSignature (JavaStep.java:82)
    at com.boozallen.aiops.mda.metamodel.element.java.JavaStep.getBaseImports (JavaStep.java:163)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.apache.velocity.runtime.parser.node.PropertyExecutor.execute (PropertyExecutor.java:149)
    at org.apache.velocity.util.introspection.UberspectImpl$VelGetterImpl.invoke (UberspectImpl.java:722)
    at org.apache.velocity.runtime.parser.node.ASTIdentifier.execute (ASTIdentifier.java:217)
    at org.apache.velocity.runtime.parser.node.ASTReference.execute (ASTReference.java:368)
    at org.apache.velocity.runtime.parser.node.ASTReference.value (ASTReference.java:704)
    at org.apache.velocity.runtime.directive.Foreach.render (Foreach.java:237)
    at org.apache.velocity.runtime.parser.node.ASTDirective.render (ASTDirective.java:304)
    at org.apache.velocity.runtime.parser.node.SimpleNode.render (SimpleNode.java:439)
    at org.apache.velocity.Template.merge (Template.java:358)
    at org.apache.velocity.Template.merge (Template.java:262)
    at org.technologybrewery.fermenter.mda.generator.AbstractGenerator.generateFile (AbstractGenerator.java:67)
    at com.boozallen.aiops.mda.generator.TargetedPipelineStepJavaGenerator.generate (TargetedPipelineStepJavaGenerator.java:53)
    at org.technologybrewery.fermenter.mda.GenerateSourcesHelper.performSourceGeneration (GenerateSourcesHelper.java:290)
    at org.technologybrewery.fermenter.mda.GenerateSourcesMojo.execute (GenerateSourcesMojo.java:182)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
@ewilkins-csi ewilkins-csi added the bug Something isn't working label Jun 14, 2024
@jaebchoi jaebchoi self-assigned this Sep 24, 2024
@jaebchoi jaebchoi added this to the 1.9.0 milestone Sep 25, 2024
@ewilkins-csi ewilkins-csi modified the milestones: 1.9.0, 1.10.0 Sep 25, 2024
jaebchoi added a commit that referenced this issue Sep 26, 2024
@jaebchoi
#152 Records with the same package as a data pipeline cannot be used …
c1792d0 
…as inbound/outbound types
jaebchoi added a commit that referenced this issue Sep 27, 2024
@jaebchoi
#152 Records with the same package as a data pipeline cannot be used …
9283e96 
…as inbound/outbound types
@jaebchoi jaebchoi mentioned this issue Sep 27, 2024
Merged
jaebchoi added a commit that referenced this issue Sep 27, 2024
@jaebchoi
#152 Records with the same package as a data pipeline cannot be used …
6d56ee9 
…as inbound/outbound types
jaebchoi added a commit that referenced this issue Sep 27, 2024
@jaebchoi
#152 Records with the same package as a data pipeline cannot be used …
8ad7794 
…as inbound/outbound types
jaebchoi added a commit that referenced this issue Sep 27, 2024
@jaebchoi
Merge pull request #378 from boozallen/#152-record-same-package-error
88fd26b 
#152 Records with the same package as a data pipeline cannot be used as inbound/outbound types
@meliz19
Copy link
Contributor

meliz19 commented Sep 27, 2024

issue-152-build-success

@meliz19 meliz19 closed this as completed Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jaebchoi jaebchoi

Labels
bug Something isn't working
Projects
None yet
Milestone
1.10.0
Development

No branches or pull requests
3 participants
@ewilkins-csi @meliz19 @jaebchoi