[TACKLE-268]-Request failed with status 500 on importing invalid csv …

…file (#107) * add container image generation * Add hibernate-validation and tests * Fix test * Mock controls endpoints * Properly clean data inserted during test execution (#2) * TACKLE-271: case insensitive headers now allowed in import csv file (#82) * TACKLE-290 importer native image tests (#88) * TACKLE-290 importer native image tests * TACKLE-290: removed userTransactions in tests * TACKLE-290: convert to use endpoints * TACKLE-290: convert to use endpoints so NativeImageTests don't fail * TACKLE-290:inititialise mocks * TACKLE-290: initialise mocks * TACKLE-290: add wiremock http server to enable native image tests to call out to remote rest api * TACKLE-290: tidy up importer test code * TACKLE-290: increase test coverage for importer * TACKLE-290: added extra test * TACKLE-290: test coverage * TACKLE-290: fix problem with controls services query parameters not being picked up * TACKLE-290: remove status field from ApplicationImport * Tackle 290 - Fix native tests (#6) * Comment tests * Restart changes * comment test * comment test * comment test * Working tests * build tests inside own gh account * fix ImportServiceTest * Remove comments * restore gh workflow * restore gh workflow * TACKLE-290: fix merged test to be native-compliant * TACKLE-290 Removed 'baseUri' and changed 'mp-rest/uri' props (#7) * TACKLE-290: alter order of ImportServiceTest methods * TACKLE-290: alter test method ordering * TACKLE-290 Refactored some tests and WireMock (#8) Co-authored-by: Carlos E. Feria Vila <[email protected]> Co-authored-by: Marco Rizzi <[email protected]> * Maven pom.xml(deps-dev): Bump mockito-core from 3.8.0 to 3.12.0 (#117) Bumps [mockito-core](https://github.com/mockito/mockito) from 3.8.0 to 3.12.0. - [Release notes](https://github.com/mockito/mockito/releases) - [Commits](mockito/mockito@v3.8.0...v3.12.0) --- updated-dependencies: - dependency-name: org.mockito:mockito-core dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Maven pom.xml(deps-dev): Bump wiremock-jre8 from 2.29.0 to 2.30.1 (#120) Bumps [wiremock-jre8](https://github.com/tomakehurst/wiremock) from 2.29.0 to 2.30.1. - [Release notes](https://github.com/tomakehurst/wiremock/releases) - [Commits](wiremock/wiremock@2.29.0...2.30.1) --- updated-dependencies: - dependency-name: com.github.tomakehurst:wiremock-jre8 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Maven pom.xml(deps-dev): Bump assertj-core from 2.6.0 to 3.20.2 (#76) Bumps [assertj-core](https://github.com/assertj/assertj-core) from 2.6.0 to 3.20.2. - [Release notes](https://github.com/assertj/assertj-core/releases) - [Commits](assertj/assertj@assertj-core-2.6.0...assertj-core-3.20.2) --- updated-dependencies: - dependency-name: org.assertj:assertj-core dependency-type: direct:development update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Maven pom.xml(deps-dev): Bump postgresql from 1.15.2 to 1.16.0 (#96) Bumps [postgresql](https://github.com/testcontainers/testcontainers-java) from 1.15.2 to 1.16.0. - [Release notes](https://github.com/testcontainers/testcontainers-java/releases) - [Changelog](https://github.com/testcontainers/testcontainers-java/blob/master/CHANGELOG.md) - [Commits](testcontainers/testcontainers-java@1.15.2...1.16.0) --- updated-dependencies: - dependency-name: org.testcontainers:postgresql dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Maven pom.xml(deps-dev): Bump mockito-core from 3.12.0 to 3.12.1 (#121) Bumps [mockito-core](https://github.com/mockito/mockito) from 3.12.0 to 3.12.1. - [Release notes](https://github.com/mockito/mockito/releases) - [Commits](mockito/mockito@v3.12.0...v3.12.1) --- updated-dependencies: - dependency-name: org.mockito:mockito-core dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * TACKLE-268 Properly clean data inserted during test execution ('Import-app-8') Co-authored-by: Mark Brophy <[email protected]> Co-authored-by: Carlos E. Feria Vila <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * TACKLE-268 Enhancements after the review * TACKLE-268 Removed temporary stub in ImportServiceTest Co-authored-by: Marco Rizzi <[email protected]> Co-authored-by: Mark Brophy <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: mrizzi <[email protected]>
migtools · Aug 23, 2021 · ef41e1f · ef41e1f
1 parent 181b593
commit ef41e1f
Show file tree

Hide file tree

Showing 6 changed files with 253 additions and 11 deletions.
diff --git a/src/main/java/io/tackle/applicationinventory/entities/ApplicationImport.java b/src/main/java/io/tackle/applicationinventory/entities/ApplicationImport.java
@@ -6,59 +6,155 @@
 import io.tackle.commons.annotations.Filterable;
 import io.tackle.commons.entities.AbstractEntity;
 
-import javax.persistence.*;
+import javax.persistence.Entity;
+import javax.persistence.ManyToOne;
+import javax.persistence.Table;
+import javax.validation.constraints.Size;
 
 @Entity
 @Table(name = "application_import")
 public class ApplicationImport extends AbstractEntity {
+    public static final int APP_NAME_MAX_LENGTH = 120;
+
     private String recordType1;
+
+    @Size(max = APP_NAME_MAX_LENGTH)
     private String applicationName;
+
+    @Size(max = 250)
     private String description;
+
+    @Size(max = 250)
     private String comments;
+
+    @Size(max = 120)
     private String businessService;
+
+    @Size(max = 40)
     private String tagType1;
+
+    @Size(max = 40)
     private String tag1;
+
+    @Size(max = 40)
     private String tagType2;
+
+    @Size(max = 40)
     private String tag2;
+
+    @Size(max = 40)
     private String tagType3;
+
+    @Size(max = 40)
     private String tag3;
+
+    @Size(max = 40)
     private String tagType4;
+
+    @Size(max = 40)
     private String tag4;
+
+    @Size(max = 40)
     private String tagType5;
+
+    @Size(max = 40)
     private String tag5;
+
+    @Size(max = 40)
     private String tagType6;
+
+    @Size(max = 40)
     private String tag6;
+
+    @Size(max = 40)
     private String tagType7;
+
+    @Size(max = 40)
     private String tag7;
+
+    @Size(max = 40)
     private String tagType8;
+
+    @Size(max = 40)
     private String tag8;
+
+    @Size(max = 40)
     private String tagType9;
+
+    @Size(max = 40)
     private String tag9;
+
+    @Size(max = 40)
     private String tagType10;
+
+    @Size(max = 40)
     private String tag10;
+
+    @Size(max = 40)
     private String tagType11;
+
+    @Size(max = 40)
     private String tag11;
+
+    @Size(max = 40)
     private String tagType12;
+
+    @Size(max = 40)
     private String tag12;
+
+    @Size(max = 40)
     private String tagType13;
+
+    @Size(max = 40)
     private String tag13;
+
+    @Size(max = 40)
     private String tagType14;
+
+    @Size(max = 40)
     private String tag14;
+
+    @Size(max = 40)
     private String tagType15;
+
+    @Size(max = 40)
     private String tag15;
+
+    @Size(max = 40)
     private String tagType16;
+
+    @Size(max = 40)
     private String tag16;
+
+    @Size(max = 40)
     private String tagType17;
+
+    @Size(max = 40)
     private String tag17;
+
+    @Size(max = 40)
     private String tagType18;
+
+    @Size(max = 40)
     private String tag18;
+
+    @Size(max = 40)
     private String tagType19;
+
+    @Size(max = 40)
     private String tag19;
+
+    @Size(max = 40)
     private String tagType20;
+
+    @Size(max = 40)
     private String tag20;
+
     private String errorMessage;
+
     @Filterable(check = CheckType.EQUAL)
     public Boolean isValid = true;
+
     @Filterable
     public String filename;
 

diff --git a/src/main/java/io/tackle/applicationinventory/services/ImportService.java b/src/main/java/io/tackle/applicationinventory/services/ImportService.java
@@ -12,6 +12,7 @@
 import io.tackle.applicationinventory.entities.ImportSummary;
 import io.tackle.applicationinventory.mapper.ApplicationInventoryAPIMapper;
 import io.tackle.applicationinventory.mapper.ApplicationMapper;
+import org.apache.commons.lang3.StringUtils;
 import org.eclipse.microprofile.rest.client.inject.RestClient;
 import org.jboss.resteasy.annotations.providers.multipart.MultipartForm;
 
@@ -20,6 +21,7 @@
 import javax.persistence.EntityManager;
 import javax.transaction.Transactional;
 import javax.transaction.UserTransaction;
+import javax.validation.Validator;
 import javax.ws.rs.Consumes;
 import javax.ws.rs.POST;
 import javax.ws.rs.Path;
@@ -53,6 +55,9 @@ public class ImportService {
     @RestClient
     BusinessServiceService businessServiceService;
 
+    @Inject
+    Validator validator;
+
     @POST
     @Path("/upload")
     @Consumes(MediaType.MULTIPART_FORM_DATA)
@@ -121,14 +126,28 @@ public Response importFile(@MultipartForm MultipartImportBody data) {
     private List<ApplicationImport> writeFile(String content, String filename, ImportSummary parentObject) throws IOException {
 
         MappingIterator<ApplicationImport> iter = decode(content);
-        List<ApplicationImport> importList = new ArrayList();
+        List<ApplicationImport> importList = new ArrayList<>();
         while (iter.hasNext())
         {
             ApplicationImport importedApplication = iter.next();
-            importedApplication.setFilename(filename);
-            importedApplication.importSummary = parentObject;
-            importList.add(importedApplication);
-            importedApplication.persistAndFlush();
+
+            ApplicationImport appToPersist;
+            if (validator.validate(importedApplication).isEmpty()) {
+                appToPersist = importedApplication;
+
+                importList.add(appToPersist);
+            } else {
+                String truncatedAppName = StringUtils.truncate(importedApplication.getApplicationName().trim(), ApplicationImport.APP_NAME_MAX_LENGTH);;
+
+                appToPersist = new ApplicationImport();
+                appToPersist.setApplicationName(truncatedAppName);
+                appToPersist.setValid(false);
+                appToPersist.setErrorMessage("Max length error: one or more column's max length were exceeded");
+            }
+
+            appToPersist.setFilename(filename);
+            appToPersist.importSummary = parentObject;
+            appToPersist.persistAndFlush();
         }
         return importList;
     }

diff --git a/src/test/java/io/tackle/applicationinventory/services/ImportServiceTest.java b/src/test/java/io/tackle/applicationinventory/services/ImportServiceTest.java
@@ -438,7 +438,7 @@ protected void testImportServiceCaseInsensitiveColumnHeaders() {
     @Order(8)
     protected void testImportServiceNoTagsRetrieved() {
 
-        WireMock.stubFor(get(urlPathEqualTo("/controls/tag"))
+        final StubMapping tagStubMapping = WireMock.stubFor(get(urlPathEqualTo("/controls/tag"))
                 .willReturn(aResponse()
                         .withHeader("Content-Type", "application/json")
                         .withBody("")));
@@ -469,11 +469,8 @@ protected void testImportServiceNoTagsRetrieved() {
                 .log().body()
                 .body("[0].'errorMessage'", is("Unable to retrieve TagTypes from remote resource"));
 
-
-
-
         removeTestObjects(Collections.emptyList());
-
+        WireMock.removeStub(tagStubMapping);
     }
 
     @Test

diff --git a/src/test/java/io/tackle/applicationinventory/services/issues/IssueTACKLE268Test.java b/src/test/java/io/tackle/applicationinventory/services/issues/IssueTACKLE268Test.java
@@ -0,0 +1,120 @@
+package io.tackle.applicationinventory.services.issues;
+
+import io.quarkus.test.common.QuarkusTestResource;
+import io.quarkus.test.common.ResourceArg;
+import io.quarkus.test.junit.QuarkusTest;
+import io.restassured.RestAssured;
+import io.restassured.config.EncoderConfig;
+import io.restassured.http.ContentType;
+import io.tackle.applicationinventory.services.WireMockControlsServices;
+import io.tackle.commons.testcontainers.KeycloakTestResource;
+import io.tackle.commons.testcontainers.PostgreSQLDatabaseTestResource;
+import io.tackle.commons.tests.SecuredResourceTest;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import javax.ws.rs.core.MediaType;
+import java.io.File;
+import java.util.Arrays;
+
+import static io.restassured.RestAssured.given;
+import static org.hamcrest.Matchers.is;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+@QuarkusTest
+@QuarkusTestResource(value = PostgreSQLDatabaseTestResource.class,
+        initArgs = {
+                @ResourceArg(name = PostgreSQLDatabaseTestResource.DB_NAME, value = "application_inventory_db"),
+                @ResourceArg(name = PostgreSQLDatabaseTestResource.USER, value = "application_inventory"),
+                @ResourceArg(name = PostgreSQLDatabaseTestResource.PASSWORD, value = "application_inventory")
+        }
+)
+@QuarkusTestResource(value = KeycloakTestResource.class,
+        initArgs = {
+                @ResourceArg(name = KeycloakTestResource.IMPORT_REALM_JSON_PATH, value = "keycloak/quarkus-realm.json"),
+                @ResourceArg(name = KeycloakTestResource.REALM_NAME, value = "quarkus")
+        }
+)
+@QuarkusTestResource(WireMockControlsServices.class)
+// https://issues.redhat.com/browse/TACKLE-268
+public class IssueTACKLE268Test extends SecuredResourceTest {
+
+    @BeforeAll
+    public static void init() {
+        PATH = "/file/upload";
+    }
+
+    @Test
+    protected void testImportServiceLongCSVColumnValues() {
+        ClassLoader classLoader = getClass().getClassLoader();
+        File importFile = new File(classLoader.getResource("long_characters_columns.csv").getFile());
+
+        given()
+                .config(RestAssured.config().encoderConfig(EncoderConfig.encoderConfig().encodeContentTypeAs("multipart/form-data", ContentType.JSON)))
+                .contentType(MediaType.MULTIPART_FORM_DATA)
+                .accept(MediaType.MULTIPART_FORM_DATA)
+                .multiPart("file", importFile)
+                .multiPart("fileName", "long_characters_columns.csv")
+                .when().post(PATH)
+                .then()
+                .statusCode(200);
+
+        final long importSummaryId = Long.parseLong(given()
+                .accept("application/hal+json")
+                .when()
+                .get("/import-summary")
+                .then()
+                .statusCode(200)
+                .body("_embedded.import-summary.size()", is(1),
+                        "_embedded.import-summary[0].importStatus", is("Completed"),
+                        "_embedded.import-summary[0].validCount", is(1),
+                        "_embedded.import-summary[0].invalidCount", is(2))
+                .extract().path("_embedded.import-summary[0].id").toString());
+
+        final String csv = given()
+                .accept("text/csv")
+                .queryParam("importSummaryId", importSummaryId)
+                .when()
+                .get("/csv-export")
+                .body()
+                .print();
+        String[] csvFields = csv.split(",");
+
+        int numberOfRows = (int) Arrays.stream(csvFields).filter("\n"::equals).count();
+        assertEquals(2, numberOfRows);
+
+        assertEquals(1, (int) Arrays.stream(csvFields).filter("Import-app-9"::equals).count());
+        assertEquals(1, Arrays.stream(csvFields).filter(f -> f.startsWith("\"Very-long-app-name-name-")).count());
+
+        // Clean test data to not alter other tests execution
+        // Remove the successfully imported 'Import-app-8' application
+        final long importApp8Id = Long.parseLong(given()
+                .queryParam("name", "Import-app-8")
+                .accept(ContentType.JSON)
+                .when()
+                .get("/application")
+                .then()
+                .statusCode(200)
+                .body("size()", is(1))
+                .extract()
+                .path("[0].id")
+                .toString());
+
+        given()
+                .accept(ContentType.JSON)
+                .pathParam("id", importApp8Id)
+                .when()
+                .delete("/application/{id}")
+                .then()
+                .statusCode(204);
+
+        // Remove the import summary record (on cascade also the ApplicationImport entities will be deleted)
+        given()
+                .accept(ContentType.JSON)
+                .pathParam("id", importSummaryId)
+                .when()
+                .delete("/import-summary/{id}")
+                .then()
+                .statusCode(204);
+    }
+}
diff --git a/src/test/java/io/tackle/applicationinventory/services/issues/NativeIssueTACKLE268IT.java b/src/test/java/io/tackle/applicationinventory/services/issues/NativeIssueTACKLE268IT.java
@@ -0,0 +1,6 @@
+package io.tackle.applicationinventory.services.issues;
+
+import io.quarkus.test.junit.NativeImageTest;
+
+@NativeImageTest
+public class NativeIssueTACKLE268IT extends IssueTACKLE268Test {}
diff --git a/src/test/resources/long_characters_columns.csv b/src/test/resources/long_characters_columns.csv
@@ -0,0 +1,4 @@
+Record Type 1,Application Name,Description,Comments,Business Service,Tag Type 1,Tag 1,Tag Type 2,Tag 2,Tag Type 3,Tag 3,Tag Type 4,Tag 4,Tag Type 5,Tag 5,Tag Type 6,Tag 6,Tag Type 7,Tag 7,Tag Type 8,Tag 8,Tag Type 9,Tag 9,Tag Type 10,Tag 10,Tag Type 11,Tag 11,Tag Type 12,Tag 12,Tag Type 13,Tag 13,Tag Type 14,Tag 14,Tag Type 15,Tag 15,Tag Type 16,Tag 16,Tag Type 17,Tag 17,Tag Type 18,Tag 18,Tag Type 19,Tag 19,Tag Type 20,Tag 20
+,Import-app-8,Food vendors maintain,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
+,Import-app-9,Description for imported app,Food vendors maintain their meFood vendors maintain their menu and price data which is curated and presented to customers via OrderhubFood vendors maintain their menu and price data which is curated and presented to customers via OrderhubFood vendors maintain their menu and price data which is curated and presented to customers via OrderhubFood vendors maintain their menu and price data which is curated and presented to customers via OrderhubFood vendors maintain their menu and price data which is curated and presented to customers via Orderhubnu and price data which is curated and presented to customers via Orderhub,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
+,Very-long-app-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name-name,Description,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,