Merge pull request #22 from SasinduDilshara/development-revamp

Revamp the CSV public APIS

README.md

@@ -12,14 +12,14 @@ The Ballerina CSV Data Library is a comprehensive toolkit designed to facilitate
 ## Features
 
 - **Versatile CSV Data Input**: Accept CSV data as a string, byte array, or a stream and convert it into a subtype of ballerina records or lists.
-- **CSV to anydata Value Conversion**: Transform CSV data into expected type which is subtype of ballerina records or lists.
-- **Projection Support**: Perform selective conversion of CSV data subsets into ballerina records or lists values through projection.
+- **CSV to anydata Value transformation**: Transform CSV data into expected type which is subtype of ballerina record arrays or anydata arrays.
+- **Projection Support**: Perform selective conversion of CSV data subsets into ballerina record array or anydata array values through projection.
 
 ## Usage
 
 ### Converting CSV string to a record array
 
-To convert a CSV string into a record value, you can use the `parseStringToRecord` function from the library. The following example demonstrates how to transform a CSV document into an array of records.
+To convert a CSV string into a record array value, you can use the `parseString` function from the library. The following example demonstrates how to transform a CSV document into an array of records.
 
 ```ballerina
 import ballerina/data.csv;
@@ -36,7 +36,7 @@ public function main() returns error? {
                                Clean Code,Robert C. Martin,2008
                                The Pragmatic Programmer,Andrew Hunt and David Thomas,1999`;
 
-    Book[] books = check csv:parseStringToRecord(csvString);
+    Book[] books = check csv:parseString(csvString);
     foreach var book in books {
         io:println(book);
     }
@@ -45,7 +45,7 @@ public function main() returns error? {
 
 ### Converting external CSV document to a record value
 
-For transforming CSV content from an external source into a record value, the `parseStringToRecord`, `parseBytesToRecord`, `parseStreamToRecord`, `parseStringToList`, `parseBytesToList`and `parseStreamToList` functions can be used. This external source can be in the form of a string or a byte array/byte-block-stream that houses the CSV data. This is commonly extracted from files or network sockets. The example below demonstrates the conversion of an CSV value from an external source into a record value.
+For transforming CSV content from an external source into a record value, the `parseString`, `parseBytes` and `parseStream` functions can be used. This external source can be in the form of a string or a byte array/byte-block-stream that houses the CSV data. This is commonly extracted from files or network sockets. The example below demonstrates the conversion of an CSV value from an external source into a record value.
 
 ```ballerina
 import ballerina/data.csv;
@@ -60,12 +60,12 @@ type Book record {
 public function main() returns error? {
     // Read the CSV content as a string
     string csvContent = check io:fileReadString("path/to/file.csv");
-    Book[] book = check csv:parseStringToRecord(csvContent);
+    Book[] book = check csv:parseString(csvContent);
     io:println(book);
 
     // Read the CSV content as a stream
     stream<byte[], io:Error?> csvStream = check io:fileReadBlocksAsStream("path/to/file.csv");
-    Book[] book2 = check csv:parseStreamToRecord(csvStream);
+    Book[] book2 = check csv:parseStream(csvStream);
     io:println(book2);
 }
 ```
@@ -74,8 +74,8 @@ Make sure to handle possible errors that may arise during the file reading or CS
 
 ## CSV to record array/anydata array of array representation
 
-The CSV Object can be represented as a value of type record/map array or string array of array in Ballerina which facilitates a structured and type-safe approach to handling CSV data.
-The conversion of CSV data to subtype of record array or anydata array of array representation is a fundamental feature of the library.
+The CSV Object can be represented as a value of type `record/map array` or `string array of array` in Ballerina, which facilitates a structured and type-safe approach to handling CSV data.
+The conversion of CSV data to subtype of `record array` or `anydata array of array` representation is a fundamental feature of the library.
 
 ```ballerina
 import ballerina/data.csv;
@@ -88,9 +88,19 @@ type Book record {
 
 public function main() returns error? {
     string[][] bookArray = [["Clean Code","2008"],["Clean Architecture","2017"]];
+    Book[] bookRecords = [{name: "Clean Code", year: 2008}, {name: "Clean Architecture", year: 2017}];
 
-    Book[] author = check csv:parseListAsRecordType(bookArray, customHeaders = ["name", "year"]);
-    io:println(author);
+    // Parse and output a record array from a source of string array of arrays.
+    Book[] books = check csv:parseList(bookArray, {customHeaders: ["name", "year"]});
+    io:println(books);
+
+    // Parse and output a tuple array from a source of string array of arrays.
+    [string, int][] books2 = check csv:parseList(bookArray, {customHeaders: ["name", "year"]});
+    io:println(books2);
+
+    // Transform CSV records to a string array of arrays.
+    [string, int][] books3 = check csv:transform(bookRecords);
+    io:println(books3);
 }
 ```
 
@@ -122,7 +132,7 @@ public function main() returns error? {
 
     // The CSV data above contains publisher and year fields which are not 
     // required to be converted into a record field.
-    Book[] book = check csv:parseRecordAsRecordType(csvContent);
+    Book[] book = check csv:transform(csvContent);
     io:println(book);
 }
 ```

ballerina-tests/build.gradle

@@ -30,8 +30,9 @@ def testCommonTomlFilePlaceHolder = new File("${project.rootDir}/build-config/re
 def ballerinaDist = "${project.rootDir}/target/ballerina-runtime"
 def distributionBinPath =  "${ballerinaDist}/bin"
 def testCoverageParam = "--code-coverage --coverage-format=xml --includes=io.ballerina.lib.data.csvdata.*:ballerina.*"
-def testPackages = ["user-config-tests", "type-compatible-tests", "parse-string-array-types-tests", "unicode-tests", "constraint-validation-tests",
-                    "parse-list-types-tests", "parse-record-types-tests", "parse-string-record-types-tests", "union-type-tests"]
+def testPackages = ["union-type-tests", "user-config-tests", "parse-string-record-types-tests", 
+            "parse-string-array-types-tests", "parse-list-types-tests", "parse-record-types-tests", 
+            "type-compatible-tests", "unicode-tests", "constraint-validation-tests"]
 def testCommonPackage = "csv-commons"
 
 def stripBallerinaExtensionVersion(String extVersion) {

ballerina-tests/constraint-validation-tests/Ballerina.toml

@@ -13,4 +13,4 @@ version = "0.1.0"
 graalvmCompatible = true
 
 [build-options]
-graalvmBuildOptions = "-H:IncludeLocales=en_US,fr_FR"
+graalvmBuildOptions = "-H:+IncludeAllLocales"

ballerina-tests/constraint-validation-tests/tests/constraint_validation_test.bal

@@ -30,54 +30,100 @@ type ConstrainedRec record {
     string b;
 };
 
+@constraint:Array {length: 4}
+type ConstrainedList int[][];
+
+@test:Config
+function testConstraintWithLists() returns error? {
+    ConstrainedList|csv:Error cList1 = csv:parseString(string `1 
+                                            2 
+                                            3 
+                                            4`, {header: false, customHeadersIfHeaderAbsent: ["a", "b", "c", "d"]});
+    test:assertEquals(cList1, [[1], [2], [3], [4]]);
+
+    cList1 = csv:parseString(string `1 
+                                     2 
+                                     3
+                                     4
+                                     5
+                                     6`, {header: false, customHeadersIfHeaderAbsent: ["a", "b", "c", "d", "e", "f"]});
+    test:assertTrue(cList1 is csv:Error);
+    test:assertTrue((<error>cList1).message().startsWith("Validation failed")
+                    && (<error>cList1).message().includes("length"));
+
+    cList1 = csv:parseString(string `1 
+                                     2 
+                                     3
+                                     4
+                                     5
+                                     6`, {header: false, customHeadersIfHeaderAbsent: ["a", "b", "c", "d", "e", "f"], enableConstraintValidation: false});
+    test:assertEquals(cList1, [[1], [2], [3], [4], [5], [6]]);
+
+    cList1 = csv:transform([{"a": 1}, {"a": 1}, {"a": 1}, {"a": 1}], {});
+    test:assertEquals(cList1, [[1], [1], [1], [1]]);
+
+    cList1 = csv:transform([{"a": 1}, {"a": 1}, {"a": 1}, {"a": 1}, {"a": 1}, {"a": 1}], {});
+    test:assertTrue(cList1 is csv:Error);
+    test:assertTrue((<error>cList1).message().startsWith("Validation failed")
+                    && (<error>cList1).message().includes("length"));
+
+    cList1 = csv:parseList([["1"], ["2"], ["3"], ["4"]], {customHeaders: ["a", "b", "c", "d"]});
+    test:assertEquals(cList1, [[1], [2], [3], [4]]);
+
+    cList1 = csv:parseList([["1"], ["2"], ["3"], ["4"], ["5"], ["6"]], {customHeaders: ["a", "b", "c", "d"]});
+    test:assertTrue(cList1 is csv:Error);
+    test:assertTrue((<error>cList1).message().startsWith("Validation failed")
+                    && (<error>cList1).message().includes("length"));
+}
+
 @test:Config
 function testConstraintWithRecords() returns error? {
-    ConstrainedRec[]|csv:Error cRec1 = csv:parseStringToRecord(string `a,b
+    ConstrainedRec[]|csv:Error cRec1 = csv:parseString(string `a,b
                 4,abc
                 3, cde`);
     test:assertEquals(cRec1, [{a: 4, b: "abc"}, {a: 3, b: "cde"}]);
 
-    ConstrainedRec[]|csv:Error cRec1_2 = csv:parseStringToRecord(string `a,b
+    ConstrainedRec[]|csv:Error cRec1_2 = csv:parseString(string `a,b
                 4,abc
                 3, cde`, {enableConstraintValidation: false});
     test:assertEquals(cRec1_2, [{a: 4, b: "abc"}, {a: 3, b: "cde"}]);
 
-    cRec1 = csv:parseStringToRecord(string `a,b
+    cRec1 = csv:parseString(string `a,b
                 4,abc
                 11, cde`);
     test:assertTrue(cRec1 is csv:Error);
     test:assertTrue((<error>cRec1).message().startsWith("Validation failed")
                     && (<error>cRec1).message().includes("maxValue"));
 
-    cRec1 = csv:parseStringToRecord(string `a,b
+    cRec1 = csv:parseString(string `a,b
                 4,abc
                 5, "b"`, {});
     test:assertTrue(cRec1 is csv:Error);
     test:assertTrue((<error>cRec1).message().startsWith("Validation failed")
                     && (<error>cRec1).message().includes("minLength"));
 
-    cRec1 = csv:parseRecordAsRecordType([{"a": 4, "b": "abc"}, {"a": 3, "b": "cde"}], {});
+    cRec1 = csv:transform([{"a": 4, "b": "abc"}, {"a": 3, "b": "cde"}], {});
     test:assertEquals(cRec1, [{a: 4, b: "abc"}, {a: 3, b: "cde"}]);
 
-    cRec1 = csv:parseRecordAsRecordType([{"a": 4, "b": "abc"}, {"a": 11, "b": "cde"}], {});
+    cRec1 = csv:transform([{"a": 4, "b": "abc"}, {"a": 11, "b": "cde"}], {});
     test:assertTrue(cRec1 is csv:Error);
     test:assertTrue((<error>cRec1).message().startsWith("Validation failed")
                     && (<error>cRec1).message().includes("maxValue"));
 
-    cRec1 = csv:parseRecordAsRecordType([{"a": 4, "b": "abc"}, {"a": 5, "b": "b"}], {});
+    cRec1 = csv:transform([{"a": 4, "b": "abc"}, {"a": 5, "b": "b"}], {});
     test:assertTrue(cRec1 is csv:Error);
     test:assertTrue((<error>cRec1).message().startsWith("Validation failed")
                     && (<error>cRec1).message().includes("minLength"));
 
-    cRec1 = csv:parseListAsRecordType([["4", "abc"], ["3", "cde"]], ["a", "b"]);
+    cRec1 = csv:parseList([["4", "abc"], ["3", "cde"]], {customHeaders: ["a", "b"]});
     test:assertEquals(cRec1, [{a: 4, b: "abc"}, {a: 3, b: "cde"}]);
 
-    cRec1 = csv:parseListAsRecordType([["4", "abc"], ["11", "cde"]], ["a", "b"]);
+    cRec1 = csv:parseList([["4", "abc"], ["11", "cde"]], {customHeaders: ["a", "b"]});
     test:assertTrue(cRec1 is csv:Error);
     test:assertTrue((<error>cRec1).message().startsWith("Validation failed")
                     && (<error>cRec1).message().includes("maxValue"));
 
-    cRec1 = csv:parseListAsRecordType([["4", "abc"], ["5", "b"]], ["a", "b"]);
+    cRec1 = csv:parseList([["4", "abc"], ["5", "b"]], {customHeaders: ["a", "b"]});
     test:assertTrue(cRec1 is csv:Error);
     test:assertTrue((<error>cRec1).message().startsWith("Validation failed")
                     && (<error>cRec1).message().includes("minLength"));

ballerina-tests/csv-commons/Dependencies.toml

@@ -5,7 +5,7 @@
 
 [ballerina]
 dependencies-toml-version = "2"
-distribution-version = "2201.9.0"
+distribution-version = "2201.10.0-20240801-104200-87df251c"
 
 [[package]]
 org = "ballerina"

ballerina-tests/csv-commons/test_utils.bal

@@ -34,6 +34,6 @@ public function generateErrorMessageForInvalidHeaders(string value, string 'type
     return string `value '${value}' cannot be cast into '${'type}', because fields in '${'type}' or the provided expected headers are not matching with the '${value}'`;
 }
 
-public function generateErrorMessageForInvalidCustomHeader(string header) returns string{
-    return string `Invalid header value: '${header}'`;
+public function generateErrorMessageForInvalidCustomHeader(string header) returns string {
+    return string `Header '${header}' cannot be find in data rows`;
 }

ballerina-tests/parse-list-types-tests/Ballerina.toml

@@ -13,4 +13,4 @@ version = "0.1.0"
 graalvmCompatible = true
 
 [build-options]
-graalvmBuildOptions = "-H:IncludeLocales=en_US,fr_FR"
+graalvmBuildOptions = "-H:+IncludeAllLocales"

ballerina-tests/parse-list-types-tests/Dependencies.toml

@@ -5,7 +5,7 @@
 
 [ballerina]
 dependencies-toml-version = "2"
-distribution-version = "2201.9.0"
+distribution-version = "2201.10.0-20240801-104200-87df251c"
 
 [[package]]
 org = "ballerina"