The scan API does not properly handle newline characters in windows OS

[madhukaw]

Description:

$Subject

This is because the check here[1] becomes false in windows, but in ubuntu, it becomes true and works as expected.

Steps to reproduce:

Run this test case in windows:

@test:Config {
    groups: ["api"]
}
function testReadTOMLString() returns error? {
    string input = string `
        outer:
          inner: {outer: inner}
        seq:
          - - [[nested, sequence]]
        int: 1
        bool: true
        float: 1.1`;
    json output = check readString(input);

    test:assertEquals(output, {
        "outer": {
            "inner": {
                "outer": "inner"
            }
        },
        "seq": [[[["nested", "sequence"]]]],
        "int": 1,
        "bool": true,
        "float": <decimal>1.1
    });
}

Output:

Invalid character '
' for a '<yaml-document>'.

Affected Versions:
0.1.0

OS, DB, other environment details and versions:
Windows

[1] https://github.com/ballerina-platform/module-ballerina-yaml/blob/b6356efb53513341bc831fa02130867b41f63df5/modules/lexer/lexer.bal#L66

[SasinduDilshara]

Imo, this issue is occurred because windows read new line as \r\n and ubuntu take new line as \n.

• https://github.com/ballerina-platform/module-ballerina-yaml/blob/b6356efb53513341bc831fa02130867b41f63df5/modules/lexer/lexer.bal#L66
• https://github.com/ballerina-platform/module-ballerina-yaml/blob/b6356efb53513341bc831fa02130867b41f63df5/modules/lexer/scan.bal#L161

Above if checks only checks the ubuntu case, those are not check the windows case.

To resolve this we need to update lexer to handle \r\n and update above if cases to handle \r\n.

[nipunayf]

Currently, there is no API to determine the running OS during runtime in Ballerina. Waiting for an API as tracked here: #4931 to resolve this issue.

[github-actions]

This issue is NOT closed with a proper Reason/ label. Make sure to add proper reason label before closing. Please add or leave a comment with the proper reason label now.

      - Reason/EngineeringMistake - The issue occurred due to a mistake made in the past.
      - Reason/Regression - The issue has introduced a regression.
      - Reason/MultipleComponentInteraction - Issue occured due to interactions in multiple components.
      - Reason/Complex - Issue occurred due to complex scenario.
      - Reason/Invalid - Issue is invalid.
      - Reason/Other - None of the above cases.