awaiting between creating a readline interface, and using async iterator will cause iterator to sometimes skip.

[Tatsujinichi]

• Version:v12.13.0
• Platform:Linux myuser 4.15.0-74-generic Proposal: Introduce CI builds to GitHub PR process #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Subsystem:
  readline.createInterface
  asyncIterator

What steps will reproduce the bug?

if you perform something like
https://github.com/nodejs/node/blob/2a7432dadec08bbe7063d84f1aa4a6396807305c/test/parallel/test-readline-async-iterators.js

async function testSimple() {
  for (const fileContent of testContents) {
    fs.writeFileSync(filename, fileContent);

    const readable = fs.createReadStream(filename);
    const rli = readline.createInterface({
      input: readable,
      crlfDelay: Infinity
    });

    const iteratedLines = [];
    for await (const k of rli) {
      iteratedLines.push(k);
    }

    const expectedLines = fileContent.split('\n');
    if (expectedLines[expectedLines.length - 1] === '') {
      expectedLines.pop();
    }
    assert.deepStrictEqual(iteratedLines, expectedLines);
    assert.strictEqual(iteratedLines.join(''), fileContent.replace(/\n/gm, ''));
  }
}

If you add some kind of await xxx() between creating the interface and iterating, the iterator will miss lines. In my case I input 100k lines from a file, then output those same lines to a new file. several thousand lines will go missing.

How often does it reproduce? Is there a required condition?

100%
Need to add some await async between creating the interface and using for await

What is the expected behavior?

not to miss iterations

What do you see instead?

Additional information

[devsnek]

readline isn't a stream, if you don't attach a handler you will miss events. that being said, exposing an async iterable doesn't make that very clear.

[BridgeAR]

As @devsnek outlined there's nothing actually wrong with the behavior. We might want to improve our docs when it comes to that?

[preyunk]

@devsnek @BridgeAR I would like to take up this issue, could you guide me through the process for achieving the same as I am a first time contributor.

[jfriend00]

Similar problem shown here: https://stackoverflow.com/questions/62885667/why-does-this-readline-async-iterator-not-work-properly.

I see only this output in the console:

before for() loop
finished
finally
done

The for await (const line1 of rl1) loop never goes into the for loop - it just skips right over it:

const fs = require('fs');
const readline = require('readline');
const { once } = require('events');

async function test(file1, file2) {
    try {
        const stream1 = fs.createReadStream(file1);
        await once(stream1, 'open');
        const rl1 = readline.createInterface({input: stream1, crlfDelay: Infinity});

        const stream2 = fs.createReadStream(file2);
        await once(stream2, 'open');
        const rl2 = readline.createInterface({input: stream2, crlfDelay: Infinity});

        console.log('before for() loop');
        for await (const line1 of rl1) {
            console.log(line1);
        }
        console.log('finished');
    } finally {
        console.log('finally');
    }
}

test("data/numbers.txt", "data/letters.txt").then(() => {
    console.log(`done`);
}).catch(err => {
    console.log('Got rejected promise:', err);
})

[jfriend00]

So, there is no intent to ever actually "fix" this. You're just going to document it as is and leave it that way forever? So, no asynchronous operations are permitted between creating the interface and consuming the async iterator to read the lines? What if you're trying to use this along with some other asynchronous stuff together? That's just a busted implementation IMO. Documenting the existing behavior is OK for a stop-gap, but not really OK for just sweeping the issue under the rug forever.

[Trott]

@nodejs/readline Should this be re-opened?

[govindrai]

I agree with @jfriend00. This behaves like an improper implementation. Even after following the guidelines to prevent loss of line data, we are facing issues where have a function that is doing some heavy async operations on the result of each line (i.e. each line may take 5 or minutes to complete) and what we get is readline running some lines and eventually hanging and not issuing any more lines -> the program hangs forever.

async function heavyAsyncOperationsPerLine() {
    const rl = readline.createInterface({ input: getReadStream() }); // this file has 800K lines
 // called directly after 
    for await (const line of rl) {
        await operation1()
        await operation2()
    }
}