Allow to load Github documents using stream

[Njuelle]

Hey,

Since I tried to load documents from a large Github repository using GithubRepoLoader. I ran on JavaScript heap out of memory error.
It could be nice to allow to load Github documents using stream

I started to work on a PR that allow this behaviour using stream with generators.

Proposed usage:

const loader = new GithubRepoLoader(
  "https://github.com/langchain-ai/langchainjs",
  {
    branch: "main",
    recursive: false,
    unknown: "warn",
    maxConcurrency: 3, // Defaults to 2
  }
);

const docs = [];
for await (const doc of loader.loadAsStream()) {
  docs.push(doc);
}

console.log({ docs });

wdyt?

[dosubot]

🤖

Hello @Njuelle,

Nice to meet you! I'm Dosu, a bot that's here to assist with your issues, answer questions, and help you contribute to the project. While we wait for a human maintainer, I'll be doing my best to provide you with assistance.

I see you've posted an issue and I'm currently reviewing it. I'll be providing a full response shortly. Hang tight!

[dosubot]

🤖

That sounds like a great idea! Loading Github documents using streams can help prevent memory issues when dealing with large repositories. Your proposed usage with generators looks good as well. It would be really helpful for users who need to load documents from a Github repository. Thank you for taking the initiative to work on a PR for this feature. I encourage you to continue working on it and submit the PR when you're ready. The maintainers will review it and decide if it should be merged. Keep up the good work!

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.