[Bug][Module Name] Enterprise Github Plugin Collects only X number of Data - Devlake Version: v1.0.1@e061ef2

[BhoopalanI]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

I have a github repo which has more than 1200 PRs. I'm trying to populate the data into my Devlake DB. However, Devlake collects only most recent 800 PRs. More than 400 PRs which are the oldest aren't collected. I tested this by quering the DB with the following queries:

select
*
from 
	_raw_github_api_pull_requests

select
*
from 
	pull_requests

I have also attached the log of the Github data collection.

Version: v1.0.1@e061ef2
oct_3_local.zip

What do you expect to happen

The Github Plugin is meant to collect all the data from oldest to most recent

How to reproduce

Please collect data from a large repository and test with the above mentioned query. The older data wouldn't be collected.

Anything else

No response

Version

v1.0.1@e061ef2

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

Hello @BhoopalanI! 👋 I'm here to help you debug, answer questions, and even guide you on becoming a contributor.

A similar issue was previously reported and closed. The problem with the Enterprise Github Plugin collecting only the most recent 800 pull requests is due to the API request configuration. The direction parameter is set to desc, which retrieves pull requests in descending order, starting from the most recent. The PageSize is set to 100, and the pagination logic likely limits the total number of pull requests collected to 800 due to the number of pages processed or a similar constraint ^[1].

To continue talking to Dosu, mention @dosu.

[klesh]

Please check if they were created earlier than Time After from your project under the Sync policy section.

[BhoopalanI]

@klesh It works after changing the Time After. Thank you. This can be considered closed.