significant more API calls for same code

[aspyker]

I'm researching deeper, but with 1.72 I get:

remaining calls 4364
about to look up repo EVCache
2016-06-28 21:28:20 INFO GithubAccess:53 - repo = EVCache, forks = 82, stars = 314
2016-06-28 21:28:20 DEBUG GithubAccess:56 - openIssues = 0, openPullRequests = 0
2016-06-28 21:28:22 DEBUG GithubAccess:107 - daysSinceLastCommit = 0
2016-06-28 21:28:23 DEBUG GithubAccess:111 - numContribitors = 16, contributorEmails = ArrayBuffer(smadappa, senugula, jkschneider, vuzilla, ScottMansfield, rspieldenner, elandau, gitter-badger, rdegnan, nadavc, aspyker, trigan-d, pauloricardomg, kedargsm, quidryan, Randgalt)
2016-06-28 21:28:24 DEBUG GithubAccess:146 - avg days to close 14 issues = 357 days
2016-06-28 21:28:24 DEBUG GithubAccess:128 - avg days to close 13 pull requests = 185 days
2016-06-28 21:28:24 DEBUG GithubAccess:96 - repo json = {"asOfISO":"2016-06-29T04:27:43Z","asOfYYYYMMDD":"2016-06-29","repo_name":"EVCache","public":true,"osslifecycle":"UNKNOWN","forks":82,"stars":314,"numContributors":16,"issues":{"openCount":0,"closedCount":14,"avgTimeToCloseInDays":357},"pullRequests":{"openCount":0,"closedCount":13,"avgTimeToCloseInDays":185},"commits":{"daysSinceLastCommit":0},"contributors":["smadappa","senugula","jkschneider","vuzilla","ScottMansfield","rspieldenner","elandau","gitter-badger","rdegnan","nadavc","aspyker","trigan-d","pauloricardomg","kedargsm","quidryan","Randgalt"]}
remaining calls 4341

resulting in 23 API calls

with 1.76, I get:

remaining calls 4640
about to look up repo EVCache
2016-06-28 21:22:22 INFO GithubAccess:53 - repo = EVCache, forks = 82, stars = 314
2016-06-28 21:22:22 DEBUG GithubAccess:56 - openIssues = 0, openPullRequests = 0
2016-06-28 21:23:01 DEBUG GithubAccess:107 - daysSinceLastCommit = 0
2016-06-28 21:23:03 DEBUG GithubAccess:111 - numContribitors = 16, contributorEmails = ArrayBuffer(smadappa, senugula, jkschneider, vuzilla, ScottMansfield, rspieldenner, elandau, gitter-badger, rdegnan, nadavc, aspyker, trigan-d, pauloricardomg, kedargsm, quidryan, Randgalt)
2016-06-28 21:23:04 DEBUG GithubAccess:146 - avg days to close 14 issues = 357 days
2016-06-28 21:23:04 DEBUG GithubAccess:128 - avg days to close 13 pull requests = 185 days
2016-06-28 21:23:04 DEBUG GithubAccess:96 - repo json = {"asOfISO":"2016-06-29T04:20:26Z","asOfYYYYMMDD":"2016-06-29","repo_name":"EVCache","public":true,"osslifecycle":"UNKNOWN","forks":82,"stars":314,"numContributors":16,"issues":{"openCount":0,"closedCount":14,"avgTimeToCloseInDays":357},"pullRequests":{"openCount":0,"closedCount":13,"avgTimeToCloseInDays":185},"commits":{"daysSinceLastCommit":0},"contributors":["smadappa","senugula","jkschneider","vuzilla","ScottMansfield","rspieldenner","elandau","gitter-badger","rdegnan","nadavc","aspyker","trigan-d","pauloricardomg","kedargsm","quidryan","Randgalt"]}
remaining calls 4409

resulting in 231 API calls

This is running this code:

https://github.com/Netflix/osstracker/blob/master/osstracker-scraper/src/main/scala/com/netflix/oss/tools/osstrackerscraper/GithubAccess.scala#L52

[aspyker]

This seems to be in the issue code, making getCommitInfo a NOOP results in:

1.76 - without issues
remaining calls 4993
about to look up repo EVCache
2016-06-28 21:44:49 INFO GithubAccess:53 - repo = EVCache, forks = 82, stars = 314
2016-06-28 21:44:50 DEBUG GithubAccess:56 - openIssues = 0, openPullRequests = 0
2016-06-28 21:44:50 DEBUG GithubAccess:147 - avg days to close 14 issues = 357 days
2016-06-28 21:44:51 DEBUG GithubAccess:129 - avg days to close 13 pull requests = 185 days
2016-06-28 21:44:51 DEBUG GithubAccess:96 - repo json = {"asOfISO":"2016-06-29T04:44:26Z","asOfYYYYMMDD":"2016-06-29","repo_name":"EVCache","public":true,"osslifecycle":"UNKNOWN","forks":82,"stars":314,"numContributors":0,"issues":{"openCount":0,"closedCount":14,"avgTimeToCloseInDays":357},"pullRequests":{"openCount":0,"closedCount":13,"avgTimeToCloseInDays":185},"commits":{"daysSinceLastCommit":0},"contributors":[]}
remaining calls 4989 (-4)

1.72 - without issues
remaining calls 4334
about to look up repo EVCache
2016-06-28 21:42:16 INFO GithubAccess:53 - repo = EVCache, forks = 82, stars = 314
2016-06-28 21:42:17 DEBUG GithubAccess:56 - openIssues = 0, openPullRequests = 0
2016-06-28 21:42:18 DEBUG GithubAccess:147 - avg days to close 14 issues = 357 days
2016-06-28 21:42:18 DEBUG GithubAccess:129 - avg days to close 13 pull requests = 185 days
2016-06-28 21:42:18 DEBUG GithubAccess:96 - repo json = {"asOfISO":"2016-06-29T04:41:51Z","asOfYYYYMMDD":"2016-06-29","repo_name":"EVCache","public":true,"osslifecycle":"UNKNOWN","forks":82,"stars":314,"numContributors":0,"issues":{"openCount":0,"closedCount":14,"avgTimeToCloseInDays":357},"pullRequests":{"openCount":0,"closedCount":13,"avgTimeToCloseInDays":185},"commits":{"daysSinceLastCommit":0},"contributors":[]}
remaining calls 4330 (-4)

[aspyker]

More granularity:

1.72
remaining calls 4740
about to look up repo EVCache
2016-06-28 21:58:31 INFO GithubAccess:53 - repo = EVCache, forks = 82, stars = 314
2016-06-28 21:58:31 DEBUG GithubAccess:56 - openIssues = 0, openPullRequests = 0
before listCommits as List remaining calls 4739 (-1)
before sortBy remaining calls 4732 (-8)
before calc of lastCommitDate/daysBetween remaining calls 4732 (-8)
2016-06-28 21:58:33 DEBUG GithubAccess:110 - daysSinceLastCommit = 0
before filter by defined author remaining calls 4732 (-8)
before map to login (and leave function) remaining calls 4716 (-24)
2016-06-28 21:58:36 DEBUG GithubAccess:116 - numContribitors = 16, contributorEmails = ArrayBuffer(smadappa, senugula, jkschneider, vuzilla, ScottMansfield, rspieldenner, elandau, gitter-badger, rdegnan, nadavc, aspyker, trigan-d, pauloricardomg, kedargsm, quidryan, Randgalt)
2016-06-28 21:58:36 DEBUG GithubAccess:151 - avg days to close 14 issues = 357 days
2016-06-28 21:58:37 DEBUG GithubAccess:133 - avg days to close 13 pull requests = 185 days
2016-06-28 21:58:37 DEBUG GithubAccess:96 - repo json = {"asOfISO":"2016-06-29T04:58:22Z","asOfYYYYMMDD":"2016-06-29","repo_name":"EVCache","public":true,"osslifecycle":"UNKNOWN","forks":82,"stars":314,"numContributors":16,"issues":{"openCount":0,"closedCount":14,"avgTimeToCloseInDays":357},"pullRequests":{"openCount":0,"closedCount":13,"avgTimeToCloseInDays":185},"commits":{"daysSinceLastCommit":0},"contributors":["smadappa","senugula","jkschneider","vuzilla","ScottMansfield","rspieldenner","elandau","gitter-badger","rdegnan","nadavc","aspyker","trigan-d","pauloricardomg","kedargsm","quidryan","Randgalt"]}
remaining calls 4713 (-27)

1.76
remaining calls 4982
about to look up repo EVCache
2016-06-28 21:53:41 INFO GithubAccess:53 - repo = EVCache, forks = 82, stars = 314
2016-06-28 21:53:41 DEBUG GithubAccess:56 - openIssues = 0, openPullRequests = 0
before listCommits as List remaining calls 4981 (-1)
before sortBy remaining calls 4974 (-8)
before calc of lastCommitDate/daysBetween remaining calls 4766 (-216)
2016-06-28 21:54:19 DEBUG GithubAccess:110 - daysSinceLastCommit = 0
before filter by defined author remaining calls 4766 (-216)
before map to login (and leave function) remaining calls 4750 (-232)
2016-06-28 21:54:22 DEBUG GithubAccess:116 - numContribitors = 16, contributorEmails = ArrayBuffer(smadappa, senugula, jkschneider, vuzilla, ScottMansfield, rspieldenner, elandau, gitter-badger, rdegnan, nadavc, aspyker, trigan-d, pauloricardomg, kedargsm, quidryan, Randgalt)
2016-06-28 21:54:23 DEBUG GithubAccess:151 - avg days to close 14 issues = 357 days
2016-06-28 21:54:23 DEBUG GithubAccess:133 - avg days to close 13 pull requests = 185 days
2016-06-28 21:54:23 DEBUG GithubAccess:96 - repo json = {"asOfISO":"2016-06-29T04:53:21Z","asOfYYYYMMDD":"2016-06-29","repo_name":"EVCache","public":true,"osslifecycle":"UNKNOWN","forks":82,"stars":314,"numContributors":16,"issues":{"openCount":0,"closedCount":14,"avgTimeToCloseInDays":357},"pullRequests":{"openCount":0,"closedCount":13,"avgTimeToCloseInDays":185},"commits":{"daysSinceLastCommit":0},"contributors":["smadappa","senugula","jkschneider","vuzilla","ScottMansfield","rspieldenner","elandau","gitter-badger","rdegnan","nadavc","aspyker","trigan-d","pauloricardomg","kedargsm","quidryan","Randgalt"]}
remaining calls 4747 (-235)

This with the following code:

  // TODO: Is there a faster way to only pull the last commit?
  def getCommitInfo(repo: GHRepository) : CommitInfo = {
    println(Console.RED + s"before listCommits as List remaining calls ${getRemainingHourlyRate()}" + Console.RESET)
    val commits = repo.listCommits().asList()
    println(Console.RED + s"before sortBy remaining calls ${getRemainingHourlyRate()}" + Console.RESET)
    val orderedCommits = commits.sortBy(_.getCommitShortInfo.getCommitter().getDate())
    println(Console.RED + s"before calc of lastCommitDate/daysBetween remaining calls ${getRemainingHourlyRate()}" + Console.RESET)
    val lastCommitDate = orderedCommits(orderedCommits.length - 1).getCommitShortInfo().getCommitter().getDate()
    //logger.debug(s"commits, first = ${orderedCommits(0).getSHA1}, last = ${orderedCommits(orderedCommits.length - 1).getSHA1()}")
    val daysSinceLastCommit = daysBetween(lastCommitDate, new Date())
    logger.debug(s"daysSinceLastCommit = ${daysSinceLastCommit}")

    println(Console.RED + s"before filter by defined author remaining calls ${getRemainingHourlyRate()}" + Console.RESET)
    val contributors = commits.filter { commit => Option(commit.getAuthor()).isDefined }
    println(Console.RED + s"before map to login (and leave function) remaining calls ${getRemainingHourlyRate()}" + Console.RESET)
    val contributorLogins = contributors.map(contributor => contributor.getAuthor().getLogin()).distinct
    logger.debug(s"numContribitors = ${contributorLogins.length}, contributorEmails = ${contributorLogins}")
    CommitInfo(commits.length, daysSinceLastCommit, contributorLogins.toList)
  }

[aspyker]

seems to be related to:

c0a05e0

The call to getCommitShortInfo in 1.72 seemed to return an already populated value whereas in 1.76 it does another API call. This explains why the number of calls went up by exactly the number of commits.

[aspyker]

@kohsuke I'm wondering if the commit I mentioned is correct. It doesn't seem like the pre-caching it does is important to getCommitShortInfo. This is limiting the use of your library at Netflix and I wanted to know if I should submit a PR to reverse the previous change.

[kohsuke]

Thanks for the great detective work! Indeed that was a problem. I fixed that in 1.77. My apologies for the delay.

[aspyker]

Thanks. Confirmed fix and updated Netflix osstracker. @kohsuke