Code Intelligence 3.20 Tracking issue

[aidaeology]

Plan

Goal: Get the top three most popular language indexers: lsif-go, lsif-java, lsif-clang stable by working directly on implementation with customers to identify and resolve any remaining issues that prevent the tool from being use in a wide release. This goal takes top priority for this milestone and each member of the team is leading the initiative on an indexer.

Goal: Perform research spikes of features such as Bazel integration (to improve monorepos support) in order to find the best approach, and scope the development effort for adding new core capabilities to code intel platform.

Goal: Improve the reliability and usability of code intel tools by triaging and resolving UI bugs and performance issues (i.e. the bundle manager process) for an improved developer experience.

Availability

Period is from 2020-08-20 to 2020-09-20. Please write the days you won't be working and the number of working days for the period. US/Canada Holidays during this period: Labor Day (Monday, Sept 7th)

• Aida: 21 days
• Eric: 21 days
• Noah: 21 days
• Garo: 10 days

Tracked issues

@Strum355: 12.00d

• 🚚 LSIF-Java Delivery #13017 12d
• Find references to common Java class method name without false-positives #3418

@aidaeology: 5.00d

• Perl code intelligence not working #8303 👩🐛
• Make bundle manager janitor more lenient during disaster recovery #12168 🧶
• update lsif.dev #12501
• Remove -endpoint from src-cli usage #11878 🧶
• Create successful and reproducible indexes of 20 OSS repos #12
• 🚚 LSIF-Go Delivery #13015 5d
• Updated code intel team process #1537
• Update code intel quick start #13333

@efritz: 15.50d

• Setup on-call rotation for code intel #13379
• UI Tooltips for asymmetric precision of hover and definition results #12706 0.5d
• 🚚 LSIF-Go Delivery #13015 5d
• lsif-go fails on zoekt #12487 🐛
• bug: no J2D or find-refs of switch-case blocks over an object's type #117 🐛
• codeintel: Investigate lsif-go memory issues #13184 2d 🐛
• RFC 199: Implement user code executor for code intel indexer #12307
• RFC 199: Deploy new indexer service #12709
• RFC 199: Replace docker commands with firecracker commands #12708 5d
• RFC 199: Deploy indexer service that uses firecracker #13382
• RFC 199: Gather metrics from indexer service #13494
• system includes are wonky on macos 10.15 #2 🐛
• Slow Precise Code-Intel Tooltips in C/C++ Code #13733
• codeintel/design: Propose replacing SQLite with Postgres #13070 2d 🕵️
• codeintel/design: Go to definition of unindexed/uncloned code #13137 0.5d 🕵️
• codeintel: Review alerts #13074 0.5d 🧶
• Hovering on a specific class sometimes takes a while to load all the references #12569 🐛
• codeintel: Reduce data returned from code intel ranges requests #13746
• codeintel: Wrap index commands in a firecracker VM #13421

@gbrik: 5.00d

• 🚚 LSIF-clang Delivery #12349 4d
• Investigate effort for Bazel integration #13202 1d 🕵️
• no output produced for seemingly well-formed compile_commands.json #4 🐛
• doesn't build on ubuntu 18.04 #3 🐛
• doesn't work on arch linux #1 🐛
• get cmake to find LLVM and Clang library paths automatically #10
• bug: files with no content other than namespaces produces invalid lsif output #9 🐛
• clean up the repo structure and structure the documentation #11
• github.com/gabime/spdlog doesn't produce LSIF output for template definitions #14 🐛
• github.com/nlohmann/json on macOS fails #13 🐛
• infer project root automatically if not specified #15
• Create successful and reproducible indexes of 20 OSS repos #12

Legend

• 👩 Customer issue
• 🐛 Bug
• 🧶 Technical debt
• 🛠️ Roadmap
• 🕵️ Spike
• 🔒 Security issue
• Pull Request

[aidaeology]

@efritz @Strum355 @gbrik please add your availability and list any days off.

[aidaeology]

@gbrik Please update this issue with your availability during the period for this milestone.

[efritz]

Captain's log; Stardate 2020/08/24-2020/08/28

Worked on addressing memory issues in lsif-go for a customer that had some particular OOM issues in a (somewhat) constrained environment (index machine has 12GB). Released lsif-go v1.1.4 to reduce memory requirements.

Worked with @Strum355 with lsif-test utilities to help him out with writing lsif-java and other devs writing indexers in the future. Did this in a few pairing sessions this week.

Also put additional effort into making the precise code intel indexer service use firecracker VMs instead of raw containers on the host. This has been a bit slow-moving as it takes a full merge+buildkite cycle to test out any changes on the target machine. Not stuck on anything currently - just trying to squash the bugs as they come.

Additionally scheduled meetings with 2 of the 3 companies listed for lsif-go delivery.

Captain's log; Stardate 2020/08/31-2020/09/04

Plan to meet with 2 (or more) of the companies we've scheduled for lsif-go delivery and get feedback on the lsif-go tool and uploading process. These meetings will also involved @aidaeology and @macraig (when available).

Will continue any development effort available on RFC 199 to get the indexer service to interact properly with Firecracker on the compute host.

[Strum355]

Completed 2020/08/24-28
Submitted 4 PRs to Spoon that are undergoing review that have helped bring lsif-java to a point where it can parse google/gson and pass lsif-validator

Added annotation and enum declaration emitting to lsif-java

Added an improved (over Microsoft's lsif-util) lsif graphviz dot visualizer to sourcegraph/lsif-test with support for only outputting a subgraph from an ID to a user defined distance with @efritz

Planned 2020/08/31-09/04
Get spoon PRs closer to being merged or merged

Submit more PRs to spoon as they arise from exceptions caused in lsif-java, which will increase the stability across other repos

Investigate paths for improved gradle handling for improved stability

[shrouxm]

Completed 2020/08/24-28

Came back from break, added support to lsif-go for indexing a subset of files, added the API/plumbing for asking an sg instance to accept and process an incremental index.

Planned 2020/08/31-09/04

Validate the incremental indexer (maybe need to factor out some of the worker processor code?)

Bazel research spike, focusing on go and clang, java if spare time.

Start tackling lsif-clang bugs.

[aidaeology]

@gbrik Please update this issue with your availability during the period for this milestone.

[efritz]

Completed 2020/08/31-2020/09/04

Met with two site-admins about adopting LSIF for Go. Fixed a number of lsif-go bugs and feature suggestions that arose from those meetings. Paired with Noah and Garo a fair bit on LSIF utilities, the precise code intel indexer, and lsif-clang.

Planned 2020/09/7-2020/09/11

Try to ship the lsif-go fixes and have another round of meetings with our delivery targets. Start looking into SQLite -> Postgres migration in slack time.

[shrouxm]

Completed 2020/08/31-2020/09/04

Incremental indexing:

• Finished prototype of lsif-go incremental indexing (speed increase is highly project dependent, but I saw 10x package loading speedup and 100x conversion speed increase on aws-sdk-go).
• Wired up the code which determines which files to perform the incremental index on.
• For reference, the only remaining work is to wire up the function that merges precise intel DBs, and measure accuracy. But that's on the backburner in favor of lsif-clang.

lsif-clang:

• Filed a bunch of issues detailing plan for pushing lsif-clang forward.
• Fixed several outstanding bugs.
• Successfully uploaded c++ LSIF indexes for several more OSS projects.
• Paired with eric & aida on running the indexer on more OSS projects.

Planned 2020/09/07-2020/09/11

• Finish the new format for lsif-clang documentation, which blocks:
  • Bug dev-chat to help with the effort to run lsif-clang on many OSS projects.
• Continue pushing through the lsif-clang bugs
• Document any weird flags or tips on getting a working compilation_database.json based on the OSS projects being indexed.
• Do the C++ part of the bazel research spike (for c++, bazel is just as important a target as any other build system, as opposed to other langs where it's more niche compared to the language's official package manager).

[Strum355]

will be updated on Sunday when caught up for friday

Completed 2020/08/31-2020/09/04
Got 2 Spoon PRs merged (INRIA/spoon#3552, INRIA/spoon#3549)
1 further has been improved+is under review by Spoon maintainers INRIA/spoon#3553 (comment)
Submitted another PR to Spoon to vastly improve the reliability of Spoons pom.xml handling (taking into account hierarchies et al) INRIA/spoon#3562
Submitted follow-up PR to allow for consuming/indexing maven modules that are only specified in certain maven profiles, which should decrease the number of Java files not indexed in certain maven projects INRIA/spoon#3569
Paired with Eric on Firecracker to allow for caching docker images on local registry

<space reserved>

Planned 2020/09/07-2020/09/11
More Spoon exceptions to be addressed, all day every day
Follow up session with Eric to fix an issue we came across wrt default docker bridge not allowing container DNS discovery
Gradle investigation depending on progress tomorrow

[aidaeology]

Seems like a lot of lsif-java issues are due to problems in our dependency on Spoon lib - if this pattern keeps up after this set of resolutions we need to discuss whether we should be continuing using a lib that is proving unreliable.

[efritz]

Planned 2020/09/7-11

• Re-met with a few customers about lsif-go delivery.
• Saw to a few incidental fixes to auto indexing, namely https://github.com/sourcegraph/sourcegraph/pull/13701.
• Wrote initial draft of RFC 235 to propose moving SQLite data into Postgres. Wrote a test migration script to judge the size and performance of such an instance. Looks like it will be fine!
• Spent a day to determine the root cause of https://github.com/sourcegraph/sourcegraph/issues/13733 which seemed to be blocking lsif-clang delivery. Discovered root cause and several other bugs in the process (see ticket conversation).

Planned 2020/09/14-18

• Review alerts for code intel services.
• Continue to meet with customers about lsif-go delivery.
• Get RFC 235 as far along as I can before the next milestone (hopefully we can start implementing it in 3.21).
• Begin to propose solutions (in the form of an RFC) for https://github.com/sourcegraph/sourcegraph/issues/13137.

[Strum355]

Completed 2020/09/07-2020/09/11

• All outstanding Spoon PRs merged, no official release yet
• Brief investigation into https://github.com/javaparser/javaparser with Eric showed promise
• Hack session with Eric on adding non-default Docker bridge support to Ignite didn't bear immediate results, Go docker client confirmed bad 👎
• Got working alternative Gradle->Maven Gradle script to work for Netflix public repos. 😰

Planned 2020/09/14-2020/09/18

• Try address Netflix public repo issues
• Test on more Netflix public repos
• Start javaparser RFC?

[shrouxm]

Completed 2020/09/07-2020/09/11

• switched focus to resolving customer issues with lsif-clang, they got an index uploaded but performance was poor so i helped eric debug that
• rewrote lsif-clang documentation (still waiting for someone to test it to be merged, anyone with a macOS please run through the new instructions on the simplify-cmake branch!)
• continued to work through OSS projects with indexing, uploaded an index for 18M LoC of LLVM, which required upping the worker's memory limit

Planned 2020/09/14-2020/09/18

• bazel research spike
• work through 5 more OSS projects with lsif-clang
• merge lsif-clang doc updates
• close as many open lsif-clang issues as possible

[davejrt]

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.20 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly.
When in doubt, reach out!

Thank you

[efritz]

Next milestone at https://github.com/sourcegraph/sourcegraph/issues/13987.