Search: 3.18 Tracking issue

[attfarhan]

Plan

Our Q2 OKRs are:

• allow users to search across contexts of their organization’s code as outlined in RFC 136.
• Sourcegraph.com WAU grows by 5x to N6 WAU.

The problems we want to solve this iteration:

1. New Sourcegraph users are not given onramps for successful onboarding.

   Solution: provide a guided way for users to search relevant repositories via repository groups pages.

2. Sourcegraph is missing familiar search features found in alternative solutions like OpenGrok. These missing features have actively turned away potential users and customers:

   I use them [boolean operators] every single day and the lack of support in Sourcegraph immediately disqualified it for us.

   There is a longstanding history of requests for this feature and it is a clear gap in our product [1, 2, 3]
   Solution: implement full hierarchical search with AND/OR queries behind a feature flag.

3. Customers have asked us for a solution to search across older versions of their code. These code states live on, because they map to enterprise releases that are still being used by their customers. Some of these requests come from customers that have been on OpenGrok previously, and have spun up multiple OpenGrok instances to allow them to search old versions of code. They want to use Sourcegraph, but want a seamless solution to do so.

   Solution: enable indexing on non-main branches so searches using version contexts are fast. We have built the version contexts feature without indexing in the previous iteration.

Availability

Period is from June 22 to July 17 (20 days). Please write the days you won't be working and the number of working days for the period.

• @attfarhan : 15d (out July 6 - 10)
• @rvantonder: 19d (I will likely take 1 or 2d off between July 11 and July 24, but not committing to this yet).
• @asdine: 20d
• @keegancsmith: TODO

Workload

@unassigned

• Surface alert if type: other than file is used with stable: #9715
• Support or-expressions for repo: and file: parameters #11009

@attfarhan

• "Expand/collapse all" buttons not working #11596 🐛
• Border-radius wrong in repogroup page search query input groups #11955 🐛
• Fancy quotes on repogroup pages #11953 🐛
• Kubernetes repogroup icon is not transparent, looks bad in dark theme #11956 🐛
• Feedback for Custom Search Pages and new Homepage Design #11922

@keegancsmith: 10.00d

• Index multiple (non-master) branches #6728 10d 👩

@rvantonder: 5.00d

• Migrate all search query processing to new parser #10844 2.5d
• search: buggy handling of escapable characters in quoted strings for field values #10442 🐛
• Unable to perform a literal search with a query that contains a : #10171
• Add structural search help message #11593 0.5d
• Search inputs that don't work but should #8780
• Expose and/or query parsing result through GQL API #9979 1.5d
• Implement and/or-expressions for file and repo search (AKA hierarchical search) #9974 👩
• Index repos for search page repogroups on sourcegraph.com #11962 0.5d

Legend

• 👩 Customer issue
• 🐛 Bug
• 🧶 Technical debt
• 🛠️ Roadmap
• 🕵️ Spike
• 🔒 Security issue
• Pull Request

[rvantonder]

Last week

• QA testing and major bug fix to release and/or operators in 3.17
• Blog video for and/or operators in 3.17
• Search docs (still work in progress, almost done)
• Discussions, ideas, contributions for curated repogroups / searches / pages

This week

• Planning
• Merge search docs improvements
• Expose query parse result in GQL for web frontend work/UI improvements
• Some tech debt for existing and/or queries, e.g., alerts
• Search optimization for 'or'-operators on constant values (transform to regex)
• With what's left, start to implement full hierarchical search queries

[attfarhan]

Last week

• Researched react-bootstrap in response to https://github.com/sourcegraph/sourcegraph/pull/11404. Started replacing components to make sure it's a suitable replacement.
• Lots of discussion on RFC 159 - Searching relevant repositories
• 3 coding challenges

This week

• RFC 159 MVP implementation (https://github.com/sourcegraph/sourcegraph/issues/11526)

[asdine]

Last week

• I was off on monday and tuesday
• Worked on a proposal for AND operator performance improvements (speed and memory usage) that shrinks the search scope incrementally.
• Researched alternative design based on PR (Improve AND operand performance #11559) comments

This week

• Finish the work on the AND operator
• Anything that comes next (perhaps Exclude directories from search results #6870 or Zoekt improvements?)

[felixfbecker]

@sourcegraph/search could https://github.com/sourcegraph/sourcegraph/issues/10820 potentially make it on the 3.18 plan? 🙂

[rvantonder]

@felixfbecker thanks for raising. That would need at least part of the hierarchical search work from syntax perspective (so I will be doing groundwork for it). Will be doing planning today and maybe we can squeeze it in, depends a little on how much work needs to happen in the backend.

[nicksnyder]

@attfarhan Can you please add a section at the top to describe "why" we are working on these things and how these things tie back to OKRs.

Here is an example for code intel: https://github.com/sourcegraph/sourcegraph/issues/11412

[rvantonder]

Last week

Added planning items, merge search query language reference, added search alerts, work for annotating queries with ranges and exposing these via GQL, interviews. Made progress on simplifying or-operators but the PR isn't ready yet, needs refactors.

This week

• Refactor query code to support full spectrum of parse tree data via GQL and support or-operator simplification. Start prototyping full hierarchical search.
• Introduce a settings flag that uses the new query parser for all queries. The flag will be instrumental for migrating the processing smoothly and offering a fallback in case there are bugs.

[attfarhan]

Last week

• RFC 159 MVP implementation: https://github.com/sourcegraph/sourcegraph/pull/11783#issuecomment-651085368, tracking in https://github.com/sourcegraph/sourcegraph/issues/11526
• Reviewed proposals for example repogroups
• 2 coding exercises, candidate follow-up call

This week

• Make as much progress as possible towards getting RFC 159 MVP shipped. Will be out next week.

[keegancsmith]

Note: My main update will go on the cloud team tracker. I'll still post search related updates here, but today is mostly about search so cross-posting: https://github.com/sourcegraph/sourcegraph/issues/11494#issuecomment-651120375

[nicksnyder]

@attfarhan Thanks for adding the plan section to the description! I think the problem statements could be improved. Right now it reads like problem = !X, solution do X (the problem statement presupposes the solution). Instead, make this problem statement more user focused (e.g., when users do X then run into problem Y. solution => do Z to solve Y).

New Sourcegraph users are not given onramps for successful onboarding.

How do we know this is a problem, what evidence do we have, what exactly is failing?

Users can perform complex and nested search queries to achieve feature parity with OpenGrok.

Why is feature parity with OpenGrok important? (I know that it is, but we need to document why). E.g. customers X and Y (don't say names but link to them in hubspot) decided to not use Sourcegraph because we didn't support Z.

Allow organizations to seamlessly search across contexts of their organizations code

This isn't a problem statement, it is a solution :)

[attfarhan]

Team update

cc @nicksnyder

OKR and work plan update:

We are tracking well towards our work plan. Last week, we made good progress towards the updated homepage and repository group pages. The first pass was implemented, and there is a draft PR open for the RFC 159 MVP. Rijnard is making progress towards enabling OR operators. Keegan is continuing to make progress on indexing non-main branches, he expects to spend this week continuing to implement this.

Notable shipped items:

• Major improvements to the search query language reference documentation
• Search alerts
• Work for annotating queries with ranges and exposing these via GQL

Nothing to flag in terms of items that are off-track or removed from the work plan.

[rvantonder]

Last week

Bulk of my time spent thinking about and restructuring parser logic (currently captured by #11947). A couple of wins happen at the same time (uniform validation, less code and more shared implementation, better for migration) and also better behavior for corner cases/heuristics. Other than that, added a GQL endpoint for getting the parse tree, added onboarding items to #11779, gave feedback about search pages, investigating and helping us index repos on Sourcegraph.com.

This week

• Introduce a parser migration settings flag that uses the new query parser for all queries (didn't get to this last week, but it's a small piece of work).

• Prototype full hierarchical search.

[rvantonder]

Last week

• Helped with the search pages effort (repo indexing, feedback)
• Helped @stefanhengl get up to speed and working on search code and search: support globbing for selected keywords #12093
• Refactor to expose user settings in the right place for easing parser migration. Turns out it's not necessary but I didn't know b/c the Go GQL library does some magic.
• I started prototyping full hierarchical but it's not in a ready state for feature flagging for customers.

This week

I really wanted to have hierarchical search in shape for 3.18 release tag but it won't make it with the couple of extra items I worked on this week. This week I will continue the hierarchical search work--I actually expect it to be able to go live (behind feature flag) this week on Sourcegraph.com/cloud--it is just not ready for 3.18. Goal for 3.19 is to have this functionality on by default, and I expect that is still on course.

[stefanhengl]

Last week (= first week)

• Local development setup
• Attended sync meetings: Web, Search, Cloud, Distribution, Marketing, Code Intel
• 1:1 with: Tomas, Keegan, Pooja, Rijnard
• Started working on adding support for globbing of search field values: Support unix-style globbing patterns for search field values #10094
• Drafted RFC 198 (related to Support unix-style globbing patterns for search field values #10094)

This week

• Gather comments for RFC 198 (related to Support unix-style globbing patterns for search field values #10094)
• Continue work on Support unix-style globbing patterns for search field values #10094
• Attend more sync meetings: Sales, Campaign, Product/Design
• 1:1 with: Loic, Fahran
• Pick up next ticket from onboarding issue if possible

[attfarhan]

Last week

• Vacation

This week

• Fix outstanding issues with repogroup homepage + repogroup pages
• Work on cloud private code CTAs
• Start on tour implementation

[daxmc99]

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.18 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly.
When in doubt, reach out!

Thank you