search: increase default indexed-search timeout to 10s (#8720)

Previously zoekt would timeout after 3s by default. This is a very
conservative time, but on an established instance is enough. However, we have
had two new customers this week run into unstable result counts due to this
timeout. They needed to tune there resource allocations to fix the underlying
issue, but the user experience was poor and confusing. So we increase the
timeout to make it clear that search is slow rather than broken.

This new timeout is far greater than we expect zoekt to take. The reason we
use this timeout is we would rather be slow than inaccurate. Additionally if a
user opts in to some timeout settings, those override the default.

CHANGELOG.md

@@ -21,6 +21,7 @@ All notable changes to Sourcegraph are documented in this file.
 - A regression in repo-updater was fixed that lead to every repository's git clone being updated every time the list of repositories was synced from the code host. [#8501](https://github.com/sourcegraph/sourcegraph/issues/8501)
 - The default timeout of indexed search has been increased. Previously indexed search would always return within 3s. This lead to broken behaviour on new instances which had yet to tune resource allocations. [#8720](https://github.com/sourcegraph/sourcegraph/pull/8720)
 - Bitbucket Server older than 5.13 failed to sync since Sourcegraph 3.12. This was due to us querying for the `archived` label, but Bitbucket Server 5.13 does not support labels. [#8883](https://github.com/sourcegraph/sourcegraph/issues/8883)
+- The default timeout of indexed search has been increased. Previously indexed search would always return within 3s. This lead to broken behaviour on new instances which had yet to tune resource allocations. [#8720](https://github.com/sourcegraph/sourcegraph/pull/8720)
 
 ### Removed
 

cmd/frontend/graphqlbackend/textsearch_test.go

@@ -567,7 +567,7 @@ func Test_zoektSearchHEAD(t *testing.T) {
 				repos:           singleRepositoryRevisions,
 				useFullDeadline: false,
 				searcher:        &fakeSearcher{result: &zoekt.SearchResult{}},
-				since:           func(time.Time) time.Duration { return 4 * time.Second },
+				since:           func(time.Time) time.Duration { return 15 * time.Second },
 			},
 			wantFm:            nil,
 			wantLimitHit:      false,

cmd/frontend/graphqlbackend/zoekt.go

@@ -47,7 +47,7 @@ func zoektResultCountFactor(numRepos int, query *search.TextPatternInfo) int {
 
 func zoektSearchOpts(k int, query *search.TextPatternInfo) zoekt.SearchOptions {
 	searchOpts := zoekt.SearchOptions{
-		MaxWallTime:            3 * time.Second,
+		MaxWallTime:            10 * time.Second,
 		ShardMaxMatchCount:     100 * k,
 		TotalMaxMatchCount:     100 * k,
 		ShardMaxImportantMatch: 15 * k,