[dbnode] Direct conversion of encoded tags to doc.Metadata #3087

vpranckaitis · 2021-01-14T09:07:56Z

What this PR does / why we need it:

Direct conversion from encoded tags to doc.Metadata to improve performance.

Special notes for your reviewer:

Does this PR introduce a user-facing and/or backwards incompatible change?:

NONE

Does this PR require updating code package or user-facing documentation?:

NONE

src/dbnode/storage/index/convert/convert_benchmark_test.go

linasm

Did not fully review it just yet (until we decide on which implementation is better).

linasm · 2021-01-14T09:47:47Z

src/x/serialize/decoder_fast.go

@@ -27,40 +27,76 @@ import (

 // TagValueFromEncodedTagsFast returns a tag from a set of encoded tags without
 // any pooling required.
+


Looks like the comment above got misplaced.

linasm · 2021-01-14T09:53:19Z

src/x/serialize/decoder_fast.go

@@ -27,40 +27,76 @@ import (

 // TagValueFromEncodedTagsFast returns a tag from a set of encoded tags without
 // any pooling required.
+
+type TagVisitor interface {
+	Length(n int)


Nit: I believe SetLength might be more intuitive here.

linasm · 2021-01-14T10:02:59Z

src/x/serialize/decoder_fast.go

 func TagValueFromEncodedTagsFast(
 	encodedTags []byte,
 	tagName []byte,
 ) ([]byte, bool, error) {
+	visitor := findTagValueVisitor{


Can we be sure this refactoring won't affect the performance of TagValueFromEncodedTagsFast?

linasm · 2021-01-14T10:10:14Z

src/x/serialize/decoder_fast.go

+
+func VisitTags(
+	encodedTags []byte,
+	visitor TagVisitor,


This is a nice pattern, but I'm afraid that using an interface here will cause allocation of any implementing struct (which we try to avoid at all cost in these code paths). You might be able to work around this by passing functions directly (instead of using an interface), but then the code will get uglier.
Let's see how it looks and performs (also I think you should check for the allocations, I think there are some benchmark/compiler options for this).

vpranckaitis · 2021-01-15T12:00:58Z

src/dbnode/storage/index/convert/convert_benchmark_test.go

+		require.NoError(b, err)
+	}
+}
+


BenchmarkFromSeriesIDAndTagIter-12 720519 1741 ns/op 1046 B/op 5 allocs/op BenchmarkFromSeriesIDAndTags-12 940041 1380 ns/op 806 B/op 2 allocs/op BenchmarkFromSeriesIDAndEncodedTags-12 1000000 1043 ns/op 806 B/op 2 allocs/op BenchmarkFromSeriesIDAndEncodedTags2-12 960764 1139 ns/op 806 B/op 2 allocs/op BenchmarkFromSeriesIDAndFastTagIter-12 1000000 1182 ns/op 806 B/op 2 allocs/op BenchmarkFromSeriesIDAndTagIter_TagDecoder-12 253219 4654 ns/op 903 B/op 4 allocs/op

linasm · 2021-01-15T15:23:02Z

~~@vpranckaitis could you please split this into two draft PRs? It has become difficult to follow what is the alternative to what.~~
Disregard this comment, I was able to figure out what belongs where :)

linasm · 2021-01-18T06:56:09Z

src/dbnode/storage/index/convert/convert.go

+
+	var (
+		byteOrder                = binary.LittleEndian
+		headerMagicNumber uint16 = 10101


Let's make it a global const (also worth changing the same value in src/x/serialize/types.go to be a const).

linasm · 2021-01-18T07:07:13Z

src/x/serialize/decoder_fast.go

-
-		bytesName := encodedTags[:numBytesName]
-		encodedTags = encodedTags[numBytesName:]
+func DecodeTag(encodedTags []byte) ([]byte, []byte, []byte, error) {


Nit: this takes a representation of multiple tags as an input, I think a different name (DecodeNextTag or smth.) might be slightly more appropriate.

linasm · 2021-01-18T07:08:44Z

src/x/serialize/decoder_fast.go

+	encodedTags = encodedTags[2:]

-		if len(encodedTags) < 2 {
-			return nil, false, fmt.Errorf("missing size for tag value: index=%d", i)
-		}
+	bytesName := encodedTags[:numBytesName]
+	encodedTags = encodedTags[numBytesName:]

-		numBytesValue := int(byteOrder.Uint16(encodedTags[:2]))
-		encodedTags = encodedTags[2:]
+	if len(encodedTags) < 2 {
+		return nil, nil, nil, fmt.Errorf("missing size for tag value")
+	}

-		bytesValue := encodedTags[:numBytesValue]
-		encodedTags = encodedTags[numBytesValue:]
+	numBytesValue := int(byteOrder.Uint16(encodedTags[:2]))
+	encodedTags = encodedTags[2:]


Just to be sure, please also benchmark the implementation where you would not slide the slice each time, but only use it's original form, and adjust the indices instead.

Did a benchmark, it's slightly slower.

linasm · 2021-01-18T07:10:49Z

src/x/serialize/decoder_fast.go

@@ -31,48 +31,71 @@ func TagValueFromEncodedTagsFast(
 	encodedTags []byte,
 	tagName []byte,
 ) ([]byte, bool, error) {
+	var (


As we are changing the implementation of this function with ambitious name (Fast), I believe we also need a benchmark for it to make sure it does not regress noticeably (or maybe there already is an existing benchmark for it?).

linasm · 2021-01-18T07:16:21Z

src/x/serialize/decoder_fast_iter.go

+	err error
+}
+
+func NewFastTagDecoder() id.SortedTagIterator {


The comment of SortedTagIterator: iterates over a set of tag pairs sorted by tag names.
This decoder returns tags in the same order they appear in encodedTags, can we be sure we can provide this guarantee?

can we be sure we can provide this guarantee?

No. I've used this interface here because it fits for this case and to show that there's already a similar use-case in the codebase (thin iterator on tags encoded in a byte array). The idea is that using a similar / matching interface might allow more code reuse in the future.

If we decide to go with the iterator approach, I'll either rename the interface or make a copy of it.

vpranckaitis · 2021-01-18T08:48:25Z

src/x/serialize/decoder_fast.go

+	encodedTags = encodedTags[2:]

-		if len(encodedTags) < 2 {
-			return nil, false, fmt.Errorf("missing size for tag value: index=%d", i)
-		}
+	bytesName := encodedTags[:numBytesName]
+	encodedTags = encodedTags[numBytesName:]

-		numBytesValue := int(byteOrder.Uint16(encodedTags[:2]))
-		encodedTags = encodedTags[2:]
+	if len(encodedTags) < 2 {
+		return nil, nil, nil, fmt.Errorf("missing size for tag value")
+	}

-		bytesValue := encodedTags[:numBytesValue]
-		encodedTags = encodedTags[numBytesValue:]
+	numBytesValue := int(byteOrder.Uint16(encodedTags[:2]))
+	encodedTags = encodedTags[2:]


Did a benchmark, it's slightly slower.

vpranckaitis · 2021-01-18T08:54:40Z

src/x/serialize/decoder_fast_iter.go

+	err error
+}
+
+func NewFastTagDecoder() id.SortedTagIterator {


can we be sure we can provide this guarantee?

No. I've used this interface here because it fits for this case and to show that there's already a similar use-case in the codebase (thin iterator on tags encoded in a byte array). The idea is that using a similar / matching interface might allow more code reuse in the future.

If we decide to go with the iterator approach, I'll either rename the interface or make a copy of it.

vpranckaitis · 2021-01-18T13:37:20Z

src/dbnode/storage/index/convert/decoder_fast_benchmark_test.go

+			testData[i].tagName)
+		require.NoError(b, err)
+	}
+}


BenchmarkTagValueFromEncodedTagsFast-12 5903398 214 ns/op 0 B/op 0 allocs/op BenchmarkTagValueFromEncodedTagsFast2-12 5129425 240 ns/op 0 B/op 0 allocs/op

vpranckaitis · 2021-01-18T13:38:16Z

src/x/serialize/decoder_fast.go

@@ -31,48 +31,71 @@ func TagValueFromEncodedTagsFast(
 	encodedTags []byte,
 	tagName []byte,
 ) ([]byte, bool, error) {
+	var (


linasm

LGTM with some nits.
Also, now that we have settled the question about alternative implementations, could you log the final benchmark results illustrating the improvement?

src/dbnode/storage/index/convert/convert.go

linasm · 2021-01-19T10:22:15Z

src/dbnode/storage/index/convert/convert_test.go

@@ -171,6 +172,113 @@ func TestFromSeriesIDAndTagIterReuseBytesFromSeriesId(t *testing.T) {
 	}
 }

+func TestFromSeriesIDAndEncodedTagsInvalid(t *testing.T) {


Nit: why not merge it into TestFromSeriesIDAndEncodedTagsInvalidEncodedTags? Initially I thought this was the only "invalid" test case that we have.

a5436e8#diff-4b7b04cd8028a198a21da071cd0a8bbcc9d2f7fd0f6f4bd896a73019eccdfdfeR223-R238

linasm · 2021-01-19T10:23:30Z

src/dbnode/storage/index/convert/convert_test.go

+func TestFromSeriesIDAndEncodedTagsValid(t *testing.T) {
+	var (
+		id   = ident.StringID("foo")
+		tags = ident.NewTags(
+			ident.StringTag("bar", "baz"),
+		)
+		encodedTags = toEncodedTags(t, tags)
+	)
+	d, err := convert.FromSeriesIDAndEncodedTags(id, encodedTags)
+	assert.NoError(t, err)
+	assertContentsMatch(t, id, tags.Values(), d)
+}
+
+func TestFromSeriesIDAndEncodedTagsReuseBytesFromSeriesId(t *testing.T) {


I think these two could be just a single test function for a happy path. TestFromSeriesIDAndEncodedTagsReuseBytesFromSeriesId verifies the logics in multiple cases, its coverage goes far beyond just byte reuse.

a5436e8#diff-4b7b04cd8028a198a21da071cd0a8bbcc9d2f7fd0f6f4bd896a73019eccdfdfeL175-R183

src/x/serialize/encoder.go

Co-authored-by: Linas Medžiūnas <[email protected]>

linasm · 2021-01-19T11:59:34Z

src/dbnode/storage/index/convert/convert.go

@@ -188,6 +190,76 @@ func FromSeriesIDAndTagIter(id ident.ID, tags ident.TagIterator) (doc.Metadata,
 	return d, nil
 }

+// FromSeriesIDAndEncodedTags converts the provided series id and encoded tags into a doc.Metadata.
+func FromSeriesIDAndEncodedTags(id ident.ID, encodedTags ts.EncodedTags) (doc.Metadata, error) {


Please change ident.ID to ident.BytesID. I've just realized that using an interface here will force an allocation on some of the code paths where we will be using this function.

Conflicts: src/dbnode/storage/index/convert/convert.go src/dbnode/storage/index/convert/convert_benchmark_test.go src/dbnode/storage/index/convert/convert_test.go src/x/serialize/decoder_fast.go

codecov · 2021-01-20T08:38:41Z

Codecov Report

Merging #3087 (a8e1368) into master (a8e1368) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3087   +/-   ##
=======================================
  Coverage    72.3%    72.3%           
=======================================
  Files        1083     1083           
  Lines      100021   100021           
=======================================
  Hits        72362    72362           
  Misses      22626    22626           
  Partials     5033     5033

Flag	Coverage Δ
aggregator	`75.8% <0.0%> (ø)`
cluster	`84.8% <0.0%> (ø)`
collector	`84.3% <0.0%> (ø)`
dbnode	`78.8% <0.0%> (ø)`
m3em	`74.4% <0.0%> (ø)`
m3ninx	`73.1% <0.0%> (ø)`
metrics	`20.0% <0.0%> (ø)`
msg	`74.2% <0.0%> (ø)`
query	`67.4% <0.0%> (ø)`
x	`80.3% <0.0%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8e1368...f4f96c0. Read the comment docs.

* master: [DOCS] Update to cluster docs (#3084) [dbnode][coordinator] Ensure docs limit is propagated for search and aggregate RPCs (#3108) [query] Take bounds into account for list endpoints (#3110) Add warning to changing blocksize (#3096) Add support for dynamic query limit overriding (#3090) [tests] test setups exported to allow us to use it from other packages (#3042) [query] Implemented Graphite's pow function (#3048) [dbnode] Direct conversion of encoded tags to doc.Metadata (#3087) [tests] Skip flaky TestWatchNoLeader (#3106) [dbnode] Faster search of tag bytes in convert.FromSeriesIDAndTags (#3075) Replace bytes.Compare() == 0 with bytes.Equal() (#3101) Capture seekerMgr instead Rlock (#3104) [m3db] Check bloom filter before stream request allocation (#3103)

…as downsample option is set to "all : false" (#2) * [dbnode] Refactor wide query path (m3db#2826) * [dbnode] Introduce Aggregator type (m3db#2840) * Improve some slow tests (m3db#2881) * Improe some slow tests * lint * lint * goddamn imports * [query] Remove dead code in prom package (m3db#2871) * [dbnode] Refactoring dbShard (m3db#2848) * [ci] Use gofmt to rename generated identifiers (m3db#2883) * Order index segments in multi segments builder in descending order. (m3db#2875) * [all] Return a 504 error if timeout. (m3db#2886) * [cluster] Store shards in sorted form (m3db#2890) * [docs] Update buildkite doc path (m3db#2891) * Add new website relaunch (m3db#2892) * [docs] Fix operator docs no longer being built and published (m3db#2895) * [aggregator] Remove msgpack support (m3db#2894) * [dbnode] Remove dead code in ts package (m3db#2898) * [aggregator] Reduce delays in integration tests (m3db#2896) * [query] Query handlers refactoring (m3db#2872) * Initial commit for proposed query handlers refactoring: * updated CustomHandler interface * unified how query routes are added. no need to pass router around and wrapping is done in one place. * routes are registered with names so they could be easily found later when custom handlers are registered. * trying to make linter happy * linter fixes * revert old behaviour * Make sure route methods are taken into account when adding and searching for named route. * fixed code formatting * [dbnode] Refactor wide query path (m3db#2826) * [dbnode] Introduce Aggregator type (m3db#2840) * [coordinator] Set default namespace tag to avoid colliding with commonly used "namespace" label (m3db#2878) * [coordinator] Set default namespace tag to avoid colliding with common "namespace" default value * Use defined constant * Add downsampler test case to demonstrate override namespace tag Co-authored-by: Wesley Kim <[email protected]> * Improve some slow tests (m3db#2881) * Improe some slow tests * lint * lint * goddamn imports * Changes after code review. * [query] Remove dead code in prom package (m3db#2871) * Register separate route for each method. * linter fixes * removed code duplication in hasndler_test * Fail if route was already registered. * formatted code * Update src/query/api/v1/httpd/handler_test.go Co-authored-by: Vilius Pranckaitis <[email protected]> * Update src/query/api/v1/httpd/handler_test.go Co-authored-by: Vilius Pranckaitis <[email protected]> * More handler tests. Co-authored-by: arnikola <[email protected]> Co-authored-by: Linas Medžiūnas <[email protected]> Co-authored-by: Rob Skillington <[email protected]> Co-authored-by: Wesley Kim <[email protected]> Co-authored-by: Vytenis Darulis <[email protected]> Co-authored-by: Vilius Pranckaitis <[email protected]> * [docs] Add missing single quote to shell command (m3db#2893) * [docs] Update theme rebased (m3db#2899) * Update README to temporarily remove the logo (m3db#2902) * [docs] Small changes to website/docs (m3db#2900) * [changelog] v1.0.0 changelog update (m3db#2904) * [DOCS] Fix broken images and add formatting to links (m3db#2903) * [DOCS] Correct grammar issue on home page (m3db#2905) * Fix seedNodes config in local stack yaml (m3db#2907) * [dtest] Fix API incompatibilities in docker harness (m3db#2849) * Correct old typo (m3db#2906) * [dtest] API tests for /query and /query_range endpoints (m3db#2873) * [dbnode] Query logging (m3db#2888) * [query] Add lookback duration from query config (m3db#2913) * [query] Fix invalid query resulting in 500 (m3db#2910) * [query] Fix /m3query returning 500 on invalid query argument (m3db#2916) * [linter] more optimal linter invocation for "all" case (m3db#2863) * [dbnode] Remove dead code in encoding package (m3db#2920) * [dbnode] Fix client config respecting value for connect consistency (m3db#2914) * [storage] Add Aliyun storage class (m3db#2908) * Update ci submodule to latest (m3db#2921) This was accidentally reverted to an earlier commit in https://github.com/m3db/m3/pull/2907/files. * [m3cluster/etcd] Use zap logger for etcd integration test store (m3db#2915) * [docs] Parameterize menus (m3db#2918) * [coordinator] Validate placement on set placement endpoint unless force set (m3db#2922) * Update development guide (m3db#2832) * [dbnode] Properly rebuild index segments if they fail verification. (m3db#2879) * [dbnode] Add configurability for regexp DFA and FSA limits (m3db#2926) * [DOCS] Separate config to allow for dev and prod setups (m3db#2909) * [aggregator] Use NaN instead of math.MaxFloat64 and add sentinel guards (m3db#2929) * [aggregator] Return NaN instead of 0 for NaN gauge values (m3db#2930) * [query] Allow Graphite variadic functions to omit variadic args (m3db#2882) * [query] Graphite query timeout propagation and add per endpoint status/latency metrics (m3db#2880) * [query] Fix incorrect content type in m3query/ error response (m3db#2917) * [dbnode] Remove dead code in fs package (m3db#2932) * [dbnode] Add DocRef() to NS and multiple series metadata types (m3db#2931) * Revert "[dbnode] Properly rebuild index segments if they fail verification. (m3db#2879)" (m3db#2944) This reverts commit c6fe28d. * [aggregator/client] Include instance ID in write errors (m3db#2945) * [dbnode] Update series metadata type name. (m3db#2946) * [dbnode] Remove some dead code from peersSource (m3db#2779) * Prom read instant handler refactoring (m3db#2928) * [query] Allow customizing request parsing for /query handler * fix lint errors * fix another lint error * PR comments * add an interface for hooks * Removed read instant handler. * Added some comments. * Uses functional options pattern. * Added comments for exported functions. * Simplified options. * Got rid of 2 fields in readHandler. Co-authored-by: Vilius Pranckaitis <[email protected]> Co-authored-by: Rob Skillington <[email protected]> * [coordinator] Extract new namespace validation logics (m3db#2919) * [linter] Fix linter not finding any issues (m3db#2948) * [dbnode] Add shard to ID batch (m3db#2764) * [dbnode] Integrate ns cold flusher into tile aggregation. (m3db#2923) * [tests] Use prometheus v2.22 docker image in docker-integration-tests (m3db#2951) * [dbnode] Widen exposure of namespace method (m3db#2952) * [dtest] Allow skipping docker image build by providing existing image (m3db#2942) * [tests] Fix deprecated Prometheus metric in docker-integration-tests (m3db#2953) * [lint] Update instances of Reuseable -> Reusable (m3db#2950) * [integration] Expose TestIndexWrite methods (m3db#2956) * [docs] Add diagram to comparator readme (m3db#2288) * [coordinator] Consider uninitialized case for UnaggregatedClusterNamespace with dynamic clusters (m3db#2957) * [coordinator] Consider uninitialized case for UnaggregatedClusterNamespace with dynamic clusters * Fixup ready_test * Apply suggestions from code review Co-authored-by: arnikola <[email protected]> * PR feedback * Revert go mod changes Co-authored-by: arnikola <[email protected]> * [dbnode] Simplified interface for MatchesSeriesIters (m3db#2954) * [dbnode] Add wide filter (m3db#2949) * [dbnode] Fix integration code under race with TChannel channel options mutation issue (m3db#2958) * [dbnode] Additional params for Aggregator (m3db#2959) * [cluster] Expose underlying staged placement version in ActiveStagedPlacement (m3db#2963) * [DOCS] Add versions to docs config (m3db#2935) * Add versions to docs config Signed-off-by: ChrisChinchilla <[email protected]> * Update Victor theme Signed-off-by: ChrisChinchilla <[email protected]> * [DOCS] SEO fixes (m3db#2937) * Change site title Signed-off-by: ChrisChinchilla <[email protected]> * Remove duplicated file Signed-off-by: ChrisChinchilla <[email protected]> * Reorder config items Signed-off-by: ChrisChinchilla <[email protected]> * Fix menus, title and relevant config Signed-off-by: ChrisChinchilla <[email protected]> * Update Victor theme Signed-off-by: ChrisChinchilla <[email protected]> * Change path configuration Signed-off-by: ChrisChinchilla <[email protected]> * Change title back to param Signed-off-by: ChrisChinchilla <[email protected]> * [docs] Update docs to clarify Flush and Buffer in Storage Engine (m3db#2511) * * Expand a bit about the type of flushes and how buffer is structured due to accommodate the two types of writes * Emphasis few terms that were hard to read at first time * Fixes several typos and punctuation. * Change links Signed-off-by: ChrisChinchilla <[email protected]> * Fix links Signed-off-by: ChrisChinchilla <[email protected]> * Apply formatting Signed-off-by: ChrisChinchilla <[email protected]> Co-authored-by: Chris Chinchilla <[email protected]> * [aggregator] Split up TCP and M3Msg clients (m3db#2964) * [aggregator] Add ActivePlacement method to TCP client (m3db#2971) * [watchmanager] Do not leak tickers in stopped watches (m3db#2972) * [etcd] Clean up logging in WatchManager (m3db#2973) * [etcd] Log watch error source details (m3db#2974) * [etcd] Expose cached etcd clients in cluster etcd config service (m3db#2975) * [dbnode] Prevent potential division by 0 in StreamingWriter (m3db#2965) * [dbnode] Fix shard time ranges superset logic. Cleanup and add tests. (m3db#2968) * [docs] Fix link for July 2020 community meeting recording (m3db#2978) * [query] Fix query metrics assigning wrong status codes for errors (m3db#2960) * [cluster] Fix deadlock in new ActiveStagedPlacement method (m3db#2980) * [coordinator] add graphite names to aggregation types (m3db#2970) * Fix broken og meta tags on static pages (m3db#2982) Signed-off-by: ChrisChinchilla <[email protected]> * [docs] Remove reference to adding namespace to coordinator config (m3db#2977) Co-authored-by: Chris Chinchilla <[email protected]> * [coordinator] [query] Add readiness probe for probing current consistency level achievability (m3db#2976) * [cluster] Fix watch resource leak/hang (m3db#2984) * Remove odd uses of satori/go.uuid package (m3db#2985) * [tools] Add benchmarking support to read_data_files (m3db#2986) * [dbnode] Implement shard.ScanData (m3db#2981) * [dbnode] Add TChannel channel configuration (m3db#2989) * [aggregator] Fix panic on shutdown in ForwardedWriter (m3db#2987) * [DOCS] Fix grammar and create code snippet for step in quickstart (m3db#2961) * Fix grammar and create code snippet for step in quickstart Signed-off-by: ChrisChinchilla <[email protected]> * Update other files with readiness Signed-off-by: ChrisChinchilla <[email protected]> * [aggregator] Use a single goroutine to emit metrics for all TCP client queues (m3db#2991) * [aggregator] Lockless ratelimiter (m3db#2988) * [x/net/http] Add Proxy field to HTTPClientOptions (m3db#2994) * [dbnode] Skip out of retention index segments during bootstrap. (m3db#2992) * [aggregator] keep metric type during the aggregation (m3db#2941) * Update k8s doc references (m3db#2998) * [m3coordinator] Allow downsampler to configure custom rule store (m3db#2996) * [rules] Set tags field of mapping rules history conversion (m3db#2999) * [query] Fix for panic when deleting a namespace (m3db#2990) * [coordinator] Update ApplyCustomRuleStore interface (m3db#3000) * [cluster] Close etcd watches completely before recreating (m3db#2997) * [coordinator] Add WriteNamespaces API to rules store (m3db#3001) * [dbnode] Add ability to force enable cold writes from config (m3db#3002) * Adding Hostinger Logo (m3db#3006) * [DOCS] Add clustering getting started guides for binaries and kubernetes (m3db#2795) * Move other quickstarts Signed-off-by: ChrisChinchilla <[email protected]> Move configuration from common steps as it no longer matches k8s steps Signed-off-by: ChrisChinchilla <[email protected]> Squashed commit of the following: commit cfffc26414de48e6e5d4784f0908aaebb19743fe Author: ChrisChinchilla <[email protected]> Date: Mon Oct 26 15:57:52 2020 +0100 wip Signed-off-by: ChrisChinchilla <[email protected]> commit e8b9766 Author: ChrisChinchilla <[email protected]> Date: Mon Oct 26 15:53:36 2020 +0100 Move previous guides to clustering quickstart section Signed-off-by: ChrisChinchilla <[email protected]> Signed-off-by: ChrisChinchilla <[email protected]> Update section Signed-off-by: ChrisChinchilla <[email protected]> Fix links Signed-off-by: ChrisChinchilla <[email protected]> Fix links add redirects Signed-off-by: ChrisChinchilla <[email protected]> Remove aggregator mention Signed-off-by: ChrisChinchilla <[email protected]> Remove aggregator mention Signed-off-by: ChrisChinchilla <[email protected]> Refactor Signed-off-by: ChrisChinchilla <[email protected]> Remove Docker fo rnow Signed-off-by: ChrisChinchilla <[email protected]> Remove and correct broken links Signed-off-by: ChrisChinchilla <[email protected]> Revert "Remove Docker fo rnow" This reverts commit 7b56da1. Fix more broken links Signed-off-by: ChrisChinchilla <[email protected]> Draft docker guide Signed-off-by: ChrisChinchilla <[email protected]> Remove duplicate content Signed-off-by: ChrisChinchilla <[email protected]> Remove embedded cluster note Signed-off-by: ChrisChinchilla <[email protected]> Move other quickstarts Signed-off-by: ChrisChinchilla <[email protected]> Squashed commit of the following: commit cfffc26414de48e6e5d4784f0908aaebb19743fe Author: ChrisChinchilla <[email protected]> Date: Mon Oct 26 15:57:52 2020 +0100 wip Signed-off-by: ChrisChinchilla <[email protected]> commit e8b9766 Author: ChrisChinchilla <[email protected]> Date: Mon Oct 26 15:53:36 2020 +0100 Move previous guides to clustering quickstart section Signed-off-by: ChrisChinchilla <[email protected]> Signed-off-by: ChrisChinchilla <[email protected]> Update section Signed-off-by: ChrisChinchilla <[email protected]> Fix links Signed-off-by: ChrisChinchilla <[email protected]> Fix links add redirects Signed-off-by: ChrisChinchilla <[email protected]> Remove aggregator mention Signed-off-by: ChrisChinchilla <[email protected]> Remove aggregator mention Signed-off-by: ChrisChinchilla <[email protected]> Refactor Signed-off-by: ChrisChinchilla <[email protected]> Remove and correct broken links Signed-off-by: ChrisChinchilla <[email protected]> Respond to feedback Signed-off-by: ChrisChinchilla <[email protected]> * Fix rendering issues Signed-off-by: ChrisChinchilla <[email protected]> * Update theme Signed-off-by: ChrisChinchilla <[email protected]> * Remove failing git info flag Signed-off-by: ChrisChinchilla <[email protected]> * Fix links Signed-off-by: ChrisChinchilla <[email protected]> * Respond to feedback Signed-off-by: ChrisChinchilla <[email protected]> * Respond to feedback and fix rebased files Signed-off-by: ChrisChinchilla <[email protected]> * Remove helm mentions Signed-off-by: ChrisChinchilla <[email protected]> * Respond to feedback Signed-off-by: ChrisChinchilla <[email protected]> * Fix link Signed-off-by: ChrisChinchilla <[email protected]> * Update theme to version 0.2.2 Signed-off-by: ChrisChinchilla <[email protected]> * Tidy Go mods Signed-off-by: ChrisChinchilla <[email protected]> * Quick fix to broken code tabs on quickstart (m3db#3010) Signed-off-by: ChrisChinchilla <[email protected]> * [cluster] Watch follow-ups (m3db#3007) * [coordinator] Allow clients to pass in augmentM3Tags as a config (m3db#3014) AugmentM3Tags will augment the metric type to aggregated metrics to be used within the filter for rules. If enabled, for example, your filter can specify '__m3_type__:gauge' to filter by gauges. This is particularly useful for Graphite metrics today. Furthermore, the option is automatically enabled if static rules are used and any filter contain an __m3_type__ tag. * [docs] Fix conflicting OpenAPI operation IDs (m3db#3009) * [coordinator] Include tags in view.MappingRule Equals (m3db#3015) * [dbnode] Emit consistencyResultError from fetchTaggedResultAccumulator (m3db#3016) * [dbnode] Introduce Resource Exhausted RPC Error flag for query limits (m3db#3017) * [DOCS] Fix new API paths and add Docker version (m3db#3018) * Fix new API paths and add Docker version Signed-off-by: ChrisChinchilla <[email protected]> * Force add missing shortcode Signed-off-by: ChrisChinchilla <[email protected]> * [dbnode] Add server.StorageOptions to TestSetup (m3db#3023) * [dbnode] Fix "unable to satisfy consistency requirements" error message (m3db#3025) * [coordinator] Pass along instrument options into custom rule store fn (m3db#3027) * [DOCS] Fix services ready path (m3db#3026) * Fix services ready path Signed-off-by: ChrisChinchilla <[email protected]> * Update theme Signed-off-by: ChrisChinchilla <[email protected]> * [query] Add missing "status" field to PromQL error responses (m3db#2933) * [dbnode] Add a few tests for query limit exceeded error (m3db#3030) * [cluster] Add config wiring for etcd timeouts, retries, serialized gets (m3db#3035) * [aggregator] Pack some tcp client structs for lower memory utilization (m3db#3037) * [coordinator] Remove unused DisableAutoMappingRules config (m3db#3036) * [query] Allow injecting function for rewriting error response (m3db#3008) * [DOCS] SEO and config changes (m3db#3024) * Add GitHub edit URL Signed-off-by: ChrisChinchilla <[email protected]> * Add robots generation Signed-off-by: ChrisChinchilla <[email protected]> * Update redirects Signed-off-by: ChrisChinchilla <[email protected]> * Add 404 page Signed-off-by: ChrisChinchilla <[email protected]> * Fix paths Signed-off-by: ChrisChinchilla <[email protected]> * Fi title and description Signed-off-by: ChrisChinchilla <[email protected]> * Update theme Signed-off-by: ChrisChinchilla <[email protected]> * Fix random titles Signed-off-by: ChrisChinchilla <[email protected]> * [DOCS] Add back FAQ and troubleshooting sections (m3db#3029) * FAQ and troubleshooting sections Signed-off-by: ChrisChinchilla <[email protected]> * Fix links Signed-off-by: ChrisChinchilla <[email protected]> * [dbnode] Client borrow connection API (m3db#3019) * [dbnode] Support checkpointing in cold flusher and preparing index writer per block (m3db#3040) * [coordinator] Ignore rollup rule storage policies when deciding to utilize auto-mapping rules (m3db#3044) * [coordinator] Ignore rollup rule storage policies when deciding to utiilize auto-mapping rules * IsRollupRule -> IsMappingRule * [aggregator] Fix panic on invalid unaggregated payloads (m3db#3041) * [coordinator] Add Carbon ingest latency metrics (m3db#3045) * [query] Graphite match regexp all values for __g0__ tag matcher instead of match field (m3db#3021) * [query] Update Graphite quoted strings to just escape quotes and a flag for previous escape behavior (m3db#3022) * [coordinator] Add Graphite Carbon ingest contains rule matcher for fast match (m3db#3046) * [coordinator] Add Graphite rewrite cleanup directive for cleansing incoming metrics (m3db#3047) * [coordinator] Add Graphite Carbon rewrite cleanup integration test (m3db#3049) * [query] implemented Graphite's `powSeries` function (m3db#3038) * [tools] Fix failing linter (m3db#3052) * [query] Add Graphite support for ** with metric path selectors (m3db#3020) * [aggregator/client] Metric for dropped metrics (m3db#3054) Problem: When collector fails to write a metric to aggregator, it logs an error but in practice it is almost impossible to tell whether it failed to write to both peers owning a shard or only one of them, i.e. whether data is lost or not. Solultion: Emit a metric indicating data loss. * [r2ctl] Bump urijs from 1.19.1 to 1.19.5 in /src/ctl/ui (m3db#3055) Bumps [urijs](https://github.com/medialize/URI.js) from 1.19.1 to 1.19.5. - [Release notes](https://github.com/medialize/URI.js/releases) - [Changelog](https://github.com/medialize/URI.js/blob/gh-pages/CHANGELOG.md) - [Commits](medialize/URI.js@v1.19.1...v1.19.5) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [coordinator] Add ability to turn off rule matching cache (m3db#3059) * Don't attempt to lint genny gen'd code (m3db#3063) * [dbnode][m3ninx] Rename doc.Document -> doc.Metadata (m3db#3062) * [dtest] Upgrade dockertest to v3.6.3 (m3db#3053) * [dbnode] Add ExcludeOrigin to skip localhost on BorrowConnections (m3db#3066) * [dtest] Add a method for Prom remote writes (m3db#3065) * [dbnode][m3ninx] Add support for reading document containers from an index (m3db#3050) Reading raw document metadata from an index can end up being rather expensive for metadata with a lot of tags. This commit introduces the concept of encoded metadata, which wrap the relevant section of bytes and provides an efficient reader to retrieve the non-encoded metadata without ballooning memory usage. Additionally, introduce a concept of a document that wraps either raw metadata or encoded metadata, which can be used regardless of whether an index segment is backed by metadata (from memory) or encoded metadata (read from disk) * [query] Fix aggregated namespace header to be bad request instead of internal server error (m3db#3070) * [coordinator] Add TagFilter Matches comments (m3db#3072) * [coordinator] Remove includeRollupsOnDefaultRuleFiltering flag (m3db#3073) Confirmed this is working as expected, and is not useful to configure otherwise. Co-authored-by: Rob Skillington <[email protected]> * [dbnode] Remove per-series traces on read queries (m3db#3074) * [x] Replace InnerError() method usages with xerrors.InnerError() (m3db#3068) * [aggregator] Add ActivePlacementVersion to tcp client (m3db#3071) * [coordinator] Rollout augmentM3Tags flag to true by default (m3db#3082) * [coordinator] Rollout augmentM3Tags flag to true by default * Fixup operator docs link * Fixup doc site links * [matcher/coordinator] Add latency metrics to rule matching (m3db#3083) * [dbnode][m3ninx] Use new doc.Document in query results to reduce slice allocations (m3db#3057) * [coordinator] Disable downsampler matcher cache by default (m3db#3080) * [dtest] Run tests on an existing cluster launched with docker compose (m3db#3067) * Additional metrics for peers bootstrapper (m3db#3060) * refactored instrumentation code for peers bootstrapper. added additional timers for peers bootstrapper to track individual steps. * fixed linter warnings * removed dead code * removed dead code * changes after review * added new gauge - bootstrap-retries * moved some instrumentation methods to instrumentationContexts for better encapsulation and easier usage. * forgot to set nowFn * added methods for creating instrumentation contexts. * changes after code review * [dbnode] Log namespace and shard upon invalid series read/write count (m3db#3092) * [dbnode] Fix node serving reads/marked bootstrapped after bootstrap failure (m3db#3088) * set bootstrap state to Bootstrapped when it returns no errors * added bootstrap regression test. * removed new test and updated old test to check for bootstrap states. * [cluster] Fix flaky watchmanager tests (m3db#3091) * [dbnode] Remove allocation per series ID when streaming block from disk (m3db#3093) * [dbnode] Remove reverse index sharing leftovers (m3db#3095) * Limit for time series read from disk (m3db#3094) * Limit for time series read from disk There is additional memory costs when time series need to be loaded from disk (in addition to the actual bytes being read). This new limit allows docLimits to be higher so series already in memory can easily be served without hitting limits. * [x] Make serialize.TagValueFromEncodedTagsFast() faster (m3db#3097) * Revert "[aggregator] keep metric type during the aggregation" (m3db#3099) * [m3db] Check bloom filter before stream request allocation (m3db#3103) * [m3db] Check bloom filter before stream request allocation * Add test assertions for bloom filer misses metric * Remove redundant series-read metric * Capture seekerMgr instead Rlock (m3db#3104) seekerMgr could change outside the lock, which might result in a nil pointer * Replace bytes.Compare() == 0 with bytes.Equal() (m3db#3101) * [dbnode] Faster search of tag bytes in convert.FromSeriesIDAndTags (m3db#3075) * [tests] Skip flaky TestWatchNoLeader (m3db#3106) * [dbnode] Direct conversion of encoded tags to doc.Metadata (m3db#3087) * [query] Implemented Graphite's pow function (m3db#3048) * [tests] test setups exported to allow us to use it from other packages (m3db#3042) * Disable downsample in case of agg namespace Check if there's at least one aggregated namespace with downsample disabled. If so, disabled the downsample object entirely * minor Co-authored-by: arnikola <[email protected]> Co-authored-by: Linas Medžiūnas <[email protected]> Co-authored-by: Vytenis Darulis <[email protected]> Co-authored-by: Vilius Pranckaitis <[email protected]> Co-authored-by: Bo Du <[email protected]> Co-authored-by: Chris Chinchilla <[email protected]> Co-authored-by: Rob Skillington <[email protected]> Co-authored-by: λinas <[email protected]> Co-authored-by: Wesley Kim <[email protected]> Co-authored-by: nate <[email protected]> Co-authored-by: Benjamin Raskin <[email protected]> Co-authored-by: Ryan Allen <[email protected]> Co-authored-by: Ryan Hall <[email protected]> Co-authored-by: Antanas Bastys <[email protected]> Co-authored-by: 李国忠 <[email protected]> Co-authored-by: Matt Way <[email protected]> Co-authored-by: Alex Bublichenko <[email protected]> Co-authored-by: Asaf Mesika <[email protected]> Co-authored-by: shreyassrivatsan <[email protected]> Co-authored-by: Jerome Froelich <[email protected]> Co-authored-by: Gediminas Guoba <[email protected]> Co-authored-by: Gibbs Cullen <[email protected]> Co-authored-by: Dean Wahle <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: omarkhalid79 <[email protected]>

vpranckaitis and others added 10 commits January 11, 2021 14:00

add benchmark

ac80c69

speed up the search of tag bytes in series ID

f0370d8

update tests

5e77e4a

PR comments

fe8ee23

add tests that check if bytes tag slices reuse data from series ID

4221b84

update benchmark

4927f6f

PR comments

6bb6d9b

un-base64 series ID samples

6d007e3

Merge branch 'master' into vilius/tags_to_doc_performace_improvement

9114f90

[dbnode] Direct conversion of encoded tags to doc.Metadata

a879a2a

vpranckaitis commented Jan 14, 2021

View reviewed changes

src/dbnode/storage/index/convert/convert_benchmark_test.go Outdated Show resolved Hide resolved

linasm reviewed Jan 14, 2021

View reviewed changes

vpranckaitis added 2 commits January 14, 2021 15:44

extract decoding steps into smaller functions

db2023e

thinner encoded tag iterator and benchmarks

f7c6bcd

vpranckaitis commented Jan 15, 2021

View reviewed changes

merge DecodeTagName() and DecodeTagValue() methods

f769ce0

linasm self-assigned this Jan 15, 2021

linasm reviewed Jan 18, 2021

View reviewed changes

vpranckaitis added 2 commits January 18, 2021 14:36

function that uses indexing instead of sliding slice

d3a8c38

benchmark TagValueFromEncodedTagsFast

c4296b6

vpranckaitis commented Jan 18, 2021

View reviewed changes

vpranckaitis mentioned this pull request Jan 18, 2021

[x] Make serialize.TagValueFromEncodedTagsFast() faster #3097

Merged

vpranckaitis added 5 commits January 19, 2021 10:12

revert decoder_fast.go changes

9b1ad47

revert decoder_fast_iter.go changes

ba77217

remove convert.FromSeriesIDAndEncodedTagsIndex()

3e949af

export and use variables from serialize package

e0cad90

add tests

f77ccef

vpranckaitis force-pushed the vilius/convert_encoded_tags_to_doc branch 2 times, most recently from 86ca35d to f77ccef Compare January 19, 2021 10:01

vpranckaitis marked this pull request as ready for review January 19, 2021 10:02

linasm approved these changes Jan 19, 2021

View reviewed changes

Apply suggestions from code review

45378ef

Co-authored-by: Linas Medžiūnas <[email protected]>

linasm removed their assignment Jan 19, 2021

PR comments

a5436e8

linasm reviewed Jan 19, 2021

View reviewed changes

use ident.BytesID instead of ident.ID

8b725f0

Base automatically changed from vilius/tags_to_doc_performace_improvement to master January 20, 2021 08:12

Merge branch 'master' into vilius/convert_encoded_tags_to_doc

f4f96c0

Conflicts: src/dbnode/storage/index/convert/convert.go src/dbnode/storage/index/convert/convert_benchmark_test.go src/dbnode/storage/index/convert/convert_test.go src/x/serialize/decoder_fast.go

Merge branch 'master' into vilius/convert_encoded_tags_to_doc

d782a61

vpranckaitis merged commit 611166b into master Jan 20, 2021

vpranckaitis deleted the vilius/convert_encoded_tags_to_doc branch January 20, 2021 10:11

vpranckaitis added the T: Optimization label Feb 1, 2021

linasm mentioned this pull request May 10, 2021

[dbnode] Remove tag decoders from write tagged requests. #3472

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dbnode] Direct conversion of encoded tags to doc.Metadata #3087

[dbnode] Direct conversion of encoded tags to doc.Metadata #3087

vpranckaitis commented Jan 14, 2021 •

edited

Loading

linasm left a comment

linasm Jan 14, 2021

linasm Jan 14, 2021

linasm Jan 14, 2021

linasm Jan 14, 2021

vpranckaitis Jan 15, 2021 •

edited

Loading

linasm commented Jan 15, 2021 •

edited

Loading

linasm Jan 18, 2021

linasm Jan 18, 2021

linasm Jan 18, 2021

vpranckaitis Jan 18, 2021

linasm Jan 18, 2021

vpranckaitis Jan 18, 2021

linasm Jan 18, 2021

vpranckaitis Jan 18, 2021

vpranckaitis Jan 18, 2021

vpranckaitis Jan 18, 2021

vpranckaitis Jan 18, 2021

vpranckaitis Jan 18, 2021

linasm left a comment

linasm Jan 19, 2021

vpranckaitis Jan 19, 2021

linasm Jan 19, 2021

vpranckaitis Jan 19, 2021

linasm Jan 19, 2021

vpranckaitis Jan 19, 2021

codecov bot commented Jan 20, 2021

		@@ -27,40 +27,76 @@ import (

		// TagValueFromEncodedTagsFast returns a tag from a set of encoded tags without
		// any pooling required.

[dbnode] Direct conversion of encoded tags to doc.Metadata #3087

[dbnode] Direct conversion of encoded tags to doc.Metadata #3087

Conversation

vpranckaitis commented Jan 14, 2021 • edited Loading

linasm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vpranckaitis Jan 15, 2021 • edited Loading

Choose a reason for hiding this comment

linasm commented Jan 15, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linasm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 20, 2021

Codecov Report

vpranckaitis commented Jan 14, 2021 •

edited

Loading

vpranckaitis Jan 15, 2021 •

edited

Loading

linasm commented Jan 15, 2021 •

edited

Loading