Cache config value to optimize Get config API #151

x-xy-y · 2021-12-23T11:28:55Z

读配置的过程，前后要经过好多次锁，而且会受 source 实现影响。改为直接缓存当前的配置值，读取时从当前缓存读。

修改前：

go test -bench=^Benchmark -benchtime=5s
goos: darwin
goarch: amd64
pkg: github.com/go-chassis/go-archaius/benchmark
BenchmarkFileSource-8 5213296 1142 ns/op
BenchmarkFileSourceParallelism-8 18202258 347 ns/op

修改后：

go test -bench=^Benchmark -benchtime=5s
goos: darwin
goarch: amd64
pkg: github.com/go-chassis/go-archaius/benchmark
BenchmarkFileSource-8 68292421 87.3 ns/op
BenchmarkFileSourceParallelism-8 100000000 54.8 ns/op

tianxiaoliang · 2021-12-25T03:27:36Z

先把圈复杂度的检查去掉吧，这个工具肯定又出问题了

go-archaius/.github/workflows/static_check.yml

Line 34 in 2df34b1

go get github.com/fzipp/gocyclo

tianxiaoliang · 2021-12-25T03:32:22Z

benchmark/benchmark_test.go

+
+func createFile(content string, name string, dir string) string {
+	filename := filepath.Join(dir, name)
+	f1, _ := os.Create(filename)


Go的UT已经支持临时目录特性了，不需要再这么创建，可以参考下，少维护一些代码

https://pkg.go.dev/testing#T.TempDir

go 1.13 不支持 t.TempDir ，我们需要支持 go 1.13, 这个不改吧

出自 https://go.dev/doc/devel/release
go的版本策略，为了你们的安全考虑，我不建议使用低于1.5及以下的版本，没有安全保障

实际的业务不可能是跟着 go 官方的节奏走，总会有一些版本比较落后。就算业务有意愿，各种三方库也有大量跟不上的。而且这里的并不需要做多大的修改，就能够都支持到的，何苦一定要改呢

benchmark/benchmark_test.go

source/manager.go

tianxiaoliang · 2021-12-29T08:26:07Z

先把圈复杂度的检查去掉吧，这个工具肯定又出问题了

go-archaius/.github/workflows/static_check.yml

Line 34 in 2df34b1

go get github.com/fzipp/gocyclo

得删掉这部分，门禁才能过

tianxiaoliang · 2021-12-29T08:27:50Z

source/manager.go

 			} else {
-				m.ConfigurationMap.Store(e.Key, source.GetSourceName())
+				srcName := source.GetSourceName()
+				// fixme:


这个看上去改变了程序过去的行为。可否用过archaius框架级的bool作为功能开关，控制get时是否从缓存取值

考虑用特性分支管理

x-xy-y · 2022-02-11T11:53:24Z

上回提到事件更新行为一直都有不能保证最终一致的问题
我最近加了一个单元测试 consistencytest , 并修复了相关的问题
不过呢，有一个关联的 archaius.Delete 的行为有点奇怪，它目前的实现是 source 的配置不删除(有一些 source 甚至只给了个空实现)，在 mem source 会触发配置删除事件，然后原先的单元测试和我新加的 consistencytest 都会依赖它。我理解 archaius.Delete 存在的意义就是用于单元测试，所以我修改了 mem source 的 Delete 行为，让它删除自己的 map 里的配置，再触发通知，以保证 consistencytest 的行为正确（以及我的其他修改后的单元测试可以通过）。

在目前的 master 上，(加上 mem source 的 Delete 修改) 运行 consistencytest，可以比较容易复现配置无法保证最终一致的问题.

在修改了 updateEvent 机制的 feature/cache-config-value 分支上, consistencytest 可以 pass

master 上的失败信息

--- FAIL: TestFinalConsistencyAfterConcurrentUpdateDelete (0.33s)
    consistency_test.go:73: test run 00000
    consistency_test.go:73: test run 00001
    consistency_test.go:73: test run 00002
    consistency_test.go:73: test run 00003
    consistency_test.go:73: test run 00004
    consistency_test.go:73: test run 00005
    consistency_test.go:73: test run 00006
    consistency_test.go:73: test run 00007
    consistency_test.go:73: test run 00008
    consistency_test.go:73: test run 00009
    consistency_test.go:73: test run 00010
    consistency_test.go:73: test run 00011
    consistency_test.go:73: test run 00012
    consistency_test.go:73: test run 00013
    consistency_test.go:73: test run 00014
    consistency_test.go:73: test run 00015
    consistency_test.go:73: test run 00016
    consistency_test.go:73: test run 00017
    consistency_test.go:73: test run 00018
    consistency_test.go:73: test run 00019
    consistency_test.go:73: test run 00020
    consistency_test.go:73: test run 00021
    consistency_test.go:73: test run 00022
    consistency_test.go:73: test run 00023
    consistency_test.go:73: test run 00024
    consistency_test.go:73: test run 00025
    consistency_test.go:73: test run 00026
    consistency_test.go:73: test run 00027
    consistency_test.go:73: test run 00028
    consistency_test.go:73: test run 00029
    consistency_test.go:73: test run 00030
    consistency_test.go:73: test run 00031
    consistency_test.go:73: test run 00032
    consistency_test.go:73: test run 00033
    consistency_test.go:73: test run 00034
    consistency_test.go:73: test run 00035
    consistency_test.go:73: test run 00036
    consistency_test.go:73: test run 00037
    consistency_test.go:73: test run 00038
    consistency_test.go:73: test run 00039
    consistency_test.go:73: test run 00040
    consistency_test.go:73: test run 00041
    consistency_test.go:73: test run 00042
    consistency_test.go:73: test run 00043
    consistency_test.go:73: test run 00044
    consistency_test.go:73: test run 00045
    consistency_test.go:73: test run 00046
    consistency_test.go:73: test run 00047
    consistency_test.go:73: test run 00048
    consistency_test.go:73: test run 00049
    consistency_test.go:73: test run 00050
    consistency_test.go:73: test run 00051
    consistency_test.go:73: test run 00052
    consistency_test.go:73: test run 00053
    consistency_test.go:73: test run 00054
    consistency_test.go:73: test run 00055
    consistency_test.go:73: test run 00056
    consistency_test.go:73: test run 00057
    consistency_test.go:73: test run 00058
    consistency_test.go:73: test run 00059
    consistency_test.go:73: test run 00060
    consistency_test.go:73: test run 00061
    consistency_test.go:73: test run 00062
    consistency_test.go:73: test run 00063
    consistency_test.go:73: test run 00064
    consistency_test.go:73: test run 00065
    consistency_test.go:73: test run 00066
    consistency_test.go:73: test run 00067
    consistency_test.go:73: test run 00068
    consistency_test.go:73: test run 00069
    consistency_test.go:73: test run 00070
    consistency_test.go:73: test run 00071
    consistency_test.go:73: test run 00072
    consistency_test.go:73: test run 00073
    consistency_test.go:73: test run 00074
    consistency_test.go:73: test run 00075
    consistency_test.go:73: test run 00076
    consistency_test.go:73: test run 00077
    consistency_test.go:73: test run 00078
    consistency_test.go:73: test run 00079
    consistency_test.go:73: test run 00080
    consistency_test.go:73: test run 00081
    consistency_test.go:73: test run 00082
    consistency_test.go:73: test run 00083
    consistency_test.go:73: test run 00084
    consistency_test.go:73: test run 00085
    consistency_test.go:73: test run 00086
    consistency_test.go:73: test run 00087
    consistency_test.go:73: test run 00088
    consistency_test.go:73: test run 00089
    consistency_test.go:73: test run 00090
    consistency_test.go:73: test run 00091
    consistency_test.go:73: test run 00092
    consistency_test.go:73: test run 00093
    consistency_test.go:73: test run 00094
    consistency_test.go:73: test run 00095
    consistency_test.go:73: test run 00096
    consistency_test.go:73: test run 00097
    consistency_test.go:73: test run 00098
    consistency_test.go:73: test run 00099
    consistency_test.go:73: test run 00100
    consistency_test.go:73: test run 00101
    consistency_test.go:73: test run 00102
    consistency_test.go:73: test run 00103
    consistency_test.go:73: test run 00104
    consistency_test.go:73: test run 00105
    consistency_test.go:73: test run 00106
    consistency_test.go:73: test run 00107
    consistency_test.go:73: test run 00108
    consistency_test.go:73: test run 00109
    consistency_test.go:73: test run 00110
    consistency_test.go:73: test run 00111
    consistency_test.go:73: test run 00112
    consistency_test.go:73: test run 00113
    consistency_test.go:73: test run 00114
    consistency_test.go:73: test run 00115
    consistency_test.go:73: test run 00116
    consistency_test.go:73: test run 00117
    consistency_test.go:73: test run 00118
    consistency_test.go:73: test run 00119
    consistency_test.go:73: test run 00120
    consistency_test.go:73: test run 00121
    consistency_test.go:73: test run 00122
    consistency_test.go:73: test run 00123
    consistency_test.go:73: test run 00124
    consistency_test.go:73: test run 00125
    consistency_test.go:73: test run 00126
    consistency_test.go:73: test run 00127
    consistency_test.go:73: test run 00128
    consistency_test.go:73: test run 00129
    consistency_test.go:73: test run 00130
    consistency_test.go:73: test run 00131
    consistency_test.go:73: test run 00132
    consistency_test.go:73: test run 00133
    consistency_test.go:73: test run 00134
    consistency_test.go:73: test run 00135
    consistency_test.go:73: test run 00136
    consistency_test.go:73: test run 00137
    consistency_test.go:73: test run 00138
    consistency_test.go:73: test run 00139
    consistency_test.go:73: test run 00140
    consistency_test.go:73: test run 00141
    consistency_test.go:73: test run 00142
    consistency_test.go:73: test run 00143
    consistency_test.go:73: test run 00144
    consistency_test.go:73: test run 00145
    consistency_test.go:73: test run 00146
    consistency_test.go:73: test run 00147
    consistency_test.go:73: test run 00148
    consistency_test.go:73: test run 00149
    consistency_test.go:73: test run 00150
    consistency_test.go:73: test run 00151
    consistency_test.go:73: test run 00152
    consistency_test.go:73: test run 00153
    consistency_test.go:73: test run 00154
    consistency_test.go:73: test run 00155
    consistency_test.go:73: test run 00156
    consistency_test.go:70:
        	Error Trace:	consistency_test.go:70
        	            				consistency_test.go:74
        	Error:      	Not equal:
        	            	expected: 3
        	            	actual  : 9
        	Test:       	TestFinalConsistencyAfterConcurrentUpdateDelete
FAIL
exit status 1
FAIL	github.com/go-chassis/go-archaius/consistencytest	1.878s

tianxiaoliang · 2022-02-13T12:09:16Z

cyclo检查失败了，把这个检查步骤干掉吧

x-xy-y · 2022-02-21T07:38:24Z

cyclo检查失败了，把这个检查步骤干掉吧

我删掉了 cyclo , 再看看？

tianxiaoliang · 2022-02-22T12:10:04Z

可以推送到新的banch vx.x.x.beta

x-xy-y · 2022-02-23T12:02:53Z

可以推送到新的banch vx.x.x.beta

我应该是没有权限直接推送到这个库的。能做的是创建 Pull Request . 但也不存在一个 vx.x.x.beta 的分支给我选择。是不是你要先建一下这个分支

tianxiaoliang · 2022-02-24T01:50:46Z

https://github.com/go-chassis/go-archaius/tree/v1.6.0.beta done

Cache config value to optimize Get config API

ad88741

x-xy-y force-pushed the feature/cache-config-value branch from 43bf549 to ad88741 Compare December 23, 2021 11:29

tianxiaoliang reviewed Dec 25, 2021

View reviewed changes

benchmark/benchmark_test.go Outdated Show resolved Hide resolved

tianxiaoliang reviewed Dec 25, 2021

View reviewed changes

source/manager.go Show resolved Hide resolved

tianxiaoliang requested changes Dec 25, 2021

View reviewed changes

source/manager.go Outdated Show resolved Hide resolved

source/manager.go Outdated Show resolved Hide resolved

zhuohuang yu added 2 commits December 28, 2021 17:33

wrap up cache operations

6e01a0b

refactory test

6d6b664

tianxiaoliang reviewed Dec 29, 2021

View reviewed changes

zhuohuang yu added 2 commits February 11, 2022 19:07

re-factory event notification for final consistency

8f4d909

rm unused code

dd2a27b

rm cyclo

6a47e42

x-xy-y mentioned this pull request Feb 25, 2022

Feature/cache config value #153

Merged

x-xy-y mentioned this pull request Mar 7, 2022

fix panic when update with slice / map val #156

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cache config value to optimize Get config API #151

Cache config value to optimize Get config API #151

x-xy-y commented Dec 23, 2021 •

edited

Loading

tianxiaoliang commented Dec 25, 2021 •

edited

Loading

tianxiaoliang Dec 25, 2021 •

edited

Loading

x-xy-y Dec 27, 2021

tianxiaoliang Dec 27, 2021 •

edited

Loading

x-xy-y Dec 28, 2021

tianxiaoliang commented Dec 29, 2021

tianxiaoliang Dec 29, 2021

tianxiaoliang Jan 4, 2022

x-xy-y commented Feb 11, 2022

tianxiaoliang commented Feb 13, 2022

x-xy-y commented Feb 21, 2022

tianxiaoliang commented Feb 22, 2022

x-xy-y commented Feb 23, 2022

tianxiaoliang commented Feb 24, 2022

Cache config value to optimize Get config API #151

Are you sure you want to change the base?

Cache config value to optimize Get config API #151

Conversation

x-xy-y commented Dec 23, 2021 • edited Loading

tianxiaoliang commented Dec 25, 2021 • edited Loading

tianxiaoliang Dec 25, 2021 • edited Loading

Choose a reason for hiding this comment

x-xy-y Dec 27, 2021

Choose a reason for hiding this comment

tianxiaoliang Dec 27, 2021 • edited Loading

Choose a reason for hiding this comment

x-xy-y Dec 28, 2021

Choose a reason for hiding this comment

tianxiaoliang commented Dec 29, 2021

tianxiaoliang Dec 29, 2021

Choose a reason for hiding this comment

tianxiaoliang Jan 4, 2022

Choose a reason for hiding this comment

x-xy-y commented Feb 11, 2022

tianxiaoliang commented Feb 13, 2022

x-xy-y commented Feb 21, 2022

tianxiaoliang commented Feb 22, 2022

x-xy-y commented Feb 23, 2022

tianxiaoliang commented Feb 24, 2022

x-xy-y commented Dec 23, 2021 •

edited

Loading

tianxiaoliang commented Dec 25, 2021 •

edited

Loading

tianxiaoliang Dec 25, 2021 •

edited

Loading

tianxiaoliang Dec 27, 2021 •

edited

Loading