Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

One question about maxOffset setting #235

Closed
joezxy opened this issue Dec 17, 2014 · 3 comments
Closed

One question about maxOffset setting #235

joezxy opened this issue Dec 17, 2014 · 3 comments

Comments

@joezxy
Copy link
Contributor

joezxy commented Dec 17, 2014

Hi, in RemoteClockMonitor.MonitorRemoteOffsets() implementation, if node's clock offset goes beyond the maxOffset parameter, this node will Fatalf and suicide.

Parameter maxOffset is set to 250ms by default, any reason for choosing this number?
and as the design goal, Cockroachdb should be geo-distributed, then how to set this number for cross-datacenter deployment if without some infrastructure like Atom/GPS synchronized clock using in Spanner. Does normal NTP work?

Waiting for your clarification..:-)

@tbg
Copy link
Member

tbg commented Dec 17, 2014

Hi Joezxy,

the 250ms are a first attempt at hard-coding a number that does well enough
for now. In practice, usually with NTP you should stay well below that -
but there are other events which might push a node over that boundary
(think leap seconds) and even later the offset should be measurable (or
controllable) by linking it up with an API a la TrueTime. I would suggest
that for now you take this part of the code with a grain of salt - it's the
first stab at something that'll see more iterations. I do not expect that
we will ship this out as is when we get to beta, at least not with the bit
of code that could possibly allow for a cluster-wide synchronized suicide.

Best,
Tobias

On Wed, Dec 17, 2014 at 10:19 AM, joezxy [email protected] wrote:

Hi, in RemoteClockMonitor.MonitorRemoteOffsets() implementation, if node's
clock offset goes beyond the maxOffset parameter, this node will Fatalf and
suicide.

Parameter maxOffset is set to 250ms by default, any reason for choosing
this number?
and as the design goal, Cockroachdb should be geo-distributed, then how to
set this number for cross-datacenter deployment if without some
infrastructure like Atom/GPS synchronized clock using in Spanner. Does
normal NTP work?

Waiting for your clarification..:-)


Reply to this email directly or view it on GitHub
#235.

@tbg
Copy link
Member

tbg commented Dec 17, 2014

BTW, for general questions, it's best to just mail directly to [email protected]. I'm closing this here, but feel free to follow-up on the thread.
Cheers,
Tobias

@tbg tbg closed this as completed Dec 17, 2014
@cockroach-team
Copy link

Thanks Tobias

tbg added a commit to tbg/cockroach that referenced this issue Aug 31, 2016
We have been seeing long startup times which disappear spontaneously. During a
restart of the beta cluster, the following goroutine was observed, which suggests
that we were spending a lot of time GCing replicas on startup.

    engine              ??:0                     _Cfunc_DBIterNext(cockroachdb#324, cockroachdb#323, 0, 0, 0, 0, 0, 0, 0)
    engine              rocksdb.go:1135          (*rocksDBIterator).Next(cockroachdb#235)
    storage             replica_data_iter.go:104 (*ReplicaDataIterator).Next(cockroachdb#316)
    storage             store.go:1748            (*Store).destroyReplicaData(#109, cockroachdb#317, 0, 0)
    storage             store.go:841             (*Store).Start.func2(0x101b, cockroachdb#300, 0x36, 0x40, cockroachdb#301, 0x36, 0x40, cockroachdb#315, 0x3, 0x4, ...)
    storage             store.go:734             IterateRangeDescriptors.func1(cockroachdb#306, 0x40, 0x41, cockroachdb#307, 0x92, 0x92, cockroachdb#341, 0, 0x186c, 0x4000, ...)
    engine              mvcc.go:1593             MVCCIterate(cockroachdb#329, #68, #47, #81, cockroachdb#232, 0x9, 0x10, cockroachdb#233, 0xb, 0x10, ...)
    storage             store.go:738             IterateRangeDescriptors(cockroachdb#330, cockroachdb#196, #47, #81, cockroachdb#195, #179, #110)
    storage             store.go:867             (*Store).Start(#109, cockroachdb#330, cockroachdb#196, #179, cockroachdb#185, 0x1)
    server              node.go:405              (*Node).initStores(#78, cockroachdb#330, cockroachdb#196, #98, 0x1, 0x1, #179, 0, #55)
    server              node.go:330              (*Node).start(#78, cockroachdb#330, cockroachdb#196, #42, #129, #98, 0x1, 0x1, 0, 0, ...)
    server              server.go:431            (*Server).Start(#5, cockroachdb#330, cockroachdb#196, #95, 0x1)
    cli                 start.go:368             runStart(#34, #178, 0, 0x9, 0, 0)
    cobra               command.go:599           (*Command).execute(#34, #177, 0x9, 0x9, #34, #177)
    cobra               command.go:689           (*Command).ExecuteC(#33, #70, cockroachdb#343, #72)
    cobra               command.go:648           (*Command).Execute(#33, #71, cockroachdb#343)
    cli                 cli.go:96                Run(#64, 0xa, 0xa, 0, 0)
    main                main.go:37               main()
tbg added a commit to tbg/cockroach that referenced this issue Aug 31, 2016
We have been seeing long startup times which disappear spontaneously. During a
restart of the beta cluster, the following goroutine was observed, which suggests
that we were spending a lot of time GCing replicas on startup.

    engine              ??:0                     _Cfunc_DBIterNext(cockroachdb#324, cockroachdb#323, 0, 0, 0, 0, 0, 0, 0)
    engine              rocksdb.go:1135          (*rocksDBIterator).Next(cockroachdb#235)
    storage             replica_data_iter.go:104 (*ReplicaDataIterator).Next(cockroachdb#316)
    storage             store.go:1748            (*Store).destroyReplicaData(#109, cockroachdb#317, 0, 0)
    storage             store.go:841             (*Store).Start.func2(0x101b, cockroachdb#300, 0x36, 0x40, cockroachdb#301, 0x36, 0x40, cockroachdb#315, 0x3, 0x4, ...)
    storage             store.go:734             IterateRangeDescriptors.func1(cockroachdb#306, 0x40, 0x41, cockroachdb#307, 0x92, 0x92, cockroachdb#341, 0, 0x186c, 0x4000, ...)
    engine              mvcc.go:1593             MVCCIterate(cockroachdb#329, #68, #47, #81, cockroachdb#232, 0x9, 0x10, cockroachdb#233, 0xb, 0x10, ...)
    storage             store.go:738             IterateRangeDescriptors(cockroachdb#330, cockroachdb#196, #47, #81, cockroachdb#195, #179, #110)
    storage             store.go:867             (*Store).Start(#109, cockroachdb#330, cockroachdb#196, #179, cockroachdb#185, 0x1)
    server              node.go:405              (*Node).initStores(#78, cockroachdb#330, cockroachdb#196, #98, 0x1, 0x1, #179, 0, #55)
    server              node.go:330              (*Node).start(#78, cockroachdb#330, cockroachdb#196, #42, #129, #98, 0x1, 0x1, 0, 0, ...)
    server              server.go:431            (*Server).Start(#5, cockroachdb#330, cockroachdb#196, #95, 0x1)
    cli                 start.go:368             runStart(#34, #178, 0, 0x9, 0, 0)
    cobra               command.go:599           (*Command).execute(#34, #177, 0x9, 0x9, #34, #177)
    cobra               command.go:689           (*Command).ExecuteC(#33, #70, cockroachdb#343, #72)
    cobra               command.go:648           (*Command).Execute(#33, #71, cockroachdb#343)
    cli                 cli.go:96                Run(#64, 0xa, 0xa, 0, 0)
    main                main.go:37               main()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants