-
Notifications
You must be signed in to change notification settings - Fork 3.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add new elastic job lite doc for 03-design.
- Loading branch information
Showing
4 changed files
with
172 additions
and
0 deletions.
There are no files selected for viewing
15 changes: 15 additions & 0 deletions
15
elastic-job-doc/elastic-job-lite-doc/content/03-design/index.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
+++ | ||
icon = "<b>3. </b>" | ||
date = "2016-12-12T16:06:17+08:00" | ||
title = "设计规划" | ||
weight = 0 | ||
prev = "/02-guide/customized-hook/" | ||
next = "/03-design/module/" | ||
chapter = true | ||
|
||
+++ | ||
|
||
# 本章导航 | ||
|
||
- 想了解架构设计及项目模块设计的概念,请阅读[实现原理](/03-design/lite-design)及[目录结构说明](/03-design/module)。 | ||
- Elastic-Job未来规划有哪些呢?请阅读[未来规划](/03-design/roadmap)。 |
81 changes: 81 additions & 0 deletions
81
elastic-job-doc/elastic-job-lite-doc/content/03-design/lite-design.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,81 @@ | ||
+++ | ||
toc = true | ||
date = "2016-01-28T16:14:21+08:00" | ||
title = "实现原理" | ||
weight=3 | ||
prev = "/03-design/module/" | ||
next = "/03-design/roadmap/" | ||
+++ | ||
|
||
## 弹性分布式实现 | ||
|
||
* 第一台服务器上线触发主服务器选举。主服务器一旦下线,则重新触发选举,选举过程中阻塞,只有主服务器选举完成,才会执行其他任务。 | ||
|
||
* 某作业服务器上线时会自动将服务器信息注册到注册中心,下线时会自动更新服务器状态。 | ||
|
||
* 主节点选举,服务器上下线,分片总数变更均更新重新分片标记。 | ||
|
||
* 定时任务触发时,如需重新分片,则通过主服务器分片,分片过程中阻塞,分片结束后才可执行任务。如分片过程中主服务器下线,则先选举主服务器,再分片。 | ||
|
||
* 通过上一项说明可知,为了维持作业运行时的稳定性,运行过程中只会标记分片状态,不会重新分片。分片仅可能发生在下次任务触发前。 | ||
|
||
* 每次分片都会按服务器IP排序,保证分片结果不会产生较大波动。 | ||
|
||
* 实现失效转移功能,在某台服务器执行完毕后主动抓取未分配的分片,并且在某台服务器下线后主动寻找可用的服务器执行任务。 | ||
|
||
## 注册中心数据结构 | ||
|
||
注册中心在定义的命名空间下,创建作业名称节点,用于区分不同作业,所以作业一旦创建则不能修改作业名称,如果修改名称将视为新的作业。作业名称节点下又包含4个数据子节点,分别是config, instances, sharding, servers和leader。 | ||
|
||
### config节点 | ||
|
||
作业配置信息,以JSON格式存储 | ||
|
||
### instances节点 | ||
|
||
作业运行实例信息,子节点是当前作业运行实例的主键。作业运行实例主键由作业运行服务器的IP地址和UUID构成。作业运行实例主键均为临时节点,当作业实例上线时注册,下线时自动清理。注册中心监控这些节点的变化来协调分布式作业的分片以及高可用。 | ||
可在作业运行实例节点写入TRIGGER表示该实例立即执行一次。 | ||
|
||
### sharding节点 | ||
|
||
作业分片信息,子节点是分片项序号,从零开始,至分片总数减一。分片项序号的子节点存储详细信息。每个分片项下的子节点用于控制和记录分片运行状态。节点详细信息说明: | ||
|
||
| 子节点名 | 临时节点 | 描述 | | ||
| -------------------------------- |:--------|:----------------------------------------------------------------------------| | ||
| instance | 否 | 执行该分片项的作业运行实例主键 | | ||
| running | 是 | 分片项正在运行的状态<br />如果没有此节点,并且没有completed节点,表示该分片未运行<br />仅配置monitorExecution时有效 | | ||
| completed | 否 | 分片项运行完成的状态<br />下次作业开始执行时会清理<br />仅配置monitorExecution时有效 | | ||
| failover | 是 | 如果该分片项被失效转移分配给其他作业服务器,则此节点值记录执行此分片的作业服务器IP | | ||
| misfire | 否 | 是否开启错过任务重新执行 | | ||
| disabled | 否 | 是否禁用此分片项 | | ||
|
||
### servers节点 | ||
|
||
作业服务器信息,子节点是作业服务器的IP地址。可在IP地址节点写入DISABLED表示该服务器禁用。 | ||
在新的cloud native架构下,servers节点大幅弱化,仅包含控制服务器是否可以禁用这一功能。为了更加纯粹的实现job核心,servers功能未来可能删除,控制服务器是否禁用的能力应该下放至自动化部署系统。 | ||
|
||
### leader节点 | ||
|
||
作业服务器主节点信息,分为election,sharding和failover三个子节点。分别用于主节点选举,分片和失效转移处理。 | ||
|
||
leader节点是内部使用的节点,如果对作业框架原理不感兴趣,可不关注此节点。 | ||
|
||
| 子节点名 | 临时节点 | 描述 | | ||
| -------------------------------- |:--------|:----------------------------------------------------------------------------| | ||
| election\instance | 是 | 主节点服务器IP地址<br />一旦该节点被删除将会触发重新选举<br />重新选举的过程中一切主节点相关的操作都将阻塞 | | ||
| election\latch | 否 | 主节点选举的分布式锁<br />为curator的分布式锁使用 | | ||
| sharding\necessary | 否 | 是否需要重新分片的标记<br />如果分片总数变化,或作业服务器节点上下线或启用/禁用,以及主节点选举,会触发设置重分片标记<br />作业在下次执行时使用主节点重新分片,且中间不会被打断<br />作业执行时不会触发分片| | ||
| sharding\processing | 是 | 主节点在分片时持有的节点<br />如果有此节点,所有的作业执行都将阻塞,直至分片结束<br />主节点分片结束或主节点崩溃会删除此临时节点 | | ||
| sharding\cleaning | 是 | 主节点在清理上次作业运行时状态时所持有的节点<br />每次开始新作业都需要清理上次运行完成的作业信息<br />如果有此节点,所有的作业执行都将阻塞,直至清理结束<br />主节点分片结束或主节点崩溃会删除此临时节点 | | ||
| failover\items\分片项 | 否 | 一旦有作业崩溃,则会向此节点记录<br />当有空闲作业服务器时,会从此节点抓取需失效转移的作业项 | | ||
| failover\items\latch | 否 | 分配失效转移分片项时占用的分布式锁<br />为curator的分布式锁使用 | | ||
|
||
## 流程图 | ||
|
||
### 作业启动 | ||
|
||
![作业启动](/img/principles/job_start.jpg) | ||
|
||
### 作业执行 | ||
|
||
![作业执行](/img/principles/job_exec.jpg) |
25 changes: 25 additions & 0 deletions
25
elastic-job-doc/elastic-job-lite-doc/content/03-design/module.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
+++ | ||
toc = true | ||
date = "2016-12-06T22:38:50+08:00" | ||
title = "目录结构说明" | ||
weight = 2 | ||
prev = "/03-design/index/" | ||
next = "/03-design/lite-design/" | ||
+++ | ||
|
||
``` | ||
elastic-job | ||
├──elastic-job-lite lite父模块,不应直接使用 | ||
├ ├──elastic-job-lite-core Java支持模块,可直接使用 | ||
├ ├──elastic-job-lite-spring Spring命名空间支持模块,可直接使用 | ||
├ ├──elastic-job-lite-lifecyle lite作业相关操作模块,不可直接使用 | ||
├ ├──elastic-job-lite-console lite界面模块,可直接使用 | ||
├──elastic-job-example 使用示例 | ||
├ ├──elastic-job-example-embed-zk 供示例使用的内嵌ZK模块 | ||
├ ├──elastic-job-example-jobs 作业示例 | ||
├ ├──elastic-job-example-lite-java 基于Java的使用示例 | ||
├ ├──elastic-job-example-lite-spring 基于Spring的使用示例 | ||
├ ├──elastic-job-example-lite-springboot 基于SpringBoot的使用示例 | ||
├──elastic-job-doc markdown生成文档的项目,使用方无需关注 | ||
├ ├──elastic-job-lite-doc lite相关文档 | ||
``` |
51 changes: 51 additions & 0 deletions
51
elastic-job-doc/elastic-job-lite-doc/content/03-design/roadmap.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,51 @@ | ||
+++ | ||
toc = true | ||
date = "2016-12-06T22:38:50+08:00" | ||
title = "未来线路规划" | ||
weight = 4 | ||
prev = "/03-design/lite-design/" | ||
next = "/00-overview/index/" | ||
+++ | ||
|
||
## Core | ||
- [x] Unified Job Config API | ||
- [x] Core Config | ||
- [x] Type Config | ||
- [x] Root Config | ||
- [x] Job Types | ||
- [x] Simple | ||
- [x] Dataflow | ||
- [x] Script | ||
- [ ] Http | ||
- [x] Event Trace | ||
- [x] Event Publisher | ||
- [x] Database Event Listener | ||
- [ ] Other Event Listener | ||
- [ ] Unified Schedule API | ||
- [ ] Unified Resource API | ||
|
||
## Elastic-Job-Lite | ||
- [x] Distributed Features | ||
- [x] High Availability | ||
- [x] Elastic scale in/out | ||
- [x] Failover | ||
- [x] Misfire | ||
- [x] Idempotency | ||
- [x] Reconcile | ||
- [x] Registry Center | ||
- [x] Zookeeper | ||
- [ ] Other Registry Center Supported | ||
- [x] Lifecycle Management | ||
- [x] Add/Remove | ||
- [x] Pause/Resume | ||
- [x] Disable/Enable | ||
- [x] Shutdown | ||
- [x] Restful API | ||
- [x] Web Console | ||
- [x] Job Dependency | ||
- [x] Listener | ||
- [ ] Workflow | ||
- [ ] DAG | ||
- [x] Spring Integrate | ||
- [x] Namespace | ||
- [x] Bean Injection |