StreamSet
2018-12-14 14:02:29 0 举报
StreamSet启动与集群化
作者其他创作
大纲/内容
Pipeline多节点启动
Poll Origin Stage
单节点
Create Every PipeRunners without Source Stage
Write Batch
Master
MetricsEventRunnable
Push Origin Stage
Change State And Report Error
STATE_RUNNING_ERROR
LOOP
Pipeline Source Data Cache
StageRuntime
Pipeline
Production Pipeline Runner
Request Data
1、HeartBeat，使Master感知HeartBeat，持久化存储于磁盘2、RelationOfPipelineAndSlave，Pipeline已经下发到了哪些节点上，持久化存储于磁盘3、PIpelineStatusAndInfo，各节点的Pipeline的信息和状态，缓存在内存中
STATE_FINISHING
To Go
Set
集群Pipeline启动---Master
Push Source
1、配置热加载2、数据分流3、HA4、Slave的Metrics上报
Stage Context1-2
单机Pipeline启动
Data1
More
PipelineState
集群Pipeline启动---Slave
Process Batch
Found Num of Threads
启动
ThreadHealthReporter
Master与Slave同一节点
Message Arrived
ObserverRunnable
IN
Process
Write Queue
Stage Context
Data Queue
Init
Request
Push or Poll Source Stage
Remote Source
Set Error Listener
Runner
Stage Runtime2
Rest Stage
Stage Runtime
PipelineStore
Init Source Stage
New Thread Run
Master与Slave同步与通讯
分配Pipeline下发节点
Push or Poll?
Init Connection
Stage Runtime3
STATE_RUNNING
启动模式
Checking Error
Produce/Init Connection
Stage Context2-3
STATE_FINISHED
OUTIN
Got Exception
STATE_XXX_ERROR
Found Errors
1、Pipeline下发到Slave并启动2、前端展示Pipeline的内容和Slave上的运行状态3、删除Pipeline时删除集群里的全部该PIpeline
Run
判断启动模式
Master不允许出现在Slave列表中
Slave请求Master的Source并执行后续Stage
To Next
Write File
读取Pipeline的Source Stage并写入缓存
Stage Runtime1
Manager
STATE_ERROR
Destory
Manager Resource
Click
Create Every StageRuntime
ProductionPipelineRunnable
OUT
下发Pipeline
Execute
多节点
Pipe Batch
StandaloneRunner
Got Error
Observer and Listener
Get Pipelines
Push or Poll Source
Catch Request
Poll Source Stage
0 条评论
回复 删除
下一页
