Yarn 框架(spark on yarn cluster模式为例)
2020-12-16 15:13:24 0 举报
Yarn框架调度原理(spark on yarn cluster)
作者其他创作
大纲/内容
min-allocation-mb
⑩
reducetask[cpu+ram +jar](Excuter)
......
maptask[cpu+ram + jar](Executor)
on-heap
⑤
off-heap
WordCount.jar
⑪
root
⑨
ResourceManager
④
task3
②
Container
⑫
spark Executor内存模型(Capacity Scheduler调度器的场合)
NodeManager
(max-allocation-mb)
spark on YARN Cluster模式
ExecutionMemory
UnifiedMemory
Queue
AppMaster[cpu+ram](driver)
Executor
task4
YARNRUNNER
①
YARN客户端
规整化因子
UserMemory
容器和重要配置参数(cluster模式driver和appmter同一容器)利用cgroup做cpu的隔离调度容器最小调度单元参数设值yarn.scheduler.minimum(max)-allocation-mbyarn.scheduler.minimum(max)-allocation-vcores
task5
maptask[cpu+ram +jar](Executor)
⑦
⑧
⑥
task1
③
StorageMemory
HDFS
Reserved Memory
task2
que1
职业：大专
0 条评论
回复 删除
下一页
