Crawler整体架构
2016-01-30 10:40:10 0 举报爬虫框架设计
作者其他创作
大纲/内容
Init JobDB
PersistenceDB
z1
JobProducer3
put-msg
Zookeeper
Internet
result
url-msg
http
save
page
partition3
JobProducers
update
分布式消息队列
monitor
手动添加
SpiderConsumer
SavePipeline
PageAnalyzer
get-msg
suburl
SpiderScheduler 2
partition2
AdminWeb
partition1
SpiderScheduler 1
z3
Downloader
get
JobProducer1
z2
DB Cluster
URLManager
JobProducer2
0 条评论
下一页
