scrapy的大概运行流程
2024-04-02 10:42:15 0 举报 使用 
AI智能生成
为你推荐
查看更多
根据站里大佬的源码剖析，画的一张图，只有大概的逻辑
作者其他创作
大纲/内容
scrapy.cmdline.py
scrapy.utils.project.py
scrapy.settings.__init__.py
scrapy.settings.default_settings.py（部分展示）
分支主题
scrapy.command.crawl.py
scrapy.command.__init__.py
scrapy.cralwer.py
子主题
scrpay.spiders.__init__.py
scrapy.core.engine.py
scrapy.core.downloader.py
scrapy.core.downloader.handlers.__init__.py
自由主题
_add_middleware
DownloaderMiddlewareManager
scrapy.core.scraper.py
scrapy.core.spidermw.py
scrapy.pipelines.__init__.py
scrapy.middleware.py
start_requests
spider
open_spider
from_crawler
open
enqueue_request
_dqpush
scrapy.core.scheduler.py
scrapy.utils.misc.py（创建对象的一个方法）
request_seen
scrapy.duperfilters.py
部分可更改的队列设置
scrapy.reactor.py
scrapy.dupefilter.py
Scraper.open_spider
ExecutionEngine._next_request
ExecutionEngine._needs_backout
ExecutionEngine._next_request_from_scheduler
ExecutionEngine.crawl
ExecutionEngine._schedule_request
scrapy.utils.request.py
ExecutionEngine._download
scrapy.downloader.__init__.py Downloader
scrapy.downloader.middleware.py
scrapy.downloader.handlers.__init__.py DownloadHandlers
scrapy.downloader.handlers.__init__.py DownloadHandlers
ExecutionEngine._handle_downloader_output
scrapy.core.scraper.py Scraper
scrapy.core.scraper.py Scraper
scrapy.core.spidermw.py SpiderMiddlewareManager
scrapy.__main__.py
收藏
0 条评论
回复 删除
下一页
