python爬虫入门，10分钟就够了，这可能是我见过最简单的基本教育
<div style="color: black; text-align: left; margin-bottom: 10px;">
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;"><span style="color: black;">1、</span><span style="color: black;">基本</span>入门</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">1.1什么是爬虫</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">爬虫(spider，又网络爬虫)，<span style="color: black;">指的是</span>向网站/网络发起请求，获取资源后分析并提取有用数据的程序。</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">从技术层面<span style="color: black;">来讲</span><span style="color: black;">便是</span> <span style="color: black;">经过</span>程序模拟浏览器请求站点的<span style="color: black;">行径</span>，把站点返回的HTML代码/JSON数据/二进制数据（<span style="color: black;">照片</span>、视频） 爬到本地，<span style="color: black;">从而</span>提取自己<span style="color: black;">必须</span>的数据，存放起来<span style="color: black;">运用</span>。</p>
<h1 style="color: black; text-align: left; margin-bottom: 10px;"><strong style="color: blue;">想要学习Python又缺少系统化学习<span style="color: black;">方法</span>的<span style="color: black;">伴侣</span><span style="color: black;">能够</span>关注并私信“01”免费领取系统化的Python学习资料...愿<span style="color: black;">大众</span>都能在编程这条路，越走越远。</strong></h1>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">1.2爬虫基本流程</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">用户获取网络数据的方式：</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">方式1：浏览器提交请求--->下载网页代码--->解析成页面</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">方式2：模拟浏览器发送请求(获取网页代码)->提取有用的数据->存放于数据库或文件中</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">爬虫要做的<span style="color: black;">便是</span>方式2。</p>
<div style="color: black; text-align: left; margin-bottom: 10px;"><img src="https://p3-sign.toutiaoimg.com/pgc-image/e60b4fb73f2246c58d633d7a7c8054e1~noop.image?_iz=58558&from=article.pc_detail&lk3s=953192f4&x-expires=1723890814&x-signature=VD904WjN3oaW6%2BqBjBf4PQBWizk%3D" style="width: 50%; margin-bottom: 20px;"></div>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">1</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">发起请求</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">运用</span>http库向<span style="color: black;">目的</span>站点发起请求，即发送一个Request</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">Request<span style="color: black;">包括</span>：请求头、请求体等 </p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">Request模块缺陷：<span style="color: black;">不可</span>执行JS 和CSS 代码</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">2</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">获取响应内容</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">倘若</span>服务器能正常响应，则会得到一个Response</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">Response<span style="color: black;">包括</span>：html，json，<span style="color: black;">照片</span>，视频等</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">3</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">解析内容</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">解析html数据：正则表达式（RE模块）、xpath（<span style="color: black;">重点</span><span style="color: black;">运用</span>）、beautiful soup、css</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">解析json数据：json模块</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">解析二进制数据:以wb的方式写入文件 </p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">4</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;"><span style="color: black;">保留</span>数据</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">数据库（MySQL，Mongdb、Redis）或 文件的形式。</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">1.3http协议 请求与响应</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">http协议</p>
<div style="color: black; text-align: left; margin-bottom: 10px;"><img src="https://p3-sign.toutiaoimg.com/pgc-image/23837571caec4b0d8e0632d1f1ca0f14~noop.image?_iz=58558&from=article.pc_detail&lk3s=953192f4&x-expires=1723890814&x-signature=Dl82WmyKEvtAuIg1BFcaNOuuE2A%3D" style="width: 50%; margin-bottom: 20px;"></div>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">Request</strong>：用户将自己的信息<span style="color: black;">经过</span>浏览器（socket client）发送给服务器（socket server）</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">Response：服务器接收请求，分析用户发来的请求信息，<span style="color: black;">而后</span>返回数据（返回的数据中可能<span style="color: black;">包括</span>其他链接，如：<span style="color: black;">照片</span>，js，css等）</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">ps：浏览器在接收Response后，会解析其内容来<span style="color: black;">表示</span>给用户，而爬虫程序在模拟浏览器发送请求<span style="color: black;">而后</span>接收Response后，是要提取其中的有用数据。</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">1.3.1</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"> <strong style="color: blue;">request</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"> (1) 请求方式</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">平常</span>的请求方式：GET / POST</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">（2）请求的URL</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">url<span style="color: black;">全世界</span>统一资源定位符，用来定义互联网上一个<span style="color: black;">独一</span>的资源 例如：一张<span style="color: black;">照片</span>、一个文件、一段视频都<span style="color: black;">能够</span>用url唯一确定</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">（3）请求头</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">User-agent：请求头中<span style="color: black;">倘若</span><span style="color: black;">无</span>user-agent客户端配置，服务端可能将你当做一个<span style="color: black;">违法</span>用户host；</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">cookies：cookie用来<span style="color: black;">保留</span>登录信息</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">重视</span>：<span style="color: black;">通常</span>做爬虫都会加上请求头。</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">请求头<span style="color: black;">必须</span><span style="color: black;">重视</span>的参数：</p>Referrer：<span style="color: black;">拜访</span>源至哪里来（<span style="color: black;">有些</span>大型网站，会<span style="color: black;">经过</span>Referrer 做防盗链策略；所有爬虫<span style="color: black;">亦</span>要<span style="color: black;">重视</span>模拟）User-Agent:<span style="color: black;">拜访</span>的浏览器（要加上否则会被当成爬虫程序） cookie：请求头<span style="color: black;">重视</span>携带<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">（4）请求体</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">请求体 <span style="color: black;">倘若</span>是get方式，请求体<span style="color: black;">无</span>内容 （get请求的请求体放在 url后面参数中，直接能看到） <span style="color: black;">倘若</span>是post方式，请求体是format data</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">ps：1、登录窗口，文件上传等，信息都会被附加到请求<span style="color: black;">身体</span> 2、登录，输入错误的用户名<span style="color: black;">秘码</span>，<span style="color: black;">而后</span>提交，就<span style="color: black;">能够</span>看到post，正确登录后页面<span style="color: black;">一般</span>会<span style="color: black;">转</span>，<span style="color: black;">没法</span><span style="color: black;">捉捕</span>到post</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">1.3.2</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">response</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;"> （1）响应状态码</strong></p>200：<span style="color: black;">表率</span>成功301：<span style="color: black;">表率</span><span style="color: black;">转</span>404：文件不存在403：无权限<span style="color: black;">拜访</span>502：服务器错误<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">（2）response header</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">响应头<span style="color: black;">必须</span><span style="color: black;">重视</span>的参数：Set-Cookie:BDSVRTM=0; path=/：可能有多个，是来告诉浏览器，把cookie<span style="color: black;">保留</span>下来</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">（3）preview<span style="color: black;">便是</span>网页源代码</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"> json数据</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"> 如网页html，<span style="color: black;">照片</span></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"> 二进制数据等 </p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">02</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;"><span style="color: black;">2、</span><span style="color: black;">基本</span>模块</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">2.1requests</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">requests是python实现的简单易用的HTTP库,<span style="color: black;">是由于</span>urllib的升级而来。</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">开源<span style="color: black;">位置</span>：</p>https://github.com/kennethreitz/requests
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">中文API：</p>http://docs.python-requests.org/zh_CN/latest/index.html
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">2.2re 正则表达式</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">在 Python 中<span style="color: black;">运用</span>内置的 re 模块来<span style="color: black;">运用</span>正则表达式。</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">缺点：处理数据不稳定、工作量大</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">2.3XPath</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">Xpath(XML Path Language) 是一门在 XML 文档中<span style="color: black;">查询</span>信息的语言，可用来在 XML 文档中对元素和属性进行遍历。</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">在python中<span style="color: black;">重点</span><span style="color: black;">运用</span> lxml 库来进行xpath获取（在框架中不<span style="color: black;">运用</span>lxml，框架内直接<span style="color: black;">运用</span>xpath<span style="color: black;">就可</span>）</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">lxml 是 一个HTML/XML的解析器，<span style="color: black;">重点</span>的功能是<span style="color: black;">怎样</span>解析和提取 HTML/XML 数据。</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">lxml和正则<span style="color: black;">同样</span>，<span style="color: black;">亦</span>是用 C 实现的，是一款高性能的 Python HTML/XML 解析器，<span style="color: black;">咱们</span><span style="color: black;">能够</span>利用之前学习的XPath语法，来快速的定位特定元素以及节点信息。</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">2.4BeautifulSoup</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">和 lxml <span style="color: black;">同样</span>，Beautiful Soup <span style="color: black;">亦</span>是一个HTML/XML的解析器，<span style="color: black;">重点</span>的功能<span style="color: black;">亦</span>是<span style="color: black;">怎样</span>解析和提取 HTML/XML 数据。</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><span style="color: black;">运用</span>BeautifulSoup<span style="color: black;">必须</span>导入bs4库</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">缺点：相对正则和xpath处理速度慢</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">优点：<span style="color: black;">运用</span>简单</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">2.5Json</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">JSON(JavaScript Object Notation) 是一种轻量级的数据交换格式，它使得人们很容易的进行阅读和编写。<span style="color: black;">同期</span><span style="color: black;">亦</span>方便了<span style="color: black;">设备</span>进行解析和生成。适用于进行数据交互的场景，<span style="color: black;">例如</span>网站前台与后台之间的数据交互。</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">在python中<span style="color: black;">重点</span><span style="color: black;">运用</span> json 模块来处理 json数据。Json解析网站：</p>https://www.sojson.com/simple_json.html<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">2.6threading</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;"><span style="color: black;">运用</span>threading模块创建线程，直接从threading.Thread继承，<span style="color: black;">而后</span>重写__init__<span style="color: black;">办法</span>和run<span style="color: black;">办法</span></strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">03</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;"><span style="color: black;">3、</span><span style="color: black;">办法</span>实例</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">3.1get<span style="color: black;">办法</span>实例</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">demo_get.py</strong></p>
<div style="color: black; text-align: left; margin-bottom: 10px;"><img src="https://p3-sign.toutiaoimg.com/pgc-image/468c5b242b854b8680e072e7c1b64601~noop.image?_iz=58558&from=article.pc_detail&lk3s=953192f4&x-expires=1723890814&x-signature=I9O0ZPNjYQiuuWSN3qFLEj6Tes8%3D" style="width: 50%; margin-bottom: 20px;"></div>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">3.2post<span style="color: black;">办法</span>实例</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">demo_post.py</p>
<div style="color: black; text-align: left; margin-bottom: 10px;"><img src="https://p3-sign.toutiaoimg.com/pgc-image/cea8e8dfc7c44a3992e8a3949b913f64~noop.image?_iz=58558&from=article.pc_detail&lk3s=953192f4&x-expires=1723890814&x-signature=pzzEYAnpQwxgxCV47afff%2FJ1Y0Y%3D" style="width: 50%; margin-bottom: 20px;"></div>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">3.3添加代理</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">demo_proxies.py</p>
<div style="color: black; text-align: left; margin-bottom: 10px;"><img src="https://p3-sign.toutiaoimg.com/pgc-image/49f63acb6e96431a9d4bd313c569a93d~noop.image?_iz=58558&from=article.pc_detail&lk3s=953192f4&x-expires=1723890814&x-signature=RzWJkXZx%2FxZxWkaRLV5mqk15tbg%3D" style="width: 50%; margin-bottom: 20px;"></div>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">3.4获取ajax类数据实例</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">demo_ajax.py</strong></p>
<div style="color: black; text-align: left; margin-bottom: 10px;"><img src="https://p3-sign.toutiaoimg.com/pgc-image/4ad6f616eb3f4eae8d9ac48f8a29f80b~noop.image?_iz=58558&from=article.pc_detail&lk3s=953192f4&x-expires=1723890814&x-signature=fo6qOf2zYyqaC1tsnbjP4UCIGZ0%3D" style="width: 50%; margin-bottom: 20px;"></div>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">3.5<span style="color: black;">运用</span>多线程实例</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">demo_thread.py</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">04</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;"><span style="color: black;">4、</span>爬虫框架</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">4.1Srcapy框架</strong></p>Scrapy是用纯Python实现一个为了爬取网站数据、提取结构性数据而编写的应用框架，用途非常广泛。Scrapy <span style="color: black;">运用</span>了 Twisted(其<span style="color: black;">重点</span>对手是Tornado)异步网络框架来处理网络通讯，<span style="color: black;">能够</span>加快<span style="color: black;">咱们</span>的下载速度，<span style="color: black;">不消</span>自己去实现异步框架，并且<span style="color: black;">包括</span>了<span style="color: black;">各样</span>中间件接口，<span style="color: black;">能够</span>灵活的完成<span style="color: black;">各样</span>需求。<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">4.2Scrapy架构图</strong></p>
<div style="color: black; text-align: left; margin-bottom: 10px;"><img src="https://p3-sign.toutiaoimg.com/pgc-image/1a73dcb5dc8544f69cd75ab44689cc0d~noop.image?_iz=58558&from=article.pc_detail&lk3s=953192f4&x-expires=1723890814&x-signature=dZRAwrpePo%2F29R87EiyH%2F7Xn%2Bfc%3D" style="width: 50%; margin-bottom: 20px;"></div>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">4.3Scrapy<span style="color: black;">重点</span>组件</strong></p>Scrapy Engine(引擎): 负责Spider、ItemPipeline、Downloader、Scheduler中间的通讯，信号、数据传递等。Scheduler(调度器): 它负责接受引擎发送过来的Request请求，并<span style="color: black;">根据</span><span style="color: black;">必定</span>的方式进行整理<span style="color: black;">摆列</span>，入队，当引擎<span style="color: black;">必须</span>时，交还给引擎。Downloader（下载器）：负责下载Scrapy Engine(引擎)发送的所有Requests请求，并将其获取到的Responses交还给Scrapy Engine(引擎)，由引擎交给Spider来处理，Spider（爬虫）：它负责处理所有Responses,从中分析提取数据，获取Item字段<span style="color: black;">必须</span>的数据，并将<span style="color: black;">必须</span>跟进的URL提交给引擎，再次进入Scheduler(调度器)，Item Pipeline(管道)：它负责处理Spider中获取到的Item，并进行进行后期处理（<span style="color: black;">仔细</span>分析、过滤、存储等）的<span style="color: black;">地区</span>.Downloader Middlewares（下载中间件）：你<span style="color: black;">能够</span>当作是一个<span style="color: black;">能够</span>自定义扩展下载功能的组件。Spider Middlewares（Spider中间件）：你<span style="color: black;">能够</span>理解为是一个<span style="color: black;">能够</span>自定扩展和操作引擎和Spider中间通信的功能组件（<span style="color: black;">例如</span>进入Spider的Responses;和从Spider出去的Requests）<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">4.4Scrapy的运作流程</strong></p>引擎：Hi！Spider, 你要处理哪一个网站？Spider：老大要我处理xxxx.com。引擎：你把<span style="color: black;">第1</span>个<span style="color: black;">必须</span>处理的URL给我吧。Spider：给你，<span style="color: black;">第1</span>个URL是xxxxxxx.com。引擎：Hi！调度器，我这有request请求你帮我排序入队一下。调度器：好的，正在处理你等一下。引擎：Hi！调度器，把你处理好的request请求给我。调度器：给你，这是我处理好的request引擎：Hi！下载器，你<span style="color: black;">根据</span>老大的下载中间件的设置帮我下载一下这个request请求下载器：好的！给你，这是下载好的东西。（<span style="color: black;">倘若</span>失败：sorry，这个request下载失败了。<span style="color: black;">而后</span>引擎告诉调度器，这个request下载失败了，你记录一下，<span style="color: black;">咱们</span>待会儿再下载）引擎：Hi！Spider，这是下载好的东西，并且<span style="color: black;">已然</span><span style="color: black;">根据</span>老大的下载中间件处理过了，你自己处理一下（<span style="color: black;">重视</span>！这儿responses默认是交给def parse()这个函数处理的）Spider：（处理完毕数据之后<span style="color: black;">针对</span><span style="color: black;">必须</span>跟进的URL），Hi！引擎，我<span style="color: black;">这儿</span>有两个结果，这个是我<span style="color: black;">必须</span>跟进的URL，还有这个是我获取到的Item数据。引擎：Hi ！管道 我这儿有个item你帮我处理一下！调度器！这是<span style="color: black;">必须</span>跟进URL你帮我处理下。<span style="color: black;">而后</span>从第四步<span style="color: black;">起始</span>循环，直到获取完老大<span style="color: black;">必须</span><span style="color: black;">所有</span>信息。管道``调度器：好的，<span style="color: black;">此刻</span>就做！<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">4.5制作Scrapy爬虫4步曲</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">1新建爬虫项目scrapy startproject mySpider2<span style="color: black;">知道</span><span style="color: black;">目的</span> （编写items.py）打开mySpider目录下的items.py3制作爬虫 （spiders/xxspider.py）scrapy genspider gushi365 "gushi365.com"4存储内容 （pipelines.py）设计管道存储爬取内容</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">05</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;"><span style="color: black;">5、</span>常用工具</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">5.1fidder</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">fidder是一款抓包工具，<span style="color: black;">重点</span>用于手机抓包。</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">5.2XPath Helper</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">xpath helper插件是一款免费的chrome爬虫网页解析工具。<span style="color: black;">能够</span><span style="color: black;">帮忙</span>用户<span style="color: black;">处理</span>在获取xpath路径时<span style="color: black;">没法</span>正常定位等问题。</p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">谷歌浏览器插件xpath helper 的安装和<span style="color: black;">运用</span>：</strong></p>https://jingyan.baidu.com/article/1e5468f94694ac484861b77d.html<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">06</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;"><span style="color: black;">6、</span>分布式爬虫</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">6.1scrapy-redis</strong></p>
<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;">Scrapy-redis是为了更方便地实现Scrapy分布式爬取，而<span style="color: black;">供给</span>了<span style="color: black;">有些</span>以redis为<span style="color: black;">基本</span>的组件（pip install scrapy-redis）</p>github网站：https://github.com/rolando/scrapy-redis<p style="font-size: 16px; color: black; line-height: 40px; text-align: left; margin-bottom: 15px;"><strong style="color: blue;">6.2分布式策略</strong></p>Master端(核心服务器) ：搭建一个Redis数据库，不负责爬取，只负责url指纹判重、Request的分配，以及数据的存储Slaver端(爬虫程序执行端) ：负责执行爬虫程序，运行过程中提交新的Request给Master
</div>
谢谢、感谢、感恩、辛苦了、有你真好等。 论坛的成功是建立在我们诚恳、务实、高效、创新和团结合作基础上，我们要把这种精神传递下去。 一看到楼主的气势，我就觉得楼主同在社区里灌水。
页:
[1]