scrapy爬虫框架 (2. logging模块的使用、yield scrapy.Request()函数间传参)

1.logging模块的使用

1.1scrapy项目中的使用

1.settings.py中设置LOG_LEVEL=“WARNING”
2.settings.py中设置LOG_FILE="./log.log" #这是日志保存的位置，设置后终端就不会显示日志内容
3.程序里进行日志输出：

import logging
logger=logging.getLogger(__name__) #实例化logger
class spider(scrapy.Spider):.......def parse(self,response):......logger.warning(item)

1.2普通项目

import logging
logging.basicConfig(…) #设置日志格式
logger=logging.getLogger(name) #实例化logger
在任何py文件中调用logger即可

2.同一函数间传数据（常用于翻页请求）

翻页请求：
1.找到下一页地址
2.组装成一个Request对象交给引擎

2.1首先爬取腾讯课堂第一页的代码如下

import scrapyclass NextpageSpider(scrapy.Spider):name = 'NextPage'allowed_domains = ['qq.com']start_urls = ['https://ke.qq.com/course/list?mt=1001&st=2002&tt=3019&price_min=1&page=1']def parse(self, response):datalist=response.xpath("//div[@class='market-bd market-bd-6 course-list course-card-list-multi-wrap js-course-list']//li")for data in datalist:item={
    }if data.xpath(".//h4//a/text()").extract_first() !=None:item["name"] = data.xpath(".//h4//a/text()").extract_first()item["price"] = data.xpath(".//span[@class='line-cell item-price custom-string']/text()").extract_first()item["number"] = data.xpath("normalize-space(.//span[@class='line-cell item-user custom-string']/text())").extract_first()print(item)

2.2找到下页的路径，并且注意最后一页的下页路径和第一页的下页路径不一样，根据这个来决定停止

第一页时下页的url地址：

最后一页时的下页地址：

根据这个url地址就可以通过yield构造Request对象传给引擎

# -*- coding: utf-8 -*-
import scrapyclass NextpageSpider(scrapy.Spider):name = 'NextPage'allowed_domains = ['qq.com']start_urls = ['https://ke.qq.com/course/list?mt=1001&st=2002&tt=3019&price_min=1&page=1']def parse(self, response):datalist=response.xpath("//div[@class='market-bd market-bd-6 course-list course-card-list-multi-wrap js-course-list']//li")for data in datalist:item={
    }if data.xpath(".//h4//a/text()").extract_first() !=None:item["name"] = data.xpath(".//h4//a/text()").extract_first()item["price"] = data.xpath(".//span[@class='line-cell item-price custom-string']/text()").extract_first()item["number"] = data.xpath("normalize-space(.//span[@class='line-cell item-user custom-string']/text())").extract_first()print(item)#找到下一页的url地址next_url=response.xpath("//a[@class='page-next-btn icon-font i-v-right']/@href").extract_first()if next_url != "javascript:void(0);" and next_url != None:#通过yield把地址包装成Request请求传向引擎yield scrapy.Request(next_url,callback=self.parse)#也可以是return scrapy.Request(next_url,callback=self.parse);因为return是将请求传回引擎，引擎会自动判断，如果是请求会再次爬取，是数据就会进行处理。else:print("end")

3.不同函数间传参

# -*- coding: utf-8 -*-
import scrapyclass NextpageSpider(scrapy.Spider):name = 'NextPage'allowed_domains = ['qq.com']start_urls = ['https://ke.qq.com/course/list?mt=1001&st=2002&tt=3019&price_min=1&page=1']def parse(self, response):datalist=response.xpath("//div[@class='market-bd market-bd-6 course-list course-card-list-multi-wrap js-course-list']//li")for data in datalist:item={
    }if data.xpath(".//h4//a/text()").extract_first() !=None:item["name"] = data.xpath(".//h4//a/text()").extract_first()item["price"] = data.xpath(".//span[@class='line-cell item-price custom-string']/text()").extract_first()item["number"] = data.xpath("normalize-space(.//span[@class='line-cell item-user custom-string']/text())").extract_first()yield scrapy.Request(next_url,callback=self.parse2,meta={
    "item":item}) ##把数据以字典形式通过meta参数传递给parse2函数def parse2(self,response):item=response.meta["item"]   ##取出数据

4.scrapy.Request知识点：