伦理电影全集-电影网

一些requests的基本用法

1.无参get

2.带参get

3.解析json

4.获取二进制数据

5.添加headers

6.基本post请求

7.response属性

8.状态码判断

9.文件上传

10.获取cookie

11.会话维持

12.证书验证

13.代理设置

14.超时设置

15.认证设置

1.无参get

import requests
response = requests.get("http://httpbin.org/get")
print(response.text)

输出结果为：

import requests
response = requests.get("http://httpbin.org/get")
print(response.text)

2.带参get

利用拼接字符串的方式实现带参get

import requests
response = requests.get("http://httpbin.org/get?name=123&age=22")
print(response.text)

result:
{
  "args": {
    "age": "22", 
    "name": "123"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.25.1", 
    "X-Amzn-Trace-Id": "Root=1-5fed25f3-77619660052366ab626d4517"
  }, 
  "origin": "111.14.9.241", 
  "url": "http://httpbin.org/get?name=123&age=22"
}

# 当然我们也可以用这种字典写法，结果和上面也是一样的：
import requests
data = {
    'name':123,
    'age':22
}
response = requests.get("http://httpbin.org/get",params=data)
print(response.text)

3.解析json

#ajax请求比较常用json
import requests
import json
response = requests.get("http://httpbin.org/get")
print(type(response.text))
print(response.json())
print(json.loads(response.text))
print(type(response.json()))

result:
<class 'str'>
{'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.25.1', 'X-Amzn-Trace-Id': 'Root=1-5fed2745-2ec707af69effb4d7989da28'}, 'origin': '111.14.9.241', 'url': 'http://httpbin.org/get'}
{'args': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.25.1', 'X-Amzn-Trace-Id': 'Root=1-5fed2745-2ec707af69effb4d7989da28'}, 'origin': '111.14.9.241', 'url': 'http://httpbin.org/get'}
<class 'dict'>
#可以看到response.json()和json.loads(response.text)方法的作用是一样的

4.获取二进制数据

import requests

resopnse = requests.get("https://github.com/favicon.ico")
print(type(response.text)),type(response.content)
print(response.text)
print(response.content)

result:
<class 'str'>
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.25.1", 
    "X-Amzn-Trace-Id": "Root=1-5fed2745-2ec707af69effb4d7989da28"
  }, 
  "origin": "111.14.9.241", 
  "url": "http://httpbin.org/get"
}

b'{\n  "args": {}, \n  "headers": {\n    "Accept": "*/*", \n    "Accept-Encoding": "gzip, deflate", \n    "Host": "httpbin.org", \n    "User-Agent": "python-requests/2.25.1", \n    "X-Amzn-Trace-Id": "Root=1-5fed2745-2ec707af69effb4d7989da28"\n  }, \n  "origin": "111.14.9.241", \n  "url": "http://httpbin.org/get"\n}\n'
#将二进制数据输出为图片
response = requests.get("https://github.com/favicon.ico")
with open('favicon.ico','wb') as f:
    f.write(response.content)
    f.close

5.添加headers

import requests
response = requests.get("https://www.zhihu.com/explore")
print(response.text)
result:
<html>
<head><title>403 Forbidden</title></head>
<body bgcolor="white">
<center><h1>403 Forbidden</h1></center>
<hr><center>openresty</center>
</body>
</html>

#根据结果我们可以看到，此时我们并没有成功请求到数据，这时候我们添加headers再去访问
headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36'
}
response =requests.get("https://www.zhihu.com/explore",headers=headers)
print(response.text)
 result：
    <!doctype html>
<html lang="zh" data-hairline="true" data-theme="light"><head><meta charSet="utf-8"/><title data-react-helmet="true">发现 - 知乎</title><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1"/><meta name="renderer" content="webkit"/><meta name="force-rendering" content="webkit"/><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"/><meta name="google-site-verification" content="FTeR0c8arOPKh8c5DYh_9uu98_zJbaWw53J-Sch9MTg"/><meta name="description" property="og:description" content="有问题，上知乎。知乎，可信赖的问答社区，以让每个人高效获得可信赖的解答为使命。知乎凭借认真、专业和友善的社区氛围，结构化、易获得的优质内容，基于问答的内容生产方式和独特的社区机制，吸引、聚集了各行各业中大量的亲历者、内行人、领域专家、领域爱好者，将高质量的内容透过人的节点来成规模地生产和分享。用户通过问答等交流方式建立信任和连接，打造和提升个人影响力，并发现、获得新机会。"/><link data-react-helmet="true" >

6.基本post请求

import requests
data ={'name':'123',"age":'22'}
response = requests.post("http://httpbin.org/post",data=data)
print(response.text)

result：
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "age": "22", 
    "name": "123"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "15", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.25.1", 
    "X-Amzn-Trace-Id": "Root=1-5fed2a55-09e26fa87db9bea037dfdc5b"
  }, 
  "json": null, 
  "origin": "111.14.9.241", 
  "url": "http://httpbin.org/post"
}
headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36'
}
response = requests.post("https://www.zhihu.com/explore",headers=headers)
print(response.text)

7.response属性

import requests

response = requests.get("http://www.jianshu.com")
print(type(response.status_code),response.status_code)
print(type(response.headers),response.headers)
print(type(response.cookies),response.cookies)
print(type(response.url),response.url)
print(type(response.history),response.history)

result:
    <class 'int'> 403
<class 'requests.structures.CaseInsensitiveDict'> {'Server': 'Tengine', 'Date': 'Thu, 31 Dec 2020 01:41:14 GMT', 'Content-Type': 'text/html', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Content-Encoding': 'gzip'}
<class 'requests.cookies.RequestsCookieJar'> <RequestsCookieJar[]>
<class 'str'> https://www.jianshu.com/
<class 'list'> [<Response [301]>]

8.状态码判断

import requests
response = requests.get("http://www.jianshu.com")
print(type(response.status_code),response.status_code)
print(requests.codes.ok)
exit()if not resopnse.status_code == requests.codes.ok else print("Request Successfully")

result:
<class 'int'> 403
200
Request Successfully

response = requests.get("http://www.jianshu.com")
exit()if not resopnse.status_code == 200 else print("Request Successfully")

result:
Request Successfully

9.文件上传

import requests
files = {'file':open('favicon.ico','rb')}
response = requests.post("http://httpbin.org/post",files=files)
print(response.text)

result:

10.获取cookie

import requests
response = requests.get("http://www.baidu.com")
print(response.cookies)
for key,value in response.cookies.items():
    print(key+"="+value)

result:
<RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>]>
BDORZ=27315

11.会话维持

#模拟登陆
import requests
requests.get("http://httpbin.org/cookies/set/number/123456789")
response = requests.get("http://httpbin.org/cookies")
print(response.text)

s = requests.Session()
s.get("http://httpbin.org/cookies/set/number/123456789")
response = s.get("http://httpbin.org/cookies")
print(response.text)

result:
{
  "cookies": {}
}
{
  "cookies": {
    "number": "123456789"
  }
}

12.证书验证

import requests
#访问https网站时要验证证书，如果证书有问题会返回一个ssl的错误，中断程序
response = requests.get("https://www.12306.cn")
print(resopnse.status_code)
from requests.packages import urllib3
urllib3.disable_warnings()
response = requests.get("https://www.12306.cn",verify=False)#会警告并且建议你加一个证书验证，调用urllib3的disable_warning
#resopnse = request.get("https://www.12306.cn",cert=('/path/...',"path/..."))自己设置用本地证书验证
print(resopnse.status_code)

result:
200
200

13.代理设置

import requests
proxies{
    "http":"xxxxx"
    "https":"xxxxx"
}
response = requests.get("https://www.12306.cn",proxies=proxies)

14.超时设置

import requests
response = requests.get("https://www.taobao.com",timeout=1)
print(response.status_code)

result:
200

import requests
from requests.exceptions import ReadTimeout
try:
    response = requests.get("http://httpbin.org/get",timeout=0.1)
    print(resopnse.status_code)
except ReadTimeout:
    print("Timeout")

15.认证设置

import requests
from requests.auth import HTTPBasicAuth

r = requests.get("http://xxxxx",auth=('username',"pwd"))
r = requests.get("http://xxxxx",auth=HTTPBasicAuth('username',"pwd"))
printn(r.status_code)

------------

import requests
from requests.exceptions import ReadTimeout,HTTPError,RequestException
try:
    response = requests.get("http://httpbin.org/get",timeout=0.1)
    print(resopnse.status_code)
except ReadTimeout:
    print("Timeout")
except HTTPError:
     print("HTTPError")
except RequestException:
     print("RequestException")

爬虫库—requests

一些requests的基本用法

1.无参get

2.带参get

3.解析json

4.获取二进制数据

5.添加headers

6.基本post请求

7.response属性

8.状态码判断

9.文件上传

10.获取cookie

11.会话维持

12.证书验证

13.代理设置

14.超时设置

15.认证设置

1.无参get

2.带参get

3.解析json

4.获取二进制数据

5.添加headers

6.基本post请求

7.response属性

8.状态码判断

9.文件上传

10.获取cookie

11.会话维持

12.证书验证

13.代理设置

14.超时设置

15.认证设置

JVM的一些常见问题

requests+PyQuery+pymongo联合实战

zhulei

Comments | NOTHING

取消回复

学习笔记