webClient.getPage爬取csdn页面,卡死_综合

最近在写一下简单的爬虫,然后使用webClient.getPage的时候,一请求就直接挂掉,很长时间没有反应,让人很头疼,然后去掉js,返回的又是一些看不懂的:
获取列表页的文章列表:
https://blog.csdn.net/qq_38366063/article/list/1

正确的应该是:
在这里插入图片描述

这是获取列表页返回错误的:
在这里插入图片描述

<html><script> var arg1='0BC3DE07BEF4716BE7116BB1EE989B8E4EDBB0D5'; var _0x4818=['\x63\x73\x4b\x48\x77\x71\x4d\x49','\x5a\x73\x4b\x4a\x77\x72\x38\x56\x65\x41\x73\x79','\x55\x63\x4b\x69\x4e\x38\x4f\x2f\x77\x70\x6c\x77\x4d\x41\x3d\x3d','\x4a\x52\x38\x43\x54\x67\x3d\x3d','\x59\x73\x4f\x6e\x62\x53\x45\x51\x77\x37\x6f\x7a\x77\x71\x5a\x4b\x65\x73\x4b\x55\x77\x37\x6b\x77\x58\x38\x4f\x52\x49\x51\x3d\x3d','\x77\x37\x6f\x56\x53\x38\x4f\x53\x77\x6f\x50\x43\x6c\x33\x6... function reload(x) {
     setCookie("acw_sc__v2", x);document.location.reload();} </script></html>

然后就很烦,然后想了想,要是直接get请求返回流将流转字符串转dom对象是不是就可以了,然后试了一下,真可以,美滋滋,解决了爬虫最关键问题:

 public static InputStream doGet(String urlstr, Map<String, String> headers) throws IOException {
    URL url = new URL(urlstr);HttpURLConnection conn = (HttpURLConnection) url.openConnection();conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 " +"(KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36");conn.setRequestProperty("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp," +"image/apng,*/*;q=0" +".8");if (headers != null) {
    Set<String> keys = headers.keySet();for (String key : keys) {
    conn.setRequestProperty(key, headers.get(key));}}Random random = new Random();String ip =(random.nextInt(100) + 100) + "." + (random.nextInt(100) + 100) + "." + (random.nextInt(100) + 100) + "." + (random.nextInt(100) + 100);conn.setRequestProperty("x-forwarded-for", ip);InputStream inputStream = conn.getInputStream();return inputStream;}

返回的InputStream直接转String,然后直接Jsoup解析即可:

   public static String inputStreamToString(InputStream is, String charset) throws IOException {
    byte[] bytes = new byte[1024];int byteLength = 0;StringBuffer sb = new StringBuffer();while ((byteLength = is.read(bytes)) != -1) {
    sb.append(new String(bytes, 0, byteLength, charset));}return sb.toString();}

在这里插入图片描述
还是有问题,有的页面可以直接流的方式直接写,有的不行,不知道什么原因=-=.
博客的详情页面就不行…

csdn有验证token,打开控制台,找到对应token,加上去就好了,token一般更新时间为半个小时…否则会被屏蔽…
,csdn爬取,加上token即可,自己通过webClient.getPage,就可以获取.,或者直接通过httpRequest,直接get请求也是可以的…记得传token即可.