python 爬虫抓取百度热搜
实现思路：
第1步、在百度热搜页获取热搜元素
元素类名为category-wrap_iQLoo 即我们只需要获取类名category-wrap_为前缀的元素
第2步、编写python脚本实现爬虫
import requests
from bs4 import BeautifulSoup
url = 'https://top.baidu.com/board?tab=realtime'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36'
}
response = requests.get(url, headers=headers)
response.encoding = response.apparent_encoding
soup = BeautifulSoup(response.text, 'html.parser')
hot_searches = []
# 使用CSS选择器匹配类名前缀为'category-wrap_'的元素
category_wrap_prefix_elements = soup.select('[class^="category-wrap_"]')
# 遍历并打印这些元素
for element in category_wrap_prefix_elements:
title = element.find('div', class_='c-single-text-ellipsis').get_text().strip()
link = element.find('a')['href']
print(title, link)
hot_searches.append({title, link})
print(hot_searches)控制台打印
