linux防火墙免费版添加UA屏蔽某些垃圾蜘蛛
宝塔插件的防火墙UA添加屏蔽某些垃圾蜘蛛
安装宝塔linux防火墙。
软件管理 → linux防火墙免费版→ 全局配置 → User-Agent过滤 填入下面的规则。
随便写描述后点击添加即可。
垃圾蜘蛛规则列表
(www.seokicks.de|YYSpider|Mattermost|Discord|CCBot|RepoLookoutBot|tracking|serpstatbot|Pinterestbot|SurdotlyBot|DataForSeoBot|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|Ezooms|FlightDeckReports|Linguee Bot|Web-Crawler|WellKnownBot|Yellowbrandprotectionbot|ev-crawler|NE Crawler|Facebot|GrapeshotCrawler|SemrushBot|DotBot|MegaIndex.ru|MauiBot|AhrefsBot|MJ12bot|BLEXBot|HubSpot Crawler|CriteoBot|Web-Crawler|web-crawlers|DataForSeoBot|YaK|Mail.RU_Bot|Barkrowler|crawler|SEOkicks-Robot|vxiaotou-spider|telegram|dingtalk|Twitterbot|DuckDuckGo|applebot|webprosbot|AwarioBot|Amazonbot|AmazonAdBot|YouBot/)此方法简单高效。
PS：开启防火墙后给自己的IP设置白名单，避免采集或者修改的时候出现误报
拦截垃圾蜘蛛、爬虫、采集器。降低服务器消耗。加快速度。避免不必要的消耗，导致搜索引擎收录差等。