随心泛域名/泛目录注意事项

就以之前Test玩站往站点发的泛目录站群，还有今天发的泛域名程序【百度/搜狗泛域名泛目录程序(两个版】来说一下注意事项。

泛 = 广泛 如果我们没有限制，那就是无限制，拿程序来说，如果没有限制url这些，那就是无限的页面。那么等有访问量还是蜘蛛访问的时候会吃服务器很多的资源，比如带宽、cpu。

就以玩站小弟发的这款泛域名。2个域名，虽然说只有两个，但是做了泛域名后就是无限子域名、内页。然后再加上一些垃圾蜘蛛的访问，就更吃资源了！

那么三种方法来处理：
1：升级服务器配置
2：拦截部分垃圾蜘蛛ua
3：查看网站日志分析情况

做泛目录的时候，一些没有限制ua，就会被疯狂爬，比如bing、谷歌、甚至一些其他的国外蜘蛛。甚至有些蜘蛛我们不需要~所以我们要限制一下！

nginx：

 if ($http_user_agent ~* (BLEXBot|AliApp|o-http-cl|AhrefsBot|DataForSeoBot|Barkrowler|SemrushBot|semrush|DotBot|mj12bot|SM-G900P|xx-xx)) {
      return 503;
}

注意上面是没加限制谷歌，还有bing蜘蛛访问的，如果加了是这样

if ($http_user_agent ~* (ing|oogle|BLEXBot|AliApp|o-http-cl|AhrefsBot|DataForSeoBot|Barkrowler|SemrushBot|semrush|DotBot|mj12bot|SM-G900P|xx-xx)) {
return 503;
}

apache：

RewriteEngine On
#Block spider
RewriteCond %{HTTP_USER_AGENT} "ing|oogle|BLEXBot|AliApp|o-http-cl|AhrefsBot|DataForSeoBot|Barkrowler|SemrushBot|semrush|DotBot|mj12bot|SM-G900P|xx-xx" [NC]
RewriteRule !(^robots\.txt$) - [F]

iis：

  <rule name="noua" stopProcessing="true">
                    <match url=".*" ignoreCase="false" />
                    <conditions>
                        <add input="{HTTP_USER_AGENT}" pattern="(SemrushBot|o-http-cl|MegaIndex.ru|MauiBot|BLEXBot|acoonbot|ahrefsbot|alexa|toolbar|apachebench|applebot|asktbfxtv|chinasospider|compspybot|coolpadwebkit|crawldaddy|curl|digext|dotbot|easouspider|ec2linkfinder|edisterbot|elefent|exabot|ezooms|feeddemon|feedly|heritrix|httpclient|ichiro|indy|library|jaunty|java|jikespider|jorgee|lightdeckreports|bot|mail.ru|microsoft|url|control|mj12bot|msnbot-media|obot|perl|psbot|purebot|python|python-urllib|scrapy|seokicks-robot|siteexplorer|spbot|spiderman|swebot|swiftbot|teleport|teleportpro|turnitinbot|turnitinbot-agent|universalfeedparser|wangidspider|wbsearchbot|webdup|wget|wotbox|wsanalyzer|xbfmozilla|xenu|yandexbot|yottaa|yunguance|yyspider|zmeu|Yahoo! Slurp China|Yahoo! Slurp|msnbot|msnbot-media|ia_archiver|EasouSpider|JikeSpider|YandexBot|AhrefsBot|ezooms.bot)" />
                    </conditions>
                    <action type="CustomResponse" statusCode="400" statusReason="Forbidden" statusDescription="Forbidden" />
                </rule>

直接放到伪静态就可以了，这些ua就会被限制了，减少服务器资源了