Skip to content
Page 1 of results

Tips to Control Traffic Spikes Caused by Bots Crawling Your Web Site

In general, Microsoft has four bots that you may see crawling your site, Bingbot, Adidxbot, MSNbot, or BingPreview.  The User Agent name will identify which bot is causing the spike in traffic.  If you’re using Google Analytics, you should know that Google Analytics doesn't automatically filter out crawler traffic.

If the issue is coming from one of our bots, there are tools you can use to control bot traffic.  We’re providing two links below for resources that you can use. You will need to go to the Bing Webmaster Toolbox to control the rate at which the bot crawls your page (you will also be able to set what times of the day the bot crawls).  Doing so will regulate the traffic received from the bot and will prevent any latency issues from occurring again in the future. 

You can also edit your robots.txt file and use a crawl delay to define the number of seconds the crawler must wait in between successive requests. In general, whenever a bot crawls your page you will see very high bounce back rates and it is normal to see this behavior. You will need to filter out the particular bot traffic to get a better idea of how your site is actually performing.

Webmaster Crawler Information

Bing Webmaster Toolbox

As mentioned previously, Google Analytics does not filter out crawler traffic and we cannot control this behavior on our end.  That being said, there are ways to filter bot traffic from Google Analytics which can be found in links below.  If filtering out crawler traffic from Google Analytics is important to you, you may consider utilizing some of the options outlined in these blog posts.

How to Filter Bot Traffic From Your Google Analytics

Google Analytics: Creating Automatic "Bot Alert" Emails

Finally, the last two blog posts come directly from the Google Product Forums.  The last forum link includes a solution that can be used to prevent bot traffic from ever entering Google Analytics in the first place. (Please note: You will need to sign in to Bing Ads to access the tool)

Google Product Forum: How do I filter spiders most effectively from analytics?

Google Product Forum: Anyone having issues with tracking of MSN/CPC traffic? Tacking way to high?

Questions? Comments? Feel free to leave them below.