Adblock Plus and (a little) more
Microsoft joining the ranks of referrer spammers? · 2008-12-19 08:52 by Wladimir Palant
I just noticed an unusually high amount of requests in my logs with referrer being Live Search, all with the same search request. Now that doesn’t make sense because I usually get very few people from there, about 0.5% of the requests referred by Google. And I just don’t believe that everybody suddenly decided to search for “adblockplus”. Finally, all requests came in for http://adblockplus.org/ and were redirected to http://adblockplus.org/en/ — but Live Search doesn’t even list the main page, it links directly the the English one (or German, or French, depending on your location).
I checked the logs and found lots of different IP addresses, for example 65.55.165.94. Can you guess which name it resolves into? Right, “msnbot-*.search.msn.com”. However, it doesn’t say “MsnBot” in the user agent string, instead it identifies itself as IE 6.0 on Vista (really? I thought you can only have IE 7.0 on Vista?). No robots.txt requests but a stylesheet requested every time. That way these hits are bound to be misinterpreted as valid by all log analysis tools. This cannot be a check whether the site displays different content to a bot than to a regular visitor — one request from one IP address would be sufficient then, and that IP address would certainly not resolve to “msnbot”. So, is Microsoft really that desperate? Do they have to resort to referrer spamming to increase the popularity of their search engine?
Comment [12]
Commenting is closed for this article.
Robert Wetzlmayr · 2008-12-19 10:07 · #
Yes, they are that desperate, or at least employing a few morons who work on a solid proof of Hanlon’s Razor.
MSNBOT has been hammering my sites with the exact pattern you describe since at least early this year. I’ve decided to block the originating IP addresses.
Reply from Wladimir Palant:
Interesting, I am pretty sure it never happened on my site before, at least not with more than 5 requests per day – it would show up in the daily reports my scripts generate. If it continues like this I also consider banning the IP range, but only for the case that the user agent header doesn’t say “msnbot”.
monk.e.boy · 2008-12-19 11:32 · #
You may be seeing the bot trying to see if you are modifying the content of your site depending on who is browsing it.
People change the content of their site for google bot, which they don’t like. It is common gaming SEO black hat tactic.
Interesting that it is getting the CSS. I think google does this because their bot uses Firefox to render most pages. The bot will only follow visible links — hiding SEO sculpting links from a user using hidden DIVs also hide the links from google bot.
Reply from Wladimir Palant:
As I said:
This cannot be a check whether the site displays different content to a bot than to a regular visitor — one request from one IP address would be sufficient then, and that IP address would certainly not resolve to “msnbot”.
Sander · 2008-12-19 21:15 · #
I’ve been getting these stupid requests with their fake searches for well over a year now, roughly ten a day, unceasingly even after 11 months of 403 responses to everything from the relevant IP ranges.
Microsoft’s official response on webmasterworld – http://www.webmasterworld.com/msn_microsoft_search/3424476-2-30.htm – has been:
“The traffic you are seeing is part of a quality check we run on selected pages”. (e.g. monk.e.boy’s anti-SEO-check)
As my 403-ing hasn’t resulted in me being dropped from the live search index (in fact, checking just now, I rank higher there than on google for pretty much everything), I’m just rolling my eyes at that response. The only real effect this has is inflating the perceived search-engine market share of live search.
Flore · 2008-12-22 14:25 · #
As this bot identifies itself as MSIE 6.0, wouldn’t Microsoft be trying to take back artificially marketshares from alternative browsers?
Analysis tools would count these hits into “IE6” and into “Vista” for their stats, so the marketshare of IE6 and Vista would be more important than they really are.
Reply from Wladimir Palant:
That’s not enough hits for that – forum spam bots do a “better” job here, they also identify themselves usually as Internet Explorer.
anonymous · 2008-12-22 19:13 · #
Offtopic, Zshare is now incompatible with ABP. It asks you to whitelist it before giving you the content. It seems the world is starting to block ABP. I can count several sites that ask me to whitelist them. What do we do as users? should we return to the days of ads again?
I hope you have a grassroots solution to this problem.
Thanks in advance.
Reply from Wladimir Palant:
Add “@@|http://www.zshare.net/adsense.js|” as a filter and everything will work. Generally, in the forum you will easily get help with questions like this one (no need to register, just create a topic).
anonymous · 2008-12-22 19:53 · #
Can you develop a way through which ABP can detect the scripts that check whether ads load or not and deal with them?
And I promise you, if I have a question next time I will post it in the forum. I really am sorry for interrupting the discussion here.
Reply from Wladimir Palant:
I doubt there is a general solution, the work-around will have to stay site-specific. However, those sites typically abandon anti-adblock scripts pretty soon (those scripts annoy site visitors, even the ones without Adblock Plus).
mrbene · 2008-12-24 03:13 · #
Interesting. I remember a short stretch of time where valid clicks from live.com didn’t transmit the HTTP-REFERER in most web browsers, and that lasted for about a week.
Now, if this is part of some QA (the webmasterworld discussion is light on the details) where the Trident engine is used outside of an actual browser instance, the page may actually be rendered. This is different than what most search spiders do (download HTML and extract data). This would explain the additional resource downloads. The HTTP-REFERER may be a mis-configuration rather than an intentional. Who knows.
Reply from Wladimir Palant:
Well, that referrer is clearly bogus since the page in question doesn’t have that link. What’s also strange is that the bots never go further than to the main page.
anonymous · 2008-12-24 06:13 · #
Is there an adblock plus for opera?
Reply from Wladimir Palant:
How about typing “adblock opera” into Google?
web tasarım · 2009-01-06 22:53 · #
Thank you for infos.
One very beautifull document
Best Regards mu frienz
Web Design · 2009-08-13 11:49 · #
Nice to know about it, but it never happened to my site. Still then thanks for your information.
Managed Services · 2009-09-30 14:54 · #
Great post.I really enjoy reading your blog and commenting on blogs.It really helped me a lot and will referring my friends about this blog.Thanks for sharing this post.Keep blogging.
marck_don · 2010-06-14 11:58 · #
Great information. thanks for sharing