Just how to identify internet search engine crawlers accurately within asp.net internet site?

What is the most effective means to identify crawler created task (clicks/page brows through) etc on a asp.net internet site? We have an internet site where we track leads created to exterior internet sites, we track the IP of the customer that creates the lead nonetheless we are seeing a great deal of leads created by Google and also various other search crawlers. What is the most effective means to filter this task. I have actually come across screening customer representative strings and also filtering system based upon recognize IP addresses, both readily available as information unloads from numerous resources, I am not exactly sure which is the most effective one to make use of.

Thanks.

0
2019-05-06 22:41:54
Source Share
Answers: 2

Sorry yet I have no suggestion concerning asp.net internet sites, yet any kind of trustworthy internet search engine will certainly be sending you a customer representative string which informs you it's a crawler, which need to be readily available in your log documents. An additional free gift is that they seek/ robots.txt.

My mosaic resembles this (Perl) :

sub is_bot
{
    my ($user_agent) = @_;
    if ($user_agent =~
    /msnbot
        |www\.cuil\.com
    |Yahoo!\s+Slurp
    |Googlebot
    |Speedy\sSpider
    |MLBot
    |princeton crawler
    |accelobot
    |crawler\@dotnetdotcom
    |help\.naver\.com
    |GingerCrawler
    |Sosospider
    |www.exabot.com
    |Baiduspider
    |Ask\sJeeves
    |Java\/
    |telehouse\.ru
    |Tagoobot
    |Baypup
    |SimilarPages
    |Spinn3r
    |VoilaBot
    |Yandex
    |Xenu\sLink\sSleuth
    |www\.searchme\.com
    |MJ12bot
    |kilomonkey\.com
    |Mediapartners-Google
    |Sogou\sweb\sspider
    |YoudaoBot
    |seexie\.com
    |Yahoo.*Slurp
    |YahooCacheSystem
    |crawler\@nutch\.biz
    |psbot
    |ia_archiver-web\.archive\.org
    |sbider
    |xrss\.eu
    |scoutjet
        |www\.puritysearch\.net
        |Bing
        |BaiduImagespider
        |baidu\.jp
        |facebookexternalhit
        |ssllabs\.com
        |Python-urllib
        |drupal\.org
        |HTTrack
        |Willow\s+Internet\s+Crawler\s+by\s+Twotrees
    /x) {
    return 1;
    }
    return;
}

If you search in the resource code of awstats you'll locate something far better than the above.

0
2019-05-08 18:48:34
Source

Like @Kinopiko claimed, Bots, esepcialy the leading ones like google, or bing leave a clear UserAgent.

I do not recognize what your code resembles so I can not inform you what to do yet to locate the UserAgent in Asp.Net you consider Request.UserAgent in any kind of WebForm, CodeBehind, or MVC Controller.

0
2019-05-08 18:45:01
Source