Blocking crawlers with robots.txt, but not adsense? -- PCTECH

Good News Everybody!

The new search engine is LIVE!

Please report any problems to david (at) midrange.com.

Subject: Blocking crawlers with robots.txt, but not adsense?
From: David Gibbs <david@xxxxxxxxxxxx>
Date: Wed, 04 Mar 2009 13:32:58 -0600
List-archive: <http://archive.midrange.com/pctech>
List-help: <mailto:pctech-request@midrange.com?subject=help>
List-id: PC Technical Discussion for iSeries Users <pctech.midrange.com>
List-post: <mailto:pctech@midrange.com>
List-subscribe: <http://lists.midrange.com/mailman/listinfo/pctech>, <mailto:pctech-request@midrange.com?subject=subscribe>
List-unsubscribe: <http://lists.midrange.com/mailman/listinfo/pctech>, <mailto:pctech-request@midrange.com?subject=unsubscribe>

I'm trying to setup my robots.txt so that certain directories are blocked from being indexed by search engines, but NOT by the adsense crawler.

What I want is to have adsense be able to crawl directories that start '/jtopen_' but NOT have search engines in general do so.

Here's my robots.txt ...

User-agent: *
Disallow: /jtopen_

User-agent: Mediapartners-Google*
Disallow:

Googles robots.txt checker says that both the Googlebot and adsense crawler will be blocked from /jtopen_ directories but, according to my research, it shouldn't.

Is this a problem with the checker or am I misunderstanding something about robots.txt?

Thanks!

david

As an Amazon Associate we earn from qualifying purchases.

This mailing list archive is Copyright 1997-2026 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact [javascript protected email address].

Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.