×
The internal search function is temporarily non-functional. The current search engine is no longer viable and we are researching alternatives.
As a stop gap measure, we are using Google's custom search engine service.
If you know of an easy to use, open source, search engine ... please contact support@midrange.com.
I'm trying to setup my robots.txt so that certain directories are blocked from being indexed by search engines, but NOT by the adsense crawler.
What I want is to have adsense be able to crawl directories that start '/jtopen_' but NOT have search engines in general do so.
Here's my robots.txt ...
User-agent: *
Disallow: /jtopen_
User-agent: Mediapartners-Google*
Disallow:
Googles robots.txt checker says that both the Googlebot and adsense crawler will be blocked from /jtopen_ directories but, according to my research, it shouldn't.
Is this a problem with the checker or am I misunderstanding something about robots.txt?
Thanks!
david
As an Amazon Associate we earn from qualifying purchases.
This mailing list archive is Copyright 1997-2024 by midrange.com and David Gibbs as a compilation work. Use of the archive is restricted to research of a business or technical nature. Any other uses are prohibited. Full details are available on our policy page. If you have questions about this, please contact
[javascript protected email address].
Operating expenses for this site are earned using the Amazon Associate program and Google Adsense.
midrange.com
