Coming Soon!

[Administration]

We decided to include a forum for questions, complaints, comments about our spider, RoboCrawl.

You can find additional details about our spider at http://www.robotstxt.org/wc/active/html/robocrawl.html .

You may submit your URL at the following address: http://www.canadiancontent.net/corp/suggest.html .

The spider follows the robots.txt standard and the Meta tags specifying additional instructions.

 

[Anonymous]

Robots.txt?!

How can you follow the robots.txt exclusion protocol when you don't request it from my site?

 

[Administration]

We always request a robots.txt in the root directory or a site. Has is even been spidered yet?

It may take weeks to get into the index which is still in beta (very much so). What is your site url and I'll post a bug.

 

[Anonymous]

My apologies. I grepped my server logs and you did request my robots.txt a week ago. I suppose you're caching it... Although I don't think it's unreasonable to request robots.txt a little more often than that. In fact, there has been a change in our robots.txt in the last week.

The reason I came and started to check out your site is that I found you in my raw logs today and there were enough hits to rise above the background noise... I run reporting software at the end of the week and somehow I missed that access or two on April 20th, but here I am now.

As it happens I am Canadian and operate a Canadian-based ecommerce site, so I thought you had found me somewhere in a Canadian business directory, but there's others your hitting too that aren't specifically about Canada at all, see this thread http://www.webmasterworld.com/forum11/1862.htm for a very brief discussion of your spider.

Because I'm a Canadian ecommerce site operator, I'm interested in your site and your spidering policies. I'll go read some more on your site and check your privacy policies. Is it appropriate to contact you directly via email?

Thanks...

 

[Administration]

We don't only spider Canadian sites, although we give preference to specific Canadian sites and follow links from there.

Definately send us an email! We'd appreciate more feedback as our spider was an older project which was recently restarted, in beta (actually more of an alpha) phase.

You may email spider .at. canadiancontent.net if you have any other questions, comments or suggestions.