Administrator
Posts: 178 
April 27th, 2003, 12:34 PM
We decided to include a forum for questions, complaints, comments about our spider, RoboCrawl.
You can find additional details about our spider at http://www.robotstxt.org/wc/active/html/robocrawl.html .
You may submit your URL at the following address: http://www.canadiancontent.net/corp/suggest.html .
The spider follows the robots.txt standard and the Meta tags specifying additional instructions.
We decided to include a forum for questions, complaints, comments about our spider, RoboCrawl.
You can find additional details about our spider at http://www.robotstxt.org/wc/active/html/robocrawl.html .
You may submit your URL at the following address: http://www.canadiancontent.net/corp/suggest.html .
The spider follows the robots.txt standard and the Meta tags specifying additional instructions.
Genius
Posts: 783 
May 2nd, 2003, 06:33 AM
My apologies. I grepped my server logs and you did request my robots.txt a week ago. I suppose you're caching it... Although I don't think it's unreasonable to request robots.txt a little more often than that. In fact, there has been a change in our robots.txt in the last week.
The reason I came and started to check out your site is that I found you in my raw logs today and there were enough hits to rise above the background noise... I run reporting software at the end of the week and somehow I missed that access or two on April 20th, but here I am now.
As it happens I am Canadian and operate a Canadian-based ecommerce site, so I thought you had found me somewhere in a Canadian business directory, but there's others your hitting too that aren't specifically about Canada at all, see this thread http://www.webmasterworld.com/forum11/1862.htm for a very brief discussion of your spider.
Because I'm a Canadian ecommerce site operator, I'm interested in your site and your spidering policies. I'll go read some more on your site and check your privacy policies. Is it appropriate to contact you directly via email?
Thanks...
My apologies. I grepped my server logs and you did request my robots.txt a week ago. I suppose you're caching it... Although I don't think it's unreasonable to request robots.txt a little more often than that. In fact, there has been a change in our robots.txt in the last week.
The reason I came and started to check out your site is that I found you in my raw logs today and there were enough hits to rise above the background noise... I run reporting software at the end of the week and somehow I missed that access or two on April 20th, but here I am now.
As it happens I am Canadian and operate a Canadian-based ecommerce site, so I thought you had found me somewhere in a Canadian business directory, but there's others your hitting too that aren't specifically about Canada at all, see this thread http://www.webmasterworld.com/forum11/1862.htm for a very brief discussion of your spider.
Because I'm a Canadian ecommerce site operator, I'm interested in your site and your spidering policies. I'll go read some more on your site and check your privacy policies. Is it appropriate to contact you directly via email?
Thanks...
Administrator
Posts: 178
May 2nd, 2003, 09:37 AM
We don't only spider Canadian sites, although we give preference to specific Canadian sites and follow links from there.
Definately send us an email! We'd appreciate more feedback as our spider was an older project which was recently restarted, in beta (actually more of an alpha) phase.
You may email spider .at. canadiancontent.net if you have any other questions, comments or suggestions.
We don't only spider Canadian sites, although we give preference to specific Canadian sites and follow links from there.
Definately send us an email! We'd appreciate more feedback as our spider was an older project which was recently restarted, in beta (actually more of an alpha) phase.
You may email spider .at. canadiancontent.net if you have any other questions, comments or suggestions.
