Ask The Experts About Robots.txt
Richard asks…
Do I have to make different robots.txt and sitemap.xml on every subdomain of my website?
I have a website and 5 subdomains, and only my main homapage have sitemap.xml and robots.txt. I want all of my subdomains to be indexed by search engine. do i have to make robots.txt and sitemap.xml on every subdomain?
admin answers:
Probably so.
You don’t want too many subdomains, because google will consider it spam, and penalize you by not showing you on its front pages.
Powered by Yahoo! Answers
Ask The Experts About Robots.txt
Lizzie asks…
Is it possible to view an archive of a site that has robots.txt?
I am looking for an archive of a post that someone had made on a site, but they have since left the site, and thus their post is gone as well. The site does not allow robots, so google and the likes do not have cached pages for it.
Any help?
admin answers:
If you know the exact post url, you could see it as long as it wasn’t deleted, or set to private.
Powered by Yahoo! Answers
