Search Engines
-
Hi, I have a demo site up and would like to be certain that if I have the following in my page that it will definately show not show up in a search engne: What else do I need to add to be 110% sure? Regards
-
Hi, I have a demo site up and would like to be certain that if I have the following in my page that it will definately show not show up in a search engne: What else do I need to add to be 110% sure? Regards
I think you can specify the blocked page in robots.txt..
Thanks and Regards, Michael Sync ( Blog: http://michaelsync.net)
-
Hi, I have a demo site up and would like to be certain that if I have the following in my page that it will definately show not show up in a search engne: What else do I need to add to be 110% sure? Regards
Create a robots.txt file in your root directory then just write these line in it: User-agent: * Disallow: / This will prevent search engines like Google, Yahoo, Msn and all those who respect Robots.txt Protocol crawling your web site. Hope it helps !
AliAmjad(MCP) First make it Run THEN make it Run Fast!
-
Create a robots.txt file in your root directory then just write these line in it: User-agent: * Disallow: / This will prevent search engines like Google, Yahoo, Msn and all those who respect Robots.txt Protocol crawling your web site. Hope it helps !
AliAmjad(MCP) First make it Run THEN make it Run Fast!
So what about search engines that do not respect robots. Would it help him to put the website in a password protected area? Or is that too much?
-
So what about search engines that do not respect robots. Would it help him to put the website in a password protected area? Or is that too much?
It's just a protocol and it entirely depends on the Bot whether it supports it or not and those who doesn't support it called Bad Bots. But to really protect your files from Bad bots you can setup Spider traps etc.
AliAmjad(MCP) First make it Run THEN make it Run Fast!