Dangers of robots.txt
-
I just realised that I can type http://www.bbc.co.uk/robots.txt and see all the directories that BBC does not want me to access. robots.txt seems like a very dangerous file as it gives easy information to the potential mis-users. Is there any way around it? Thanks
-
I just realised that I can type http://www.bbc.co.uk/robots.txt and see all the directories that BBC does not want me to access. robots.txt seems like a very dangerous file as it gives easy information to the potential mis-users. Is there any way around it? Thanks
hasanali00 wrote:
Is there any way around it?
This is more of a web dev question. but you can write a meta for every page if it is in HTML. I forgt what it is but I'm sure you can find it. -- modified at 8:22 Wednesday 25th October, 2006 [EDIT] Just remembered Meta: See if that works [/EDIT]
Brad Australian Have you tried Rebooting it?
-
I just realised that I can type http://www.bbc.co.uk/robots.txt and see all the directories that BBC does not want me to access. robots.txt seems like a very dangerous file as it gives easy information to the potential mis-users. Is there any way around it? Thanks
http://www.robotstxt.org/wc/faq.html#nosecurity[^]
-- Help me! I'm turning into a grapefruit! Buzzwords!
-
I just realised that I can type http://www.bbc.co.uk/robots.txt and see all the directories that BBC does not want me to access. robots.txt seems like a very dangerous file as it gives easy information to the potential mis-users. Is there any way around it? Thanks
The most amussing one is the Whitehouse's list... Well it was last time I checked a good few months ago...
-
I just realised that I can type http://www.bbc.co.uk/robots.txt and see all the directories that BBC does not want me to access. robots.txt seems like a very dangerous file as it gives easy information to the potential mis-users. Is there any way around it? Thanks
I didn't think it was meant for that. I thought it was meant for public files you don't want a search engine to index. AFAIK a search engine won't start guessing URLs so there is no need to list "hidden" files in your robot.txt when those hidden files are not linked to from an indexable file. i.e. Don't list your hidden files.
regards, Paul Watson Ireland FeedHenry needs you
eh, stop bugging me about it, give it a couple of days, see what happens.