Problem with robots.txt Disallow
-
Hi I have a problem with the robots.txt and google. I have this robots.txt file: User-agent: * Disallow: page1.html Disallow: dir_1/sub_dir_1/ Disallow: /data/ When I enter 'site:www.MySite.com' into Google search box, Goolge gets the content from the 'data' directory as well. Google should not have indexed the content of data directory. So why is google getting the results from 'data' directory, whereas I have disallowed it. How can I restrict everyone from accessing the data directory? Thanks
-
Hi I have a problem with the robots.txt and google. I have this robots.txt file: User-agent: * Disallow: page1.html Disallow: dir_1/sub_dir_1/ Disallow: /data/ When I enter 'site:www.MySite.com' into Google search box, Goolge gets the content from the 'data' directory as well. Google should not have indexed the content of data directory. So why is google getting the results from 'data' directory, whereas I have disallowed it. How can I restrict everyone from accessing the data directory? Thanks
You may need to remove the first /, as I think this takes it back to the root directory.
If you're stuck in a rut: 1) Consult the documentation* 2) Google it 3) Ask a sensible question 4) Try an ancient ritualistic knowledge summoning (:badger::badger::badger:) dance :jig: 5) Phone :bob: * - If the documentation is MSDN > 6.0 then forget it!
-
Hi I have a problem with the robots.txt and google. I have this robots.txt file: User-agent: * Disallow: page1.html Disallow: dir_1/sub_dir_1/ Disallow: /data/ When I enter 'site:www.MySite.com' into Google search box, Goolge gets the content from the 'data' directory as well. Google should not have indexed the content of data directory. So why is google getting the results from 'data' directory, whereas I have disallowed it. How can I restrict everyone from accessing the data directory? Thanks
Where did you put your robots.txt? It needs to be in root level. (ie. you should be able to read it from www.MySite.com/robots.txt) Also, the search result may be from google cache before you put the robots.txt up.