Geek asks and answers

Robots.txt and non-referenced locations

If I want to protect the folder from being crawled by standards-compliant robots, I can disable it from the robots.txt file.

Now the problem is that by hiding the folder, I show its existence to others.

So, do I need to specify a folder that I do not want to scan in the robots.txt file if there are no links to it? Good search engines only follow links; they don’t randomly search folders and files.

Thank.

+3
source share
2 answers

Since the strings Disallowin robots.txtare prefixes, you can simply mention the prefix to your "hidden" directory, which it does not share with any "public" directories.

, "" /topsecrete_donotread/, Disallow: /tops, .

+2

, "" -, - , . http://example.com/hidden_folder , , , , . , . , , hidden_folder, . , , .

, . , , http://example.com/hidden_folder/bookmarks.html. .

(, joesblog.com), , joesblog.com, url - html .

, . joesblog , - , , "joesblog.com http://example.com/hidden_folder/bookmarks.html".

, . - , , - . , , .

Edit:

robots.txt, . , "" . "" .

, , - robots.txt, , , . , @Joachim, "" .

, , , http://example.com/hidden_folder/, , , , .

+2

More articles:


All Articles