Re: [Full-disclosure] Google's robots.txt handling

["Lehman, Jim" <jim.lehman@xxxxxxxxxxxxxxxxxxx>]

It is possible to use white listing for robots.txt. Allow what you want google 
to index and deny everything else. That way google doesn't make you a goole 
dork target and someone browsing to your robots.txt file doesn't glean any 
sensitive files or folders. But this will not stop directory bruting to 
discover your publicly exposed sensitive data, that probably should not be 
exposed to the web in the first place. 

I would rather have some one pound on my server to find something, I might have 
more time to respond, rather than having mr. bad googleing for the weakness in 
the web site and only making one request to get what they are after.

http://www.sans.org/reading_room/whitepapers/awareness/robotstxt_33955
Its not a great  paper, but it might have some value for those that have not 
looked into how this file works. 


-----Original Message-----
From: full-disclosure-bounces@xxxxxxxxxxxxxxxxx 
[mailto:full-disclosure-bounces@xxxxxxxxxxxxxxxxx] On Behalf Of Hurgel Bumpf
Sent: Monday, December 10, 2012 11:26 AM
To: full-disclosure@xxxxxxxxxxxxxxxxx
Subject: [Full-disclosure] Google's robots.txt handling

Hi list,


i tried to contact google, but as they didn't answer my email,  i do forward 
this to FD.
This "security" feature is not cleary a google vulnerability, but exposes 
websites informations that are not really intended to be public.

(Additionally i have to say that i advocate robots.txt files without sensitive 
content and working security mechanisms.)

Here is an example: 

An admin has a public webservice running with folders containing sensitive 
informations. Enter these folders in his robots.txt and "protect" them from the 
indexing process of spiders. As he doesn't want the /admin/ gui to appear in 
the search results he also puts his /admin in the robots text and finaly makes 
a backup to the folder /backup.

Nevertheless these folders arent browsable but they might contain f(a)iles with 
easy to guess namestructures, non-encrypted authentications (simple AUTH) , you 
name it...

Without a robots.txt nobody would know about the existance of these folders, 
but as some folders might be linked somewhere, these folders might appear in 
search results when not defined in the robots.txt  The admin finds himself in a 
catch-22 situation where he seems to prefer the robots.txt file.

Long story short.

Although google accepts and respects the directives of the robots.txt file, 
google INDEXES these files. 

This my concern. 

http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+Disallow%3A+%2Fadmin
http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+Disallow%3A+%2Fbackup
http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+Disallow%3A+%2Fpassword

As these searches can be used less for targeted attacks, they more can be used 
to find victims. 

http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+%2FDisallow%3A+wp-admin
http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+%2FDisallow%3A+typo3
<Just be creative>

This shouldn't be a discussion about bad practice but the google feature 
itself. 

Indexing a file which is used to prevent indexing.. isn't that just paradox and 
hypocrite?

Thanks,


Conan the bavarian

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/


*******************************************************
This message (including any files transmitted with it) may contain confidential 
and/or proprietary information, is the property of Interactive Data Corporation 
and/or its subsidiaries, and is directed only to the addressee(s). If you are 
not the designated recipient or have reason to believe you received this 
message in error, please delete this message from your system and notify the 
sender immediately. An unintended recipient's disclosure, copying, 
distribution, or use of this message or any attachments is prohibited and may 
be unlawful. 
*******************************************************

_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/