[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Full-disclosure] Google's robots.txt handling
- To: full-disclosure@xxxxxxxxxxxxxxxxx
- Subject: Re: [Full-disclosure] Google's robots.txt handling
- From: Denis McMahon <denismfmcmahon@xxxxxxxxx>
- Date: Mon, 10 Dec 2012 20:36:15 +0000
On 10/12/12 19:25, Hurgel Bumpf wrote:
> I tried to contact google, but as they didn't answer my email, I do forward
> this to FD.
> This shouldn't be a discussion about bad practice but the google feature
> itself.
I seem to recall that the robots.txt exclusion standard was fairly
common before I ever heard of google, I'm thinking of the mid to late
1990s when alta vista (which at the time was I think a technology
demonstrator for VAX Alpha RISC hardware?) was the goto of search engines.
Rgds
Denis McMahon
_______________________________________________
Full-Disclosure - We believe in it.
Charter: http://lists.grok.org.uk/full-disclosure-charter.html
Hosted and sponsored by Secunia - http://secunia.com/