Comments on: check for robots.txt

Comments on: check for robots.txt http://usefulfor.com/security/2007/10/23/check-for-robotstxt/ security dojo Tue, 06 Jan 2009 13:42:24 +0000 http://wordpress.org/?v=2.5.1 By: Abhijeet Udas http://usefulfor.com/security/2007/10/23/check-for-robotstxt/#comment-54 Abhijeet Udas Wed, 27 Aug 2008 13:25:03 +0000 http://weblog.nomejortu.com/?p=12#comment-54 A possible easy way of doing it is for i in `cat File_with_ips` do echo 'trying $i' nc -nv $i 80 < robots.txt done contents of robots.txt GET /robots.txt HTTP/1.0 A possible easy way of doing it is

for i in `cat File_with_ips`
do
echo ‘trying $i’
nc -nv $i 80 < robots.txt
done

contents of robots.txt
GET /robots.txt HTTP/1.0

]]>