usefulfor.com/security » Shell Script

check for robots.txt

etd — Tue, 23 Oct 2007 15:54:04 +0000

Some times it is useful to check if a given HTTP server has a robots.txt file in it. If it exist it may disclose interesting information, useful for a pentest

From the Wikipedia:

The robots exclusion standard or robots.txt protocol is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website. The information specifying the parts that should not be accessed is specified in a file called robots.txt in the top-level directory of the website.

Here is a script that checks for the presence of the file in a list of hosts (you can download the source code). Two main parts can be distinguished: command line parsing and file download.

You can call the script in two different ways. Either you do not specify the protocol (and HTTP will be used):-

./robots.sh   ...

Or you specify the protocol with:

./robots.sh  -p [http|https]   ...

Let's see how this is done:

PROTO=( http https )
HTTP=${PROTO[0]}
FILE=/tmp/robots.txt

# command line parsing
if [ "-p" == $1 ]
then
  for bar in ${PROTO[*]}
  do
    if [ $bar == $2 ];
    then
      HTTP=$2
      HOSTS=${*:3}
    fi
  done
else
  HOSTS=$*
fi

We check if the first argument is "-p" in which case, the next argument should be one of the allowed values (those in $PROTO array). If that is the case, we strip the first two parameters and put everything else in the $HOSTS variable. At the end of the code above, $HTTP will contain either http or https and $HOSTS will consist of a list of hosts whose robots.txt file existance we want to verify.

Once we know what protocol are we using and the list of targets, the only thing left is to try to download the robots.txt file of each server:-

for foo in $HOSTS; do
  echo "================"
  echo "Server: $foo ($HTTP)"
  CODE=`wget -O $FILE $HTTP://$foo/robots.txt 2>&1 | grep HTTP | head -1 | awk '{print $6}'`
  echo "Code: $CODE"
  if [ "200" == $CODE ]
  then
    echo "Contents:"
    echo "----------------"
    cat $FILE
    rm $FILE
    echo "----------------"
  fi
done

If the response code is 200 OK we cat the file to standard output. Otherwise we just move on to the next target of the list. The only tricky bit of the previous code is:

wget -O $FILE $HTTP://$foo/robots.txt 2>&1 | grep HTTP | head -1 | awk '{print $6}'

Where we try to download the file saving it to the location specified by $FILE. In order to get the HTTP error code we redirect standard error to standard output using 2>&1.

One last word, it is acknowledged that the script does not follow HTTP redirects, but if the server replies with a redirect this means that effectively, no robots.txt file is present.

ninja iptables for your server

etd — Fri, 14 Sep 2007 11:15:37 +0000

Security is often about layers on top of layers on top of layers... And one of these layers is usually an iptables firewall installed in your server. Let's create a small script to provide our server with the kung-fu fighting techniques needed to defeat the black hats!!

You can download the script from here. But let's have it on the screen so we can walk through the rules:-

#!/bin/bash

###
### IPTables config file
### Based on the rules compiled by Ranjit San aka 'the grasshopper'
### Created 2007-09-14 by Daniel Martin Gomez
### Revision 1
###

###
### define variables
###

### path to iptables
IPT=/sbin/iptables

### This contains a list of approved Debian sites to get software updates.
DEBIAN_SITES=('194.109.137.218' '212.219.56.139' '212.219.56.133' '212.219.56.134' '212.219.56.135' '212.219.56.138')

### This contains the authorised DNS servers configured in /etc/resolv.conf.
DNS_SERVERS=('')

### This is a list of external IPs that you want to allow ssh access from.
OTHER_GATEWAYS=('')

### This is a list of hosts authorised to try ICMP probes to check if the
### server is running. This could be your ISP's IPs
CONTROL_GATEWAYS=('')

### Types of ICMP probes to allow from the previous servers
ICMP_TYPES=('echo-reply' 'destination-unreachable' 'echo-request' 'ttl-exceeded')

#### NTP servers for time synch
NTP_SERVERS=('')

### ------------------------------------------------- do not change below this line

###
### INPUT
###

### will flush the chains or all rules one by one. Therefore all new rules will be created.
$IPT -F

### allows inbound packets to be processed
$IPT -P INPUT ACCEPT

### drops packets so that they can not come through one interface and flow out of another.
$IPT -P FORWARD DROP

### This allows outbound packets to be processed
$IPT -P OUTPUT ACCEPT

### allows ICMP types (defined above) for hosts in the control list
for IP in ${CONTROL_GATEWAYS[@]}; do
for ICMP in ${ICMP_TYPES[@]}; do
$IPT -A INPUT -s $IP -p icmp --icmp-type $ICMP -j ACCEPT
done
done

### this accepts connections for http and https access from anywhere
$IPT -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
$IPT -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT

### this allows remote administration using ssh from your other gateways.
for IP in ${OTHER_GATEWAYS[@]}; do
$IPT -A INPUT -s $IP -p tcp -m tcp --dport 22 -j ACCEPT
done

### this allows packets to start a new connection or allows packets that are
### already associated with a connection, required for stateful inspection.
$IPT -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

### this allows NTP traffic from NTP server
for NTP in ${NTP_SERVERS[@]}; do
$IPT -A INPUT -s $NTP -p udp -m udp --sport 123 -j ACCEPT
done

### we are about to drop everything else, so first log the discarded traffic
### just in case we want to know what *they* are trying.
$IPT -A INPUT -j LOG

### this drops any traffic that does not match to the INPUT rules
$IPT -A INPUT -j DROP

###
### OUTPUT
###

### Allows traffic to authorised DNS servers
for IP in ${DNS_SERVERS[@]}; do
$IPT -A OUTPUT -d $IP -p udp -m udp --dport 53 -j ACCEPT
done

### Allows http traffic to debain sites for software updates.
### Initial config rule
for IP in ${DEBIAN_SITES[@]}; do
$IPT -A OUTPUT -d $IP -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
done

### this allows packets to start a new connection or allows packets that are
### already associated with a connection, required for stateful inspection.
$IPT -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

### this allows NTP traffic to the NTP servers
for NTP in ${NTP_SERVERS[@]}; do
$IPT -A OUTPUT -d $NTP -p udp -m udp --dport 123 -j ACCEPT
done

### this logs all OUTPUT traffic that does not match the rules before it beign
### dropped.
$IPT -A OUTPUT -j LOG

### this drops any traffic that does not match to the OUTPUT rules
$IPT -A OUTPUT -j DROP

Just two things to add: First, do not forget to set your own values for the variables DNS_SERVERS, OTHER_GATEWAYS, CONTROL_GATEWAYS and NTP_SERVERS. And second, if you want your kung-fu up and ready after boot you may need to issue the following:-

cd /etc/init.d/
wget http://usefulfor.com/security/files/2008/06/firewall.sh
chmod +x firewall.sh
update-rc.d firewall.sh defaults

If you ever want to remove it from the boot sequence just issue:-

update-rc.d -f firewall.sh remove

runningserver: hello? anybody out there?

etd — Wed, 20 Dec 2006 12:45:52 +0000

I have created a small ruby script to check if there are running servers on a given port number. The script is able to check a list of hosts and will output an informational message on the port status for each host.

Let's begin with the script. Then we can talk about the work it does.

Here is the code:

#!/usr/bin/ruby
##################################################################################################################
#
# runningserver.rb
# 12/DEC/2006
# etd [etd__at__nomejortu.com]
#
# Desc:
#   Script to create a connection on the specified port to check if a server is
#   listening on it.
#
# Version:
#   v1.0 [12/Dec/2006]: first released
#
###################################################################################################################
$help =< e
serverstr = "No server was found in port #{$port}"
end
puts "#{host}: #{serverstr}"
end
puts

Because this is the first ruby script in this blog I will explain it step by step. The first thing that comes in the script is the input argument parsing. Input arguments are passed as a string array, in the variable ARGV. To check if the user is requesting help or usage information we can issue:

if (
ARGV.include?('--help')  ||
ARGV.include?('-help') ||
ARGV.include?('-h') ||
(ARGV.size==0)
) then
puts
puts $help
exit
end

In the same way the script checks if the user has supplied a port number. If this is not the case, we use a default port. If the input argument -p is present, we asume the next argument (p_position+1) to be the TCP port number. It is important to get the port number as an interger. We accomplish this by issuing the .to_i call.

# clear these values in the array
ARGV[p_position] = nil
ARGV[p_position+1] = nil

With the code above we clear the elements we are not using any more. Doing so we can compact the ARGV array in order to remove all the nil elements of it just by calling the .compact function of the array.

The body of the script is pretty straight forward. For each host given as input to the script we try to open a socket on the specified port. If successful, we close the socket and that's it. To avoid hanging for ever on closed ports we are using timeout library (require 'timeout'). The timeout mechanism is very simple. You create a block and give the timeout in seconds (can be a float point value). If the block terminates before the timeout, timeout returns the value of the block. Otherwise, the exception (Timeout::Error) is raised.

begin
Timeout::timeout(3) do
#code
end
rescue Exception => e
#exception handling
end

You see? It wasn't that difficult, was it?

icmp timestamps

etd — Thu, 14 Dec 2006 18:53:12 +0000

The Timestamp is an ICMP (rfc792) message which is used for time synchronization. The Timestamp Reply message consists of the originating timestamp sent by the sender of the Timestamp as well as a receive timestamp and a transmit timestamp.

If your machine answers ICMP Timestamp messages an attacker can learn the date which is set on your machine. This may help him to defeat all your time based authentication protocols.

Here is the code of a script that can be used to check if a remote host listens Timestamp requests:

# Check if the script is being run as root exit if it is not.
if [ "$UID" -ne "0" ]
then
  echo "[ERROR] This script must be run as root"
  exit 1
fi

for foo in $*; do
  echo -n "$foo "
  output=`hping3 -c 3 --icmp-ts $foo 2>/dev/null | grep "ICMP timestamp" | wc -l`
  if (( output > 0  ))
  then
    echo "reacts to ICMP timestamp."
  else
    echo "doesn't react."
  fi
done

First we need to check that root is the one running the script because otherwise we won't be able to craft ICMP packages. For this task we will be using hping (i.e: hping3 package in Debian GNU/Linux).

The script just sends three (-c 3) ICMP Timestamps (--icmp-ts) to each of the hosts feeded in the command line. We grep the output of hping3 looking for the magic string "ICMP timestamp", and if found, we print a success message.