Kus' Blog

At Atlas we love documentation, as all of you must too! Every system and service is documented into a wiki.

The first time you are told to document a system, you log into that system and start typing commands to get what you need and copy it to a wiki, then apply the appropiate format.

The second time you apply some pipe magic with the awk magic dust, and voila! the info you need as formatted as possible

Some useful commands you will appreciate:

• Getting partition table formatted, with size (change ext3 to get other fs types):
  df -P -h -t ext3 | grep -v Filesystem | awk '{print "| "$1" | "$6" | "$2" |"}'
• Getting NFS partition mappings formatted:
  
  cat /etc/fstab | grep nfs | grep -v '#' | awk '{print "| "$1" | "$2" |"}'
• Getting apache virtual host configuration:
  
  cat your-vhost.conf | egrep "ServerName|ServerAlias|RewriteRule|DocumentRoot|ErrorLog|CustomLog|ErrorDocument|VirtualHost|SSL" | awk '{print "| | "$1" | "$2$3$4$5$6" |"}'
  This one needs to be polished a little to get rid of those virtual hosts lines and put its info on the first column.

Put this post to your favourites as it will be updated as more info is needed to format.

New job, new problems, new hacks! The next hack tries to take all the juice from linux’s pipes.

The task which I had to do was a database migration, from one mysql server to another one. The first problem were different versions, so hot copy wasn’t an option (actually few times is an option, unless it’s not in production ). The next big issue, were the size of the data, its big, really big (from my point of view) some DB reach 800M, for me is big enough to be a big problem

Mysqldump + mysql but this is good for 2 DB? I wanted to be selective in which DB to migrate, plus data should travel encrypted and transfer should delay as minimum as possible.

The script I got is this one (showing main loop, what’s interesting):

for db in `cat sm2-hf-dbs-kk.txt`
do
echo "Creant database a ${remote_sql}..."
ssh $remote_sql "echo \"create database if not exists ${db}\" | mysql -u ${db_user_rw} -p${db_pass_rw}"
echo "Fet"
echo "Donant permisos sobre la BD a rw..."
ssh $remote_sql "echo \"grant select, insert, update, lock tables, create, drop, alter on *.* to '${db_user_rw}'@'localhost';flush privileges\" | mysql -u ${remote_admin} -p${remote_admin_pass}"
echo "Fet"
echo "Fent dump gzipat i enviant a ${remote_sql}"
mysqldump --single-transaction -u ${db_user_r} -p${db_pass_r} ${db} | gzip | ssh $remote_sql "gunzip - | mysql -u ${db_user_rw} -p${db_pass_rw} ${db}"
echo "Fet"
done

The new server was empty, so we first ssh to it and from there we create a database. The next step was to give perms over the new db… I see now it can be done in one single line.

After this comes the magic. The script makes a dump of the db, it passes through gzip, this way we dramatically improve speed over the net. This pipe stream is passed through ssh to the other server unzip the info and put into the new mysql server. Pipe Magic rules!

I prefer  ssh the machine rather than setting a tunnel because the tunnel should set before the script, this way we don’t care. The other point is that the ssh should have a valid pair of keys to access it.

That’s all folks! Hope this enlightens someone…

While researching why the servers where overloaded I saw that some webpages where attacked/scanned from hundreds of different ips. Most of the where from Caravan Networks from Russia…

I know almost for sure that no webs pages in our servers have russian clients target so I wanted to ban them all, from their IP range. The problem was this range wasn’t trivial to get, so I did this little hack to get them.

Saddly it isn’t 100% effective but nearly 80%

First thing to get is a list of the attackers ip. In this case we got them from grepping through an apache access log.

cat /usr/local/apache/domlogs/kedume9/somedomain.com | grep "POST /web/ht" | cut -f 1 -d " " | sort -u malos.txt

did the trick.

Now the nice bash thing:

#!/bin/bash
for malo in `cat malos.txt`;
do
whois $malo | grep inetnum | awk '{print $2,$3,$4}' - | xargs ipcalc | grep '/' >> ranges.tmp
done
sort -u ranges.tmp > ranges.txt

As you see we are grepping inetnum from the whois as almost all ips showed the same information. This is a point of failure as it’s really possible that an ip range would have been marked in another fashion, as NetRange:. You will have to manually look for error in ipcalc.

Anyway, from a list of 2531 unique ips I got 33 internet ranges. Would be nice to check the origin country at this point.

Hope this helps to anyone

I’ve just dicovered CSSH. This little ugly tool can be found in ubuntu, and I presume it would be found on all others.

CSSH stans for Central SSH and its meant for cluster administration. In an environment where you have quite a few servers with almost exactly the same configuration you can type any command in the main window and its typed to every host at the same time.

I’ll have to try at work, cause there I have this type of environment. At home… with two connections to the same server  its not very exciting.

Try it and give your opinion!