Kus' Blog

At Atlas we love documentation, as all of you must too! Every system and service is documented into a wiki.

The first time you are told to document a system, you log into that system and start typing commands to get what you need and copy it to a wiki, then apply the appropiate format.

The second time you apply some pipe magic with the awk magic dust, and voila! the info you need as formatted as possible

Some useful commands you will appreciate:

• Getting partition table formatted, with size (change ext3 to get other fs types):
  df -P -h -t ext3 | grep -v Filesystem | awk '{print "| "$1" | "$6" | "$2" |"}'
• Getting NFS partition mappings formatted:
  
  cat /etc/fstab | grep nfs | grep -v '#' | awk '{print "| "$1" | "$2" |"}'
• Getting apache virtual host configuration:
  
  cat your-vhost.conf | egrep "ServerName|ServerAlias|RewriteRule|DocumentRoot|ErrorLog|CustomLog|ErrorDocument|VirtualHost|SSL" | awk '{print "| | "$1" | "$2$3$4$5$6" |"}'
  This one needs to be polished a little to get rid of those virtual hosts lines and put its info on the first column.

Put this post to your favourites as it will be updated as more info is needed to format.

New job, new problems, new hacks! The next hack tries to take all the juice from linux’s pipes.

The task which I had to do was a database migration, from one mysql server to another one. The first problem were different versions, so hot copy wasn’t an option (actually few times is an option, unless it’s not in production ). The next big issue, were the size of the data, its big, really big (from my point of view) some DB reach 800M, for me is big enough to be a big problem

Mysqldump + mysql but this is good for 2 DB? I wanted to be selective in which DB to migrate, plus data should travel encrypted and transfer should delay as minimum as possible.

The script I got is this one (showing main loop, what’s interesting):

for db in `cat sm2-hf-dbs-kk.txt`
do
echo "Creant database a ${remote_sql}..."
ssh $remote_sql "echo \"create database if not exists ${db}\" | mysql -u ${db_user_rw} -p${db_pass_rw}"
echo "Fet"
echo "Donant permisos sobre la BD a rw..."
ssh $remote_sql "echo \"grant select, insert, update, lock tables, create, drop, alter on *.* to '${db_user_rw}'@'localhost';flush privileges\" | mysql -u ${remote_admin} -p${remote_admin_pass}"
echo "Fet"
echo "Fent dump gzipat i enviant a ${remote_sql}"
mysqldump --single-transaction -u ${db_user_r} -p${db_pass_r} ${db} | gzip | ssh $remote_sql "gunzip - | mysql -u ${db_user_rw} -p${db_pass_rw} ${db}"
echo "Fet"
done

The new server was empty, so we first ssh to it and from there we create a database. The next step was to give perms over the new db… I see now it can be done in one single line.

After this comes the magic. The script makes a dump of the db, it passes through gzip, this way we dramatically improve speed over the net. This pipe stream is passed through ssh to the other server unzip the info and put into the new mysql server. Pipe Magic rules!

I prefer  ssh the machine rather than setting a tunnel because the tunnel should set before the script, this way we don’t care. The other point is that the ssh should have a valid pair of keys to access it.

That’s all folks! Hope this enlightens someone…

While researching why the servers where overloaded I saw that some webpages where attacked/scanned from hundreds of different ips. Most of the where from Caravan Networks from Russia…

I know almost for sure that no webs pages in our servers have russian clients target so I wanted to ban them all, from their IP range. The problem was this range wasn’t trivial to get, so I did this little hack to get them.

Saddly it isn’t 100% effective but nearly 80%

First thing to get is a list of the attackers ip. In this case we got them from grepping through an apache access log.

cat /usr/local/apache/domlogs/kedume9/somedomain.com | grep "POST /web/ht" | cut -f 1 -d " " | sort -u malos.txt

did the trick.

Now the nice bash thing:

#!/bin/bash
for malo in `cat malos.txt`;
do
whois $malo | grep inetnum | awk '{print $2,$3,$4}' - | xargs ipcalc | grep '/' >> ranges.tmp
done
sort -u ranges.tmp > ranges.txt

As you see we are grepping inetnum from the whois as almost all ips showed the same information. This is a point of failure as it’s really possible that an ip range would have been marked in another fashion, as NetRange:. You will have to manually look for error in ipcalc.

Anyway, from a list of 2531 unique ips I got 33 internet ranges. Would be nice to check the origin country at this point.

Hope this helps to anyone

Com molts webdevs ens trobem ens trobem dia a dia que hem de treballar amb un servidor de desenvolupament (per sort) i després passar els canvis a la versió en producció. Bé pel que fa els fitxers no hi ha problema, `cp kkdev kkprod`, però el problema (per mi) le trobat quan es fan canvis a la BD i no s’apunten…

Per aquest cas n’ha sortit el sevendiffs.php, no té nom més que aquest. El que fa es llistar les taules d’una BD de desenvolupament i al costat les de producció. De la manera com està les taules han d’haver estat creades en el mateix ordre així ports comparar fàcilment.

Us els deixo penjat i ja el provareu al vostre server xD.

Codi Font SevenDiffs

Realment no tenia poca feina, però el senyor Putxi m’havia reptat a fer un scriptillu per la seva inacabable tasca de buscar rimes amb els nom o cognoms de les persones. D’aquesta manera tan patètica ha sorgit el BuscaRimes! És GPL tot i que la programació fot bastanta pena, realment és codi 100% quick&dirty.

Avui d’anada cap a Barcelona han sorgit, ja, unes primeres millores:

• Usar una BD per millorar l’eficiència, tot i que disposa d’un rudementari sistema de catx .

Hi ha una limitació a la mida de la terminació a 4 caràcters.

Aquí podeu trobar el BuscaRimes! i aqui el codi font. Si us plau no us talleu en millorar-lo i envieu les vostres versions i les penjaré!