A random weblog but it's mine

A little one-liner: rename to lowercase recursively

smaftoul — Tue, 09 Jun 2009 09:33:33 +0000

Here is my little one-liner, because I used it today and I find it fun:

for f in `find .` ; do mv $f `echo $f | tr '[A-Z]' '[a-z]'` ; done

netcat as a logging tcp proxy

smaftoul — Wed, 13 May 2009 09:18:23 +0000

I felt I needed to write an article about netcat, so here is it !
Netcat is an incredibly usefull tool, that allows you to play with tcp connection easily from the shell.
Basically, as it name implies, it’s just cat over the network, but what its name doesn’t tell you is that it also can act as a socket listener.
So let’s play with pipes, here is one of my favourite use of netcat:

mkfifo proxypipe
cat proxypipe | nc -l -p 80 | tee -a inflow | nc localhost 81 | tee -a outflow 1>proxypipe

This command will redirect traffic from localhost:80 to localhost:81, in the inflow file you while find the incoming http request, in the outfile, you will find the http response from the server.
Similarly, you can do this:

cat proxypipe | nc -l 80 | tee -a inflow | sed 's/^Host.*/Host: www.google.fr/' |  nc www.google.fr 80 | tee -a outflow >proxypipe

This will allow your browser to point to google using http://localhost .
Anyway, this is my favourite but netcat has thounds of other uses, have a look at it !
It can be usefull for file transfers (gzip|nc) , performance measurement (dd|gzip), protocol debugging (replaying requests), security testing (nc does port scan) …

ubuntu and bind acting as slave

smaftoul — Fri, 17 Apr 2009 00:22:24 +0000

While installing a slave dns server with bind, I went into trouble. I could not understand why my slave zone would not synchronize. Actually I found these entries in /var/log/daemon.log:
named[24309]: dumping master file: /etc/bind/tmp-b0KyuKU5pG: open: permission denied named[24309]: transfer of 'domain.com/IN' from w.x.y.z#53: failed while receiving responses: permission denied
It appears that since hardy, ubuntu doesn’t allow the named process to write in /etc/bind/ while it’s running.
Ubuntu is configured to allow slave zone to stay in /var/cache/bind/db.domain.com

So your slave zone will look like:

zone "domain.com" IN {
        type slave;
        file "/var/cache/bind/db.domain.com";
        masters { w.x.y.z; };
};

For the details, it’s due to apparmor, and precisely the file /etc/apparmor.d/usr.sbin.named.
As shipped with ubuntu, this file contains the authorizations for the named process that restricts where bind can write its zones, and reserves /var/cache/bind/ as the directory where bind is supposed to put its slave zones.
This seems to me technically good because /etc is pretty much supposed to be “read-only able” (beside /etc/mtab and /etc/resolv.conf that you can put in /dev/shm or link from /var/etc). This makes me wonder where to put master zones that you want to change ? Probably in /var/lib/bind because it’s where dynamically updated zone are.

Installing redmine 0.8 on intrepid (ubuntu 8.10)

smaftoul — Thu, 16 Apr 2009 16:05:31 +0000

I’ve successfully insalled redmine pretty much easily but I needed to find out what packages to install with apt, which one with gem, which version …
Here is my magic receipe to install it all:

apt-get update 
apt-get install subversion mysql-server rubygems rake pwgen
# next line generates a password for the database
export PASSWORD=`pwgen -nc 8 1`
gem install -v=2.1.2 rails
echo "CREATE DATABASE redmine  DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci ; GRANT ALL PRIVILEGES ON redmine.* TO 'redmine'@'localhost' IDENTIFIED BY '$PASSWORD' WITH GRANT OPTION; FLUSH PRIVILEGES" | mysql 
cd /opt/
svn export http://redmine.rubyforge.org/svn/branches/0.8-stable redmine-0.8
cd redmine-0.8/
cat <> config/database.yml
production:
  adapter: mysql
  socket: /var/run/mysqld/mysqld.sock 
  database: redmine
  host: localhost
  username: redmine
  password: $PASSWORD
  encoding: utf8

EOF
rake db:migrate RAILS_ENV="production"
rake redmine:load_default_data RAILS_ENV="production"
apt-get remove pwgen subversion
RAILS_ENV="production" ./script/server

And that’s it ! Redmine is running on port 3000.
I did this on an EC2 instance and it works like a charm (ami-7cfd1a15).
Maybe next article will discuss running redmine in mongrel or apache, and creating an init script for having redmine running on boot !

HTTP’s 301 and 302

smaftoul — Wed, 15 Apr 2009 13:38:55 +0000

Just a little note, 301 is the HTTP code for “temporarly moved”, 302 is “permanently moved”.
Seems 302 is genereally more usefull and works better.
From what I noticed (I’m not sure about it), 302 has better SEO. Also some browsers seems to make better cache use with 302 and generating less requests on your webserver.

Hostname and underscore

smaftoul — Tue, 14 Apr 2009 15:20:42 +0000

RFC 952 and RFC 1123 explains the rules for choosing a hostname. I noticed recently that a lot of admins (including me) are using underscores in hostnames, but this doesn’t follow RFCs. This can lead to strange behaviours, such as mail not delivered with an RFC compliant mail server to an MX that have underscores in its name …
I noticed that because the “hostname” command on linux can set the hostname of a system, but the command doesn’t accept underscores. So guys, don’t use underscores !

MySQL “pretty printing”

smaftoul — Wed, 08 Apr 2009 13:57:44 +0000

I just discovered an apparently wide spread tip for mysql:

mysql> select 1, 2, 3, 4 ; +---+---+---+---+ | 1 | 2 | 3 | 4 | +---+---+---+---+ | 1 | 2 | 3 | 4 | +---+---+---+---+ 1 row in set (0.00 sec)

mysql> select 1, 2, 3, 4 \G; *************************** 1. row *************************** 1: 1 2: 2 3: 3 4: 4 1 row in set (0.00 sec)
As you might have seen, the difference comes from the “\G” !

It’s very usefull when you select lot of columns that doesn’t fit the width of your terminal !
Shame on me I didn’t knew that before !

Two usefull MySQL requests

smaftoul — Wed, 21 Jan 2009 13:45:58 +0000

To analyse a database content, I used several times theses two requests:

Number of columns for all tables in a database:
select TABLE_NAME, count(TABLE_NAME) from columns where TABLE_SCHEMA="dbname" group by TABLE_NAME order by count (TABLE_NAME);
Number of rows for all tables in a database:
select table_name, table_rows from information_schema.tables where table_schema = 'dbname' order by table_rows;

These are simple but effective to know where most of your data are if your database starting to get slow.
Also, it can help for finding a partitionning or sharding strategy.

subversion hooks and “light” continuous integration

smaftoul — Mon, 19 Jan 2009 15:46:08 +0000

I recently made something really simple which developpers found really usefull:
When a developper commits some code on a subversion repository, he wants to test if it works on an integration plateform.
In my case, this was on a php project, so no complilation, to do so, I just need to copy the file in the right place !
Actually, the right place for me was on another machine.

So, how do you copy files on a remote machine automatically when developpers commits some new code ?
Easy: use hooks

In your subversion repository you will find a directory called “hooks” , all files in this directory are called something.tmpl , those files are examples / templates.

What i did was put this content in post-commit:

REPOS="$1" 
REV="$2" 
RESULT=`/usr/bin/svnlook dirs-changed "$REPOS" -r $REV | grep -e "the/dir/in/the/repo/project"`; 
if [ ! -z "$RESULT" ]; then 
    /usr/bin/svn export file:///var/lib/svn/the/dir/in/the/repo/project /tmp/project
    /usr/bin/scp -i /home/apache/id_dsa -r /tmp/project/* user@host:/var/www/project
    /bin/rm -rf /tmp/project
fi

Then I needed to do some stuff:

chmod +x post-commit
I also needed to generate ssh keys, put the private key in /home/apache/id_dsa, make it owned by the process running subversion (for me apache, because I’m using apache’s mod_dav_svn), and copy the key to a users’ authorized_keys file …
Authorize the remote host’s key for the user running the post-commit script (su www-data then ssh host and accept the key)

I you make a mistake in your post-commit script (like I did the first time I tried), you get the error on the output when you commit the code !

Easy, fun and nice !

limits , open files, pam, daemons and scalability

smaftoul — Fri, 26 Dec 2008 11:01:28 +0000

Be carefull, some daemons have some extensive activity !
It depends on the daemon (its goal, its activity) but some opens a lot of files, some uses a lot a network connections, some both of them (and some takes a lot of cpu , others a lot of ram) …
Anyway, on unix any process is limited by the system with the ulimit mechanism, and one of these “annoying” limits is the maximum number of file descriptor , that is the maximum number of open files a single process can use including network connections (on unix, a network connections has a file descriptor associated to it).

Depending on your daemon and your case, you sometimes need to get rid of these limits (be sure you need to get rid a of these limits). To do so , on a ubuntu , I have:

Added the following to /etc/security/limits.conf:
nobody soft nofile 4096 nobody hard nofile 63536
Added the following to /etc/pam.d/su:
session required pam_limits.so

Here, I have added to the user nobody the right to open 63536 files PER PROCESS (yeah, per process, not for the user).
I added this thing to /etc/pam.d/su because when you run a daemon from an init script, it uses the su mecanism to run as the user it’s configured for.
I have added the user nobody because unfortunately , my running daemon is memcached which runs as nobody (should probably run as memcache user).

Be carefull, sometime you think you need this, but you don’t: I wanted to increase the number of files apache user can open, but it wasn’t needed: my apache was running with forks (because php needs fork), any new connection spawns a new process which has it’s 1024 file descriptor, the whole apache on the system was opening lot more than 1024 files, as I had a limit of 1100 forks running which I reached several times, but a single process wasn’t opening more than 1024 files.
You can check how much a process of files opened with lsof (which stands for LiSt Open Files):
lsof -p 3423 -n |wc -l where 3423 is the PID of the process you want to inscpect ( “-n” means don’t do dns resolution, lsof lists sockets and tries to put names to every address, a server process generally accepts a lot of connection from a lot of different hosts. Without “-n” , lsof is really slow )

That’s it, I hope yo uwill make a good use of these tips !