Here is my little one-liner, because I used it today and I find it fun:
for f in `find .` ; do mv $f `echo $f | tr '[A-Z]' '[a-z]'` ; done
A little one-liner: rename to lowercase recursively
netcat as a logging tcp proxy
I felt I needed to write an article about netcat, so here is it !
Netcat is an incredibly usefull tool, that allows you to play with tcp connection easily from the shell.
Basically, as it name implies, it’s just cat over the network, but what its name doesn’t tell you is that it also can act as a socket listener.
So let’s play with pipes, here is one of my favourite use of netcat:
mkfifo proxypipe cat proxypipe | nc -l -p 80 | tee -a inflow | nc localhost 81 | tee -a outflow 1>proxypipe
This command will redirect traffic from localhost:80 to localhost:81, in the inflow file you while find the incoming http request, in the outfile, you will find the http response from the server.
Similarly, you can do this:
cat proxypipe | nc -l 80 | tee -a inflow | sed 's/^Host.*/Host: www.google.fr/' | nc www.google.fr 80 | tee -a outflow >proxypipe
This will allow your browser to point to google using http://localhost .
Anyway, this is my favourite but netcat has thounds of other uses, have a look at it !
It can be usefull for file transfers (gzip|nc) , performance measurement (dd|gzip), protocol debugging (replaying requests), security testing (nc does port scan) …
Installing redmine 0.8 on intrepid (ubuntu 8.10)
I’ve successfully insalled redmine pretty much easily but I needed to find out what packages to install with apt, which one with gem, which version …
Here is my magic receipe to install it all:
apt-get update apt-get install subversion mysql-server rubygems rake pwgen # next line generates a password for the database export PASSWORD=`pwgen -nc 8 1` gem install -v=2.1.2 rails echo "CREATE DATABASE redmine DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci ; GRANT ALL PRIVILEGES ON redmine.* TO 'redmine'@'localhost' IDENTIFIED BY '$PASSWORD' WITH GRANT OPTION; FLUSH PRIVILEGES" | mysql cd /opt/ svn export http://redmine.rubyforge.org/svn/branches/0.8-stable redmine-0.8 cd redmine-0.8/ cat <<EOF >> config/database.yml production: adapter: mysql socket: /var/run/mysqld/mysqld.sock database: redmine host: localhost username: redmine password: $PASSWORD encoding: utf8 EOF rake db:migrate RAILS_ENV="production" rake redmine:load_default_data RAILS_ENV="production" apt-get remove pwgen subversion RAILS_ENV="production" ./script/server
And that’s it ! Redmine is running on port 3000.
I did this on an EC2 instance and it works like a charm (ami-7cfd1a15).
Maybe next article will discuss running redmine in mongrel or apache, and creating an init script for having redmine running on boot !
HTTP’s 301 and 302
Just a little note, 301 is the HTTP code for “temporarly moved”, 302 is “permanently moved”.
Seems 302 is genereally more usefull and works better.
From what I noticed (I’m not sure about it), 302 has better SEO. Also some browsers seems to make better cache use with 302 and generating less requests on your webserver.
Hostname and underscore
RFC 952 and RFC 1123 explains the rules for choosing a hostname. I noticed recently that a lot of admins (including me) are using underscores in hostnames, but this doesn’t follow RFCs. This can lead to strange behaviours, such as mail not delivered with an RFC compliant mail server to an MX that have underscores in its name …
I noticed that because the “hostname” command on linux can set the hostname of a system, but the command doesn’t accept underscores. So guys, don’t use underscores !
MySQL “pretty printing”
I just discovered an apparently wide spread tip for mysql:
mysql> select 1, 2, 3, 4 ;
+---+---+---+---+
| 1 | 2 | 3 | 4 |
+---+---+---+---+
| 1 | 2 | 3 | 4 |
+---+---+---+---+
1 row in set (0.00 sec)
mysql> select 1, 2, 3, 4 \G;
*************************** 1. row ***************************
1: 1
2: 2
3: 3
4: 4
1 row in set (0.00 sec)
As you might have seen, the difference comes from the “\G” !
It’s very usefull when you select lot of columns that doesn’t fit the width of your terminal !
Shame on me I didn’t knew that before !
Two usefull MySQL requests
To analyse a database content, I used several times theses two requests:
- Number of columns for all tables in a database:
select TABLE_NAME, count(TABLE_NAME) from columns where TABLE_SCHEMA="dbname" group by TABLE_NAME order by count (TABLE_NAME); - Number of rows for all tables in a database:
select table_name, table_rows from information_schema.tables where table_schema = 'dbname' order by table_rows;
These are simple but effective to know where most of your data are if your database starting to get slow.
Also, it can help for finding a partitionning or sharding strategy.
subversion hooks and “light” continuous integration
I recently made something really simple which developpers found really usefull:
When a developper commits some code on a subversion repository, he wants to test if it works on an integration plateform.
In my case, this was on a php project, so no complilation, to do so, I just need to copy the file in the right place !
Actually, the right place for me was on another machine.
So, how do you copy files on a remote machine automatically when developpers commits some new code ?
Easy: use hooks
In your subversion repository you will find a directory called “hooks” , all files in this directory are called something.tmpl , those files are examples / templates.
What i did was put this content in post-commit:
REPOS="$1"
REV="$2"
RESULT=`/usr/bin/svnlook dirs-changed "$REPOS" -r $REV | grep -e "the/dir/in/the/repo/project"`;
if [ ! -z "$RESULT" ]; then
/usr/bin/svn export file:///var/lib/svn/the/dir/in/the/repo/project /tmp/project
/usr/bin/scp -i /home/apache/id_dsa -r /tmp/project/* user@host:/var/www/project
/bin/rm -rf /tmp/project
fi
Then I needed to do some stuff:
chmod +x post-commit- I also needed to generate ssh keys, put the private key in /home/apache/id_dsa, make it owned by the process running subversion (for me apache, because I’m using apache’s mod_dav_svn), and copy the key to a users’ authorized_keys file …
- Authorize the remote host’s key for the user running the post-commit script (
su www-datathenssh hostand accept the key)
I you make a mistake in your post-commit script (like I did the first time I tried), you get the error on the output when you commit the code !
Easy, fun and nice !
limits , open files, pam, daemons and scalability
Be carefull, some daemons have some extensive activity !
It depends on the daemon (its goal, its activity) but some opens a lot of files, some uses a lot a network connections, some both of them (and some takes a lot of cpu , others a lot of ram) …
Anyway, on unix any process is limited by the system with the ulimit mechanism, and one of these “annoying” limits is the maximum number of file descriptor , that is the maximum number of open files a single process can use including network connections (on unix, a network connections has a file descriptor associated to it).
Depending on your daemon and your case, you sometimes need to get rid of these limits (be sure you need to get rid a of these limits). To do so , on a ubuntu , I have:
- Added the following to
/etc/security/limits.conf:
nobody soft nofile 4096
nobody hard nofile 63536
- Added the following to
/etc/pam.d/su:
session required pam_limits.so
Here, I have added to the user nobody the right to open 63536 files PER PROCESS (yeah, per process, not for the user).
I added this thing to /etc/pam.d/su because when you run a daemon from an init script, it uses the su mecanism to run as the user it’s configured for.
I have added the user nobody because unfortunately , my running daemon is memcached which runs as nobody (should probably run as memcache user).
Be carefull, sometime you think you need this, but you don’t: I wanted to increase the number of files apache user can open, but it wasn’t needed: my apache was running with forks (because php needs fork), any new connection spawns a new process which has it’s 1024 file descriptor, the whole apache on the system was opening lot more than 1024 files, as I had a limit of 1100 forks running which I reached several times, but a single process wasn’t opening more than 1024 files.
You can check how much a process of files opened with lsof (which stands for LiSt Open Files):
lsof -p 3423 -n |wc -l where 3423 is the PID of the process you want to inscpect ( “-n” means don’t do dns resolution, lsof lists sockets and tries to put names to every address, a server process generally accepts a lot of connection from a lot of different hosts. Without “-n” , lsof is really slow )
That’s it, I hope yo uwill make a good use of these tips ! 
