Archive for August, 2008
August 21, 2008 at 5:12 pm
· Filed under Uncategorized ·Tagged backup, bzip2, database, dump, gzip, mysql, mysqldump, network, ssh, system, tar, tip, tips
Similarly to my last tip (copying directory with ssh and tar) , you can also copy databases. It’s pretty simple, here is my magic command:
mysqldump -ppassword db |ssh user@remote "cat - | mysql -u dbuser -ppassword db"
Here, you can also gzip or bzip2 the input, and it should be very efficient, because mysqldump output is pure ascii with sql, gzip and bzip2 will easily find good pattern for compression.
Also, as usual, using my.cnf files, you don’t need -ppassword parameters.
Permalink
August 19, 2008 at 4:33 pm
· Filed under Uncategorized ·Tagged bandwidth, bzip2, compression, copying, directory, files, gzip, scp, shell, ssh, system, tar, tip, tips
scp is slow at copying a lot of little files. It’s probably because, it creates a ssh connection and then runs in a loop that open the file locally , reads it, sends it, creates the file remotely and restarts that loop. I think that the cost of opening files and creating files is high and tar is a lot more efficient at it (if you know exactly why, please post a comment). That’s why I often use ssh with a combination of tar to copy a directory with a lot of little files (let’s say a website with a tone of 10kB php files).
To achieve this, where I typically use:
scp -r a-directory/ user@host:
I would use:
tar cfz - a-directory | ssh user@host "tar zxvf -"
First, I create a tar file that output to stdout (tar cf -, the “-” is to output to stdout) then I pipe the output to an ssh connection that execute a command that read stdin and pass it to tar (tar xvf -, where “-” is stdin).
Surely, you can remove the “z” in the input and output tar , so the content won’t be compressed with gzip, or replace it with “j” to use bzip (with gnu tar).
that would be:
tar cf - a-directory | ssh user@host "tar xvf -" or
tar cfj - a-directory | ssh user@host "tar jxvf -"
Also, tar knows how to use ssh:
tar cvfz user@host:file.tar a-directory but this method have the drawback of creating a tar file remotely, not copying the directory, you still need to uncompress the tar on the other side.
Now some numbers, this is not a benchmark, this is just an indication, I had the idea of writing this article while copying a directory I needed to copy. Numbers may vary with size of files, number of files, local cpu, remote cpu, network bandwidth and probably other parameters:
# du -sh a-directory
109M a-directory
Read the rest of this entry »
Permalink
August 19, 2008 at 12:53 pm
· Filed under Uncategorized ·Tagged daemon, file, flash, policy, request, security, server
If you are making a flash client for your protocol or server and receive <policy-file-request/>. on your server (let’s say with a sniffer), that’s because flash version newer than 9.0.115 has changed the security policy.
Here is the explanation of the new security policy. In brief, the flash application tries to connect to port 843 of the server where the socket needs to be connected to fetch a crossdomain.xml .
This article explains how it works and gives two flashpolicyd (flash policy daemon), one version in perl, another one in python, they are both in that archive.
I also found one in ruby that handles timeouts and errors.
Maybe I will take the one in python, daemonize it, add better error handling, and will package it for ubuntu, but maybe only
Permalink
August 13, 2008 at 11:03 am
· Filed under Uncategorized ·Tagged cli, command line, network, recursive, rsync, scp, shell, ssh, symlinks, system
To copy recursively with scp, you use scp -r.
The thing is that if you have symlinks in your directory the content of the directory being pointed to by the symlinks will be copied, this is not necessarily what you want (the symlink will be followed instead of being preserved).
Scp have no option to specify that you don’t want to follow symlinks.
If you want to preserver symlinks, you should use rsync:
rsync -avz -e ssh /src/dir user@remote.host:dst/dir
Be carefull, if you preserve symlinks, the newly created symlinks on the remote server can point to a non existant path.
It’s strange that scp seem to have no option to not follow symlinks. If you’re aware of one, please drop me a comment !
Permalink
August 5, 2008 at 3:49 pm
· Filed under Uncategorized ·Tagged architecture, linux, network, networking, proxy, reverse proxy, scalability, system, unix, web
Varnish is a reverse proxy, If you don’t know varnish, this article is not interesting to you
.
This is my 4 little tips that greatly optimizes the efficiency of the caching politics:
Removing tracking, this generates a single cache entry for different urls that generates the same content (I use “gclid” as a tracking argument, this is what google uses), use this as the hashing algorithm:
sub vcl_hash {
vcl.hash += regsub(req.url, ”\?gclid.*”, ””);
hash;
}
Then we can normalize compression (different browser uses different string for the “Accept-Encoding” header). Add the following in sub vcl_recv:
if (req.http.Accept-Encoding){
if (req.http.Accept-Encoding ~ "gzip"){
set req.http.Accept-Encoding = "gzip";
}elsif (req.http.Accept-Encoding ~ "deflate" ) {
set req.http.Accept-Encoding = "deflate";
}else{
remove req.http.Accept-Encoding;
;}
}
When a cookie is generated all subsequent request for any object uses that cookie, we shall remove the cookie for all static content
In sub vcl_recv add this:
if (req.url ~ "\.(js|css|jpg|png|gif|mp3|swf|flv|xml|html|ico)$"){
remove req.http.cookie;
}
Be carefull with files with these extensions that generates dynamic content (png, jpg, gif file for captcha, html with rewrite to php or aspx ...)
To track client ip address in the log of your web server (the real one, the backend), in sub vcl_recv add this:
remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For=client.ip;
Then you can log the "X-Forwarded-For" header in your log (doing this depends on your webserver, I do that on apache and lighttpd).
Permalink
August 4, 2008 at 10:44 am
· Filed under Uncategorized ·Tagged admin, backup, database, db, mysql, script, scripting, system
Don’t need to say much, everything is in the title of this post
.
Here is my mysql database backup script:
DIR="/var/backups/db/"
MAIL="your@mail.address"
LOGFILE=$DIR/backupdb.log
function backup_db {
HOST="$1"
USER="$2"
PASS="$3"
DB="$4"
MINSIZE="$5"
BACKUPFILESUFFIX="`date +%m%d`.bz2"
DBLIST=`echo "show databases" | mysql -u backup -p$PASS -h $HOST`
NUMDB=`echo $DBLIST |wc -w`
if [ ! -d ${dir} ]; then
mkdir ${dir}
fi
if [ ! -e $DIR/count.$HOST ]; then
echo $NUMDB > $DIR/count.$HOST
fi
COUNT=`cat $DIR/count.$HOST`
if [ "$COUNT" -lt "$NUMDB" ]; then
echo -e "Databases list: $DBLIST" | mail -s "New database, maybe new backups needed!" $MAIL
echo $NUMDB > $DIR/count.$HOST
fi
/usr/bin/mysqldump -u $USER -p$PASS -h $HOST --routines $DB|bzip2 > $DIR/$DB.$BACKUPFILESUFFIX
if [ $? != 0 ] ; then
echo -e "Return code is : $? and log file contains:\n `cat $LOGFILE`" | mail -s "Backup MySQL $HOST: $DB Error" $MAIL
fi
SIZE=`du -sk $DIR/$DB.$BACKUPFILESUFFIX| cut -f1`
if [ "$SIZE" -lt $MINSIZE ]; then
echo -e "File is smaller than $MINSIZE k, printing an ls output:\n `ls -l $DIR`" | mail -s "Backup MySQL $HOST potential error" $MAIL
fi
}
# Cleaning up old files, or disks won't fill
DIR="/var/backups/db/"
find $DIR -ctime +7 -name "*bz2" -exec rm {} \; -print
backup_db "192.168.0.1" "user" "password" "database" "9999"
To backup your databases add at the end of script, one line per database, based on this format:
backup_db "host" "user" "password" "database name" "min_size"
You also need to create a backup user on your database, I use this script:
CREATE USER 'backup'@ 'backup-host' IDENTIFIED BY 'a-password';
GRANT SHOW DATABASES ON * . * TO 'backup'@ 'backup-host' IDENTIFIED BY 'a-password ;
GRANT SELECT ,
LOCK TABLES ,
SHOW VIEW ON `a_database_to_backup` . * TO 'backup'@ 'backup-host';
And also add a crontab entry:
cat <<EOF >/etc/cron.d/dbbackup
MAILTO=root
0 5 * * * root /var/backups/db/backupdb.sh 2>&1 >/var/backups/db/backupdb.log
EOF
chmod +x /etc/cron.d/dbbackup
This script have some features I have implemented that I find usefull:
- It mails you when a new databases is created (devs sometimes create a database but don’t inform me, they need backup of it, in case)
- It’s easy to add new databases to backup
- It checks for a minimum size, you know that some databases won’t be less than a fixed size, if it happens, there is probably a problem with the backup script or within the database
- It also backups stored procedures (we use the
--routines option of mysqldump)
- It has a 7 days rotation mechanism, so the disk don’t fill
- The databases are compressed
Some improvement that can be done to this script:
- Better error handling, I’m not really sure how it works, I made this script pretty much fast for my daily needs
- Use mk-parallel-dump from the maatkit
- Use .my.cnf, and don’t display password in the script, is it better ?
- please comment to give me some ideas
Permalink
August 4, 2008 at 9:55 am
· Filed under Uncategorized ·Tagged apple, dfu, iphone, ipod touch, jailbreak, pwn, pwnage
To jailbreak an iPhone or an iPod touch, you need to get the device in what is called DFU mode but there is a lot of confusion because some sites badly explains how to get in DFU mode. Remember this golden rule: if the screen is not blank, YOU ARE NOT IN DFU MODE
To get the device in DFU mode follow the instruction here (This site has simple and clear instructions).
Update: This link also explains how to put the iPhone in DFU mode.
Permalink