Posts Tagged ssh

Executing the same command on different servers: pssh

With pssh (parallel-ssh) you can execute the same command on different hosts.

Pssh is a simple python script, the uses pretty much no python module, so it’s simple to install (it’s also packaged at least in ubuntu).

To use pssh, you need to create a hosts file which contains a list of hosts (one by line) followed by a username to use on that host, then just execute this command parallel-ssh -h hosts-file "command", it will execute “command” on all the hosts that are in the given hosts-file. I copied my ssh public-key so I don’t need to type my password on any server, if you don’t have your key, pssh will prompt for a password.
Pssh has a --print option that prints the output of the command execution, host by host, on the shell you’re launching pssh from, if you don’t use that option , it creates 1 file per host with the result.

Pssh is really nice, but, would be better if I could use the aliases I use in my .ssh/config for hostnames in my hosts-file. Maybe one day, I’ll make a patch to pssh so it uses your .ssh/config to recognize hosts and users in your hosts-file. Nice tool, anyway !

Advertisements

Leave a Comment

Transfering disk images with low disk space

If you want to dump a disk to a disk image you will use for example:
dd if=/dev/hdx1 of=/tmp/disk.img
and then, you will probably copy this disk image to another machine. The thing is, if you have low disk space than the size of /dev/hdx1 on your machine, you won’t be able to dump the disk to transfer it to the other machine.
There is a solution that I use, as usually with, ssh and pipe:
ssh hostname "dd if=/dev/hdx1" |dd of=/tmp/disk.img
on the machine receiving the image or
dd if=/dev/hdx1 |ssh hostname "dd of=/tmp/disk.img"
on the machine sending the image, so the content of the disk is directly transmitted through ssh !
That’s it !
Maybe you can tune the blocksize of the dd command so the troughput is better, maybe a futur article on that 🙂

Comments (3)

ipv6 urls

URLs are written like this: protocol://host-or-address:port/path-or-function

What happens with ipv6 is that addresses contains colons (“:”) , so how do you specify the port number in your web browser ? The same happens when you do an scp: you usually do scp user@host:path/to/file /local/path, how can you differenciate the host part and the path which are also seperated with a colon ?

The answer is: USE BRACKETS !
an ipv6 url can be written like this:

Also, a scp command with ipv6 addresses can be like this:

  • scp user@[fe80::abcd:abcd:abcd:abcd]:/etc/resolv.conf /tmp

I hope it’s usefull !

Leave a Comment

copying databases

Similarly to my last tip (copying directory with ssh and tar) , you can also copy databases. It’s pretty simple, here is my magic command:
mysqldump -ppassword db |ssh user@remote "cat - | mysql -u dbuser -ppassword db"

Here, you can also gzip or bzip2 the input, and it should be very efficient, because mysqldump output is pure ascii with sql, gzip and bzip2 will easily find good pattern for compression.

Also, as usual, using my.cnf files, you don’t need -ppassword parameters.

Comments (2)

scp with little files, use ssh and tar

scp is slow at copying a lot of little files. It’s probably because, it creates a ssh connection and then runs in a loop that open the file locally , reads it, sends it, creates the file remotely and restarts that loop. I think that the cost of opening files and creating files is high and tar is a lot more efficient at it (if you know exactly why, please post a comment). That’s why I often use ssh with a combination of tar to copy a directory with a lot of little files (let’s say a website with a tone of 10kB php files).

To achieve this, where I typically use:
scp -r a-directory/ user@host:
I would use:
tar cfz - a-directory | ssh user@host "tar zxvf -"

First, I create a tar file that output to stdout (tar cf -, the “-” is to output to stdout) then I pipe the output to an ssh connection that execute a command that read stdin and pass it to tar (tar xvf -, where “-” is stdin).

Surely, you can remove the “z” in the input and output tar , so the content won’t be compressed with gzip, or replace it with “j” to use bzip (with gnu tar).

that would be:

tar cf - a-directory | ssh user@host "tar xvf -" or
tar cfj - a-directory | ssh user@host "tar jxvf -"

Also, tar knows how to use ssh:
tar cvfz user@host:file.tar a-directory but this method have the drawback of creating a tar file remotely, not copying the directory, you still need to uncompress the tar on the other side.

Now some numbers, this is not a benchmark, this is just an indication, I had the idea of writing this article while copying a directory I needed to copy. Numbers may vary with size of files, number of files, local cpu, remote cpu, network bandwidth and probably other parameters:


# du -sh a-directory
109M a-directory
Read the rest of this entry »

Comments (3)

recursive scp and symlinks

To copy recursively with scp, you use scp -r.
The thing is that if you have symlinks in your directory the content of the directory being pointed to by the symlinks will be copied, this is not necessarily what you want (the symlink will be followed instead of being preserved).
Scp have no option to specify that you don’t want to follow symlinks.
If you want to preserver symlinks, you should use rsync:
rsync -avz -e ssh /src/dir user@remote.host:dst/dir

Be carefull, if you preserve symlinks, the newly created symlinks on the remote server can point to a non existant path.
It’s strange that scp seem to have no option to not follow symlinks. If you’re aware of one, please drop me a comment !

Comments (2)