Posts Tagged networking

netcat as a logging tcp proxy

I felt I needed to write an article about netcat, so here is it !
Netcat is an incredibly usefull tool, that allows you to play with tcp connection easily from the shell.
Basically, as it name implies, it’s just cat over the network, but what its name doesn’t tell you is that it also can act as a socket listener.
So let’s play with pipes, here is one of my favourite use of netcat:

mkfifo proxypipe
cat proxypipe | nc -l -p 80 | tee -a inflow | nc localhost 81 | tee -a outflow 1>proxypipe

This command will redirect traffic from localhost:80 to localhost:81, in the inflow file you while find the incoming http request, in the outfile, you will find the http response from the server.
Similarly, you can do this:

cat proxypipe | nc -l 80 | tee -a inflow | sed 's/^Host.*/Host: www.google.fr/' |  nc www.google.fr 80 | tee -a outflow >proxypipe

This will allow your browser to point to google using http://localhost .
Anyway, this is my favourite but netcat has thounds of other uses, have a look at it !
It can be usefull for file transfers (gzip|nc) , performance measurement (dd|gzip), protocol debugging (replaying requests), security testing (nc does port scan) …

Comments (3)

Executing the same command on different servers: pssh

With pssh (parallel-ssh) you can execute the same command on different hosts.

Pssh is a simple python script, the uses pretty much no python module, so it’s simple to install (it’s also packaged at least in ubuntu).

To use pssh, you need to create a hosts file which contains a list of hosts (one by line) followed by a username to use on that host, then just execute this command parallel-ssh -h hosts-file "command", it will execute “command” on all the hosts that are in the given hosts-file. I copied my ssh public-key so I don’t need to type my password on any server, if you don’t have your key, pssh will prompt for a password.
Pssh has a --print option that prints the output of the command execution, host by host, on the shell you’re launching pssh from, if you don’t use that option , it creates 1 file per host with the result.

Pssh is really nice, but, would be better if I could use the aliases I use in my .ssh/config for hostnames in my hosts-file. Maybe one day, I’ll make a patch to pssh so it uses your .ssh/config to recognize hosts and users in your hosts-file. Nice tool, anyway !

Leave a Comment

ssmtp and gmail or google apps

Unix systems often needs a local mailer, but configuring and maintaining a mailer on each system is a timeloss.
You might have a gmail or google apps account. If it’s the case, you can easily configure a mailer on your systems which uses your gmail or google apps. To do so, I’ve used ssmtp and put this in /etc/ssmtp/ssmtp.conf:

root=postmaster
mailhub=smtp.gmail.com:587
AuthUser=your-mail@yourdomain.com
AuthPass=aStr4angeP45s
UseSTARTTLS=YES
hostname=the-hostname

That’s it, simple, effective, working …
To improve the things, maybe, we can use an IP address of the smtp server, so that if our DNS server doesn’t work, we still have mail on the system, but this has a drawback, if the server for which you gave an ip address changes or temporarly doesn’t work, you don’t have mail anymore.
ssmtp doesn’t seem to be able to have several mailhubs !

Comments (4)

Transfering disk images with low disk space

If you want to dump a disk to a disk image you will use for example:
dd if=/dev/hdx1 of=/tmp/disk.img
and then, you will probably copy this disk image to another machine. The thing is, if you have low disk space than the size of /dev/hdx1 on your machine, you won’t be able to dump the disk to transfer it to the other machine.
There is a solution that I use, as usually with, ssh and pipe:
ssh hostname "dd if=/dev/hdx1" |dd of=/tmp/disk.img
on the machine receiving the image or
dd if=/dev/hdx1 |ssh hostname "dd of=/tmp/disk.img"
on the machine sending the image, so the content of the disk is directly transmitted through ssh !
That’s it !
Maybe you can tune the blocksize of the dd command so the troughput is better, maybe a futur article on that 🙂

Comments (3)

ipv6 urls

URLs are written like this: protocol://host-or-address:port/path-or-function

What happens with ipv6 is that addresses contains colons (“:”) , so how do you specify the port number in your web browser ? The same happens when you do an scp: you usually do scp user@host:path/to/file /local/path, how can you differenciate the host part and the path which are also seperated with a colon ?

The answer is: USE BRACKETS !
an ipv6 url can be written like this:

Also, a scp command with ipv6 addresses can be like this:

  • scp user@[fe80::abcd:abcd:abcd:abcd]:/etc/resolv.conf /tmp

I hope it’s usefull !

Leave a Comment

Finding dns server version (only bind)

To find the version of running bind version remotely, you can type that command:
nslookup -q=txt -class=CHAOS version.bind. ns1.domain.com
or with dig:
dig @ns1.domain.com version.bind chaos txt
or with host:
host -t TXT -c chaos version.bind ns1.domain.com

If you don’t want your bind to show the version you are currently running, on a ubuntu system you will add a version "[Secured]"; directive in the options section of the file /etc/bind/named.conf.options

That’s it !

Leave a Comment

My 4 varnish tips

Varnish is a reverse proxy, If you don’t know varnish, this article is not interesting to you 😉 .

This is my 4 little tips that greatly optimizes the efficiency of the caching politics:

Removing tracking, this generates a single cache entry for different urls that generates the same content (I use “gclid” as a tracking argument, this is what google uses), use this as the hashing algorithm:

sub vcl_hash {
  vcl.hash += regsub(req.url, ”\?gclid.*”, ””);
  hash;
}

Then we can normalize compression (different browser uses different string for the “Accept-Encoding” header). Add the following in sub vcl_recv:

if (req.http.Accept-Encoding){
 if (req.http.Accept-Encoding ~ "gzip"){
  set req.http.Accept-Encoding = "gzip";
 }elsif (req.http.Accept-Encoding ~ "deflate" ) {
  set req.http.Accept-Encoding = "deflate";
 }else{
  remove req.http.Accept-Encoding;
 ;}
}

When a cookie is generated all subsequent request for any object uses that cookie, we shall remove the cookie for all static content
In sub vcl_recv add this:

if (req.url ~ "\.(js|css|jpg|png|gif|mp3|swf|flv|xml|html|ico)$"){
 remove req.http.cookie;
}

Be carefull with files with these extensions that generates dynamic content (png, jpg, gif file for captcha, html with rewrite  to php or aspx …)

To track client ip address in the log of your web server (the real one, the backend), in sub vcl_recv add this:

remove req.http.X-Forwarded-For;
set req.http.X-Forwarded-For=client.ip;

Then you can log the “X-Forwarded-For” header in your log (doing this depends on your webserver, I do that on apache and lighttpd).

Leave a Comment

Older Posts »