You can have the latest and greatest servers around with the fastest processors, dump trucks full of memory, and disk space for days and a simple network issue could bring everything to a screeching halt.
Servers rely on rock solid network connectivity to actually utilize all that computational horsepower. Whether it’s downloading workload information or serving customer web requests, you need to be sure your network can handle it.
Diagnosing troublesome network paths and getting an accurate performance benchmark on links isn’t always easy. Luckily, in the Linux world, we have access to a plethora of tools to help tackle some of these difficult tasks.
In this article, I will guide us through some built-in and third-party Linux utilities that make diagnosing network problems and crafting solutions piece of cake.
A network engineer’s toolkit wouldn’t be complete without iPerf. This monumentally useful network utility allows you to perform end-to-end throughput and latency testing. The biggest benefit of using iPerf is that you will have control over both ends of the connection. You won’t simply be sending traffic and waiting for it’s return you’ll have access to statistics at both sides.
In order to use iPerf properly you must setup both a client and server instance. The server instance runs alongside or on your existing servers and client instances can be started from anywhere you wish to test from.
The clients will connect to the server on a specific port and then have the ability to perform a battery of detailed tests.
Start a simple iPerf server by running the following command:
This will run iPerf and listen automatically on port
5001. Now, connect a client and begin the default bandwidth test by issuing the following:
iperf -c <host_ip_running_iperf_server>
In the following example, the iPerf test was performed on the same machine over localhost:
Client connecting to 127.0.0.1, TCP port 5001
TCP window size: 144 KByte (default)
[ 1] local 127.0.0.1 port 52340 connected with 127.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 1] 0.00-10.00 sec 141 GBytes 121 Gbits/sec
Obviously, for this test the bandwidth is quite high since the traffic never leaves the host. If you’re diagnosing slow links, hosts or other middleboxes it’s evident just how useful this can be when throughout your network.
Check out the official iPerf website below for more information:
This is an absolute work horse of the network diagnostics world. Using
tcpdump you can capture network traffic as it passes through a host and also uses it to replay or rewrite packets. With
tcpdump you can save packet capture files that can be read either in plain text or by the infamous Wireshark utility
The best part about
tcpdump is its flexibility. It can be as simple or as advanced as you want it to be. For example, let’s look at how to capture traffic destined towards Google’s public DNS resolve
sudo tcpdump host 22.214.171.124
This command tells
tcpdump to filter only for traffic sourced from or destined to the host with IP address
126.96.36.199. You should see output similar to this:
In this example, we clearly see the bidirectional traffic flow from our host (
172.16.30.223) to Google and back. Using
tcpdump you can quickly and easily figure out where specific traffic is being heard on the network.
- More information is available on the official
- The Red Hat blog has a great primer on using
tcpdumpfilters available here.
This is like ping but on steroids. With hping you can perform a whole set of additional tasks that you can’t with normal ping. While it was initially developed as a security tool, it provides useful everyday network testing functionality as well.
In hping you can spoof network addresses, send floods of packets and even run complex port scans. These are mere toys compared to the functionality of the TCP/UDP section of hping.
Using hping you can set a ton of different flags and perform detailed TCP/UDP application testing by crafting precisely the traffic you need:
-s --baseport base source port (default random)
-p --destport [+][+]<port> destination port(default 0) ctrl+z inc/dec
-k --keep keep still source port
-w --win winsize (default 64)
-O --tcpoff set fake tcp data offset (instead of tcphdrlen / 4)
-Q --seqnum shows only tcp sequence number
-b --badcksum (try to) send packets with a bad IP checksum
many systems will fix the IP checksum sending the packet
so you'll get bad UDP/TCP checksum instead.
-M --setseq set TCP sequence number
-L --setack set TCP ack
-F --fin set FIN flag
-S --syn set SYN flag
-R --rst set RST flag
-P --push set PUSH flag
-A --ack set ACK flag
-U --urg set URG flag
These can be quite handy if you’re troubleshooting a particularly cranky network application and want to try and replicate some of it’s traffic patterns.
Official hping website available below:
Want to know who is talking to a host? What ports are in use? What state an application is in? Netstat works wonders to solve these problems.
This utility has been around for almost forty years. There’s a good reason for this. Netstat shows you exactly what you need to know about who is connected to you.
The most common use for Netstat is showing open ports for a machine. Using the following command you can see this information:
You should see some output similar to this:
In the information above we can clearly see that this host is listening on a port
22 for incoming SSH connections. Netstat will also replace well-known ports with their service names for convenience.
If you’re trying to figure out if a particular port is open and listening then using Netstat is the way to go.
We saved the best for last. This is a big one.
If the tools listed previously were tools in a toolbox then Scapy is the whole damn garage. This application lets you build complete packets from scratch one layer at a time. If you have the time and patience you can craft almost any network traffic you want in Scapy.
While technically Scapy is a Python module, it can be easily installed and run just like any other command-line program. The benefit is since it’s built-in Python, you can create complex Scapy setups in an intuitive language without having to fight Bash or other annoying config files.
Scapy also provides a command-line interpreter for interactively building packets and other data. They claim it contains so much functionality:
“It can easily handle most classical tasks like scanning, tracerouting, probing, unit tests, attacks or network discovery (it can replace hping, 85% of nmap, arpspoof, arp-sk, arping, tcpdump, tshark, p0f, etc.) .”
— Official Scapy Website
This is truly a powerhouse when it comes to the entire network stack. If you’re interested in really learning how each layer works, this library will give you all the tools you need.
Check out the official Scapy website below for more usage and installation information: