Network debugging
How to approach network problems; Link layer problems; Network layer problems; traceroute; tcpdump; Transport and application layers; Ethereal
The chances are quite good that you'll have some problems somewhere when you set up your network. FreeBSD gives you a large number of tools with which to find and solve the problem.
In this chapter, we'll consider a methodology of debugging network problems. In the process, we'll look at the programs that help debugging. It will help to have your finger in Chapter 16 while reading this section.
How to approach network problems
Recall from Chapter 16 that network software and hardware operate on at least four layers. If one layer doesn't work, the ones above won't either. When solving problems, it obviously makes sense to start at the bottom and work up.
Most people understand this up to a point. Nobody expects a PPP connection to the Internet to work if the modem can't dial the ISP. On the other hand, a large number of messages to the FreeBSD-questions mailing list show that many people seem to think that once this connection has been established, everything else will work automatically. If it doesn't, they're puzzled.
Unfortunately, the Net isn't that simple. In fact, it's too complicated to give a hard-and-fast methodology at all Much network debugging can look more like magic than anything rational Nevertheless, a surprising number of network problems can be solved by using the steps below. Even if they don't solve your problem, read through them.
They might give you some ideas about where to look.
Link layer problems
To test your link layer, start with ping, ping is a relatively simple program that sends an ICMP echo packet to a specific IP address and checks the reply. ICMP is the Internet Control Message Protocol is used for error reporting and testing.
G. Lehey FreeBSD Operating System
See TCP/IP Illustrated, by Richard Stevens, for more information.
A typical ping output might look like:
$ ping bumble
PING bumble.example.org (223.147.37.156): 56 data bytes
- 64 bytes from 223.147.37.156: icmp_seq=O td=255 time= 1.137 ms
- 64 bytes from 223.147.37.156: icmp_seq=l td=255 time=0.640 ms
- 64 bytes from 223.147.37.156: icmp_seq=2 ttl=255 time=0.671 ms
- 64 bytes from 223.147.37.156: icmp_seq=3 td=255 time=0.612 ms AC
- — bumble.example.org ping statistics —
- 4packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.612/0.765/1.137/0.216 ms
In this case, we are sending the messages to the system bumble.example.org. By default, ping sends messages of 56 bytes. With the IP header, this makes packets of 64 bytes. By default, ping continues until you stop it—notice the ëñ indicating that this invocation was stopped by pressing Ctrl-C.
The information that ping gives you isn't much, but it's useful:
- • It tells you how long it takes for each packet to get to its destination and back.
- • It tells you how many packets didn't make it.
- • It also prints a summary of packet statistics.
But what if this doesn't work? You enter your ping command, and all you get is:
$ ping wait
PING wait.exanple.org (223.147.37.4): 56 data bytes
AC
- — wait.example.org ping statistics —
- 5 packets transmitted, 0 packets received, 100% packet loss
Obviously, something's wrong here. We'll look at it in more detail below. This is very different, however, from this situation:
$ ping presto
AC
In the second case, even after some time, nothing happened at alL ping didn't print the PING message, and when we hit Ctrl-C there was no further output. This is indicative of a name resolution problem ping can't print the first line until it has found the IP address of the system, in other words, until it has performed a DNS lookup. If we wait long enough, it will time out, and we get the message ping: cannot resolve presto: Unknown host. If this happens, use the IP address, not the name. DNS is an application, so we won't try to debug it until we've debugged the link and network layers.
If things don't work out, there are two possibilities:
- • If both systems are on the same network, it's a link layer problem We'll look at that first.
- • If the systems are on two different networks, it might be a network layer problem That's more complicated: we don't know which network to look at. It could be either of the networks on which the systems are located, or it could also be a problem with one of the networks on the way. How do you find out where your packets get lost? First you check the link layer. If it checks out OK, and the problem still exists, continue with the network layer on page 406.
So what can cause link layer problems? There are a number of possibilities:
• One of the interfaces (source or destination) could be misconfigured. They should both have the same range of network addresses. For example, the following two interface configurations cannot talk to each other directly, even if they're on the same physical network:
machine 1
dcO:flags=8843 machine 2 xlO: flags=8943 inet 192.168.27.1 netmask OxffffffOO broadcast 192.168.27.255 • If you see something like this on an Ethernet interface, it's pretty clear that it has a cabling problem G. Lehey FreeBSD Operating System xlO: flags=8943 inet 192.168.27.1 netmask OxffffffOO broadcast 192.168.27.255 media: Ethernet autoselect (none) status: no carrier In this case, check the physical connections. If you're using UTP, check that you have the right kind of cable, normally a "straight-through" cable. Il you accidentally use a crossover cable where you need a straight-through cable, or vice versa, you will not get any connection. Also, many hubs and switches have a "crossover” switch that achieves the same result. The method of setting the connection depends on the board you are using. PCI boards are not normally a problem, because the driver can set the parameters directly, but ISA boards can drive you crazy. In the case of very old boards, such as the Western Digital 8003, you may need to set jumpers. In others, you may need to run the setup utility under DOS, and with others you can set it with the link flags to ifconfig. For example, on a 3Com 3c509 "combo" board, you can set the connection like this: # ifconfig epO -linkO setBNC # ifconfig epO linkO -linkl set AUI # ifconfig epO linkO linkl set UTP This example is correct for the ep driver, but not necessarily for other Ethernet boards: each board has its own flags. Read the man page for the board for the correct flags. • If your interface looks OK, the next thing to do is to see whether you can send data to other machines on the network. If so, of course, you should continue your search on the machine that isn't responding. If none are working, you probably have a cabling problem. Ona wire less network, you need to check for a number of additional problems, ifconfig should show something like this: wiO: flags=8843 inet 192.168.27.17 netmask OxffffffOO broadcast 192.168.27.255 ether 00:02:2d:21:54:4c media: IEEE 802.11 Wireless Ethernet autoselect (DS/llMbps) status: associated ssid "FreeBSD IBSS" 1:"" stationname "FreeBSD WaveLAN/IEEE node" channel 3 authmode OPEN powersavemode OFF powersavesleep 100 wepmode OFF weptxkey 1 wepkey 2:64-bit 0x123456789a 3:128-bit 0xl23456789abcdefl23456789a There are many things to check here: • Do you have the same operating mode? This example shows a card operating in BSS or IBSS mode. By contrast, you might see this: media: IEEE 802.11 Wireless Ethernet autoselect (DS/llMbps
In this case, the interface is operating in so-called 'Lucent demo ad-hoc" mode, which is not the same thing as "ad-hoc" mode (which in turn is better called IBSS mode). IBSS mode ("ad-hoc") and BSS mode are compatible. IBSS mode and 'Lucent demo ad-hoc" mode are not. See Chapter 17, page 306 for further details. G. Lehey FreeBSD Operating System that ifconfig does not display the WEP key unless you are root. Your card may show associated even if the WEP key doesn't match. In such a case, it knows about the network, but it can't communicate with it. After checking all these things, you should have a connection. But you may not be home yet: $ ping 223.147.37.255 PING 223.147.37.255 (223.147.37.255): 56 data bytes FreeBSD systems do not respond to broadcast pings, but most other systems do, so this effectively counts the number of non-BSD machines on a network. • Check the times. A ping across an Ethernet should take between about 0.2 and 2 ms, a ping across a wireless connection should take between 2 and 12 ms, a ping across an ISDN connection should take about 30 ms, a ping across a 56 kb/s analogue connection should take about 100 ms, and a ping across a satellite connection should take about 250 ms in each G. Lehey FreeBSD Operating System direction. AH of these times are for idle lines, and the time can go up to over 5 seconds for a slow line transferring large blocks of data across a serial line (for example, ftping a file). In this example, some line traffic delayed the response to individual pings. Network layer problems Once we know the link layer is working correctly, we can turn our attention to the next layer up, the network layer. Well, first we should check if the problem is still with us. We need additional tools for the network layer, ping is a useful tool for telling you whether data is getting through to the destination, and if so, how much is getting through. But what if your local network checks out just fine, and you can't reach a remote network? Or if you're losing 40% of your packets to ññûëêà: http://foo.bar.org, and the remaining ones are taking up to 5 seconds to get through. Where's the problem? Based on the recent "upgrade" your ISP performed, and the fact that you've had trouble getting to other sites, you suspect that the performance problems might be occurring in the ISP's net. How can you find out? As we saw while investigating the link layer, a complete failure is often easier to fix than a partial failure. If nothing at all is getting through, you probably have a routing problem Check the routing table with netstat. On bumble, you might see: $ netstat -r Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default gw UGSc 0 8 xlO localhost localhost UH 2 525 loO 223.147.37 link#l UC 6 0 xlO sat-gw 00:80:c6:19:d3:fa UHLW 0 0 xlO 1150 bumble 00:50:da:cfil7:d3 UHLW 0 24 loO presto 00:80:c6:?9:a6:c8 UHLW 0 5 xlO 1200 freebie 00:50:da:cfi07:35 UHLW 6 760334 xlO 1159 223.147.37.255 ff:ff:ff:ff:ff:ff UHLW 1 403 xlO G. Lehey FreeBSD Operating System The default route is via gw, which is correct. The first thing is to ensure that you can ping gw; that's a link level issue, so we'll assume that you can. But what if you try to ping a remote system and you see something like this? # ping rider.fc.net PING rider.fc.net (207.170.123.194): 56 data bytes 36 bytes from gw.example.org (223.147.37.5): Destination Host Unreachable Vr HL TOS Len ID Fig off TIL Pro cks Src Dst Vr HL TOS Len ID Fig off TTL Pro cks Src Dst 4 5 00 6800 c5e7 0 0000 fe 01 2460 223.147.37.2 207.170.123. AC These are ICMP messages from gw indicating that it does not know where to send the data. This is almost certainty a routing problem; on gw you might see something like: $ netstat -r Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire localhost localhost UH 1 123 loO free-gw.example.ne exorg-gw.example. UH 223.147.37 link#l UC 11 sat-gw 00:80:c6:f9:d3:fa UHLW 5 1295329 dcO 1027 bumble 00:50:da:cfil7:d3 UHLW 2 760207 dcO 802 flame 08:00:20:76:6c:7b UHLW 2 426341 dcO 532 wantadilla 00:02:44:17:f8:da UHLW 36 19778224 dcO 1073 presto 00:80:c6:fi:a6:c8 UHLW 1 1122321 dcO 742 freebie 00:50:da:cf:07:35 UHLW 24 3279563 loO air-gw 00:00:b433:6da2 UHLW 4 2484 dcO 653 kimchi 00:00:21:ca:6efl UHLW 0 1 dcO 829 223.147.37.127 link#l UHLW 0 5 dcO G. Lehey fumble link#l UHLW 3 51246373 dcO The problem here is that there is no default route. Add it with the route command: # route add default free-gw.example.net # netstat -r Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default free-gw.example.ne UGSc 24 5724 pppO localhost localhost UH 1 123 loO ...etc See Chapter 17, page 309, for more details, including how to ensure that the routes will be added automatically at boot time. But what if the routes look right, you don't get any ICMP messages, and no data gets through? You don't always get ICMP messages when the data can't get through. The logical next place to look is free-gw.example.net, but there's a problem with that: as the administrator of exanple.org, you don't have access to example.net’s machines. You can call them up, of course, but before you do you should be reasonably sure it's their problem. You can find out more information with traceroute. traceroute traceroute sends UDP packets to the destination, but it modifies the time-to-live field in the IP header (see page 280) so that, initially at any rate, they don't get there. As we saw there, the time-to-live field specifies the number of hops that a packet can go before it is discarded. When it is, the system that discards it should send back an ICMP destination unreachable message, traceroute uses this feature and sends out packets with time-to-live set first to one, then to two, and so on. It prints the IP address of the system that sends the "destination unreachable" message and the time it took, thus giving something like a two-dimensional ping. Here's an example to ññûëêà: http://hub.FreeBSD.org: $ traceroute hub.freebsd.org traceroute to hub.freebsd.org (204.216.27.18), 30 hops max, 40 byte packets By default, traceroute tries each hop three times and prints out the times as they happen, so if the reponse time is more than about 300 ms, you'll notice it as it happens. If there is no reply after a timeout period (default 5 seconds), traceroute prints an asterisk (*). You'll also occasionally notice a significant delay at the beginning of a line, although the response time seems reasonable. In this case, the delay is probably caused by a DNS reverse lookup for the name of the system. If this becomes a problem (maybe because the global DNS servers aren't reachable), you can turn off DNS reverse lookup using the -n fag. If you look more carefully at the times in the example above, you'll see three groups of times: G. Lehey FreeBSD Operating System 85 ms to transfer. If this were a satellite connection, things would be much slower: the total distance from ground station to satellite and back to the ground is 72,000 km, which takes a total of 240 ms to propagate. Back to our problem. If we see something like the output in the previous example, we know that there's no reason to call up the people at ññûëêà: http://example.net: it's not their problem. This might just be overloading on the global Internet. On the other hand, what about this? $ traceroute hub.freebsd.org traceroute to hub.freebsd.org (204.216.27.18), 30 hops max, 40 byte packets AC You've fixed your routing problems, but you still can't get data off the system. There are a number of possibilities here: $ sysctl net.inet.ip.forwarding net.inet.ip.forwarding: 1 For a router, this value should be 1. If it's 0, change it with: # sysctl -w net.inet.ip.forwarding=l net.inet.ip.forwarding: 0 -> 1 See page 313 for further details, including how to ensure that this syscd is set correcdy when the system starts. 1 You may be trying to use a non-routable IP address such as those in the range 192.168.x.x. You can't do that. If you don't have enough globally visible IP address, you'll need to run some kind of aliasing package, such as NAT. See Chapter 22, page 393, for further details. But maybe the data gets one hop further: $ traceroute hub.freebsd.org traceroute to hub.freebsd.org (204.216.27.18), 30 hops max, 40 byte packets Igw (223.147.37.5) 1.138 ms 0.811ms 0.800 ms 2free-gw.example.net (139.130.136.129) 131.913 ms 122.231 ms 134.694 2 * * * /| * * * In this case, there is almost certainly a problem at ññûëêà: http://example.net. This would be the correct time to use the telephone. High packet loss But maybe data is getting through. Well, some data, anyway. Consider this ping session: PING freefalLFreeBSD.org (216.136.204.21): 56 data bytes In this case, we have a connection. But look carefully at those sequence numbers. At one point, four packets in a row (sequence 19 to 22) get lost. How high a packet drop rate is still acceptable? 1% or 2% is probably still (barely) acceptable. By the time you get to 10%, though, things look a lot worse. 10% packet drop rate doesn't mean that your connection slows down by 10%. For every dropped packet, you have a minimum delay of one second until TCP retries it. If that retried packet gets dropped too—which it will every 10 dropped packets if you have a 10% drop rate—the second retry takes another three seconds. If you're transmitting packets of 64 bytes over a 33.6 kb/s link, you can normally get about 60 packets through per second. With 10% packet loss, the time to get these packets through is about eight seconds, a throughput loss of 87.5%. With 20% packet loss, the results are even more dramatic. Now 12 of the 60 packets have to be retried, and 2.4 of them will be retried a second time (for three seconds delay), and 0.48 of them will be retried a third time (six seconds delay). This makes a total of 22 seconds delay, a through put degradation of nearly 96%. Theoretically, you might think that the degradation would not be as bad for big packets, such as you might have with file transfers with ftp. In fact, the situation is worse then: in most cases the packet drop rate rises sharply with the packet size, and it's common enough that ftp times out completely before it can transfer a file. To get a better over view of what's going on, let's look at another program, G. Lehey FreeBSD Operating System tcpdump. tcpdump tcpdump is a program that monitors a network interface and displays selected information that passes through it. It uses the BerkeleyPacketFilter (bpf), an optional component of the kernel It's included in recent versions of the GENERIC kernel but it's possible to remove it. If you don't configure the Berkeley?acketFilter, you get a message like: tcpdump: /dev/bpfO: device not configured tcpdump poses a potential security problem: you can use it to read anything that goes over the network. As a result, you must be root to run it. The simplest way to run it is without any parameters. This causes tcpdump to monitor and display all traffic on the first active network interface, normally Ethernet: # tcpdump tcpdump: listening on epO c93c c06d c589 c06d c5ff 007b 007b 0038 G. Lehey FreeBSD Operating System This output looks confusing at first. Let's look at it in more detail: • The first message shows the interface on which tcpdump listens. By default, it is the first running interface that it finds, tcpdump searches the list of interfaces in the sequence that ifconfig -a displays. Generally you can assume that it will find the primary Ethernet interface. If you want to listen on another interface, specify it on the command line. For example, to listen on a PPP interface, you would enter # tcpdump -i tunO G. Lehey FreeBSD Operating System for this period, so presto needs to find freebie's IP address again. Packet loss revisited Getting back to our packet loss problem, the following example shows the result of communicating on a less-than-perfect ssh connection to ññûëêà: http://hub.FreeBSD.org, specifically between port 1019 on freebie and port 22, the ssh port, on hub. To make things more readable, the names have been truncated to freebie and hub. In real-life output, they would be reported as ññûëêà: http://freebie.example.org and ññûëêà: http Ó/hub. FreeBSD, org. In addition, tcpdump reports a tos (type of service) field, which has also been removed. It doesn't interest us here. # tcpdump -i pppO host hub.freebsd.org 14:16:35.990506 freebie.1019 > hub.22: 14:16:36.552149 hub.22 > freebie.1019: 14:16:36.722290 freebie.1019 > hub.22: 14:16:39.344229 freebie.1019 > hub.22: 14:16:41.321850 freebie.1019 > hub.22: P 20:40(20) ack 77 win 17520 (DF; P 77:97(20) ack 40 win 17520 (DF . ack 97 win 17520 (DF) P 40:60(20) ack 97 win 17520 (DF; P 40:60(20) ack 97 win 17520 (DF The first line shows freebie sending bytes 20 to 40 of the stream to hub, and also acknowledging receipt of everything up to byte 77 of the stream from hub. On the next line, hub sends bytes 77 to 97 and acknowledges receiving up to byte 40 ol the stream from freebie, freebie then sends another 20 bytes and acknowledges what it has received from hub. After two seconds, freebie has not received an acknowledgment from hub that its data has been received, so on the last line it sends the packet again. G. Lehey FreeBSD Operating System This is the missing acknowledgment—t came another second later, along with some more data, freebie acknowledges receiving it, but doesn't send anymore data. 14:16:47.428694 freebie.1019 > hub.22: P 60:80(20) ack 117 win 17520 (DF) 14:16:48.590805 freebie.1019 > hub.22: P 80:100(20) ack 117 win 17520 (DI 14:16:49.055735 freebie.1019 > hub.22: P 100:120(20) ack 117 win 17520 (E 14:16:49.190703 hub.22 > freebie.1019: P 137:157(20) ack 100 win 17520 (E Five seconds later, freebie sends more data, up to byte 120 to hub. hub replies with its own data an acknowledgment up to byte 100. Unfortunately, the data it sent (bytes 137 to 157) don't line up with the last previously received data (byte 117 at 14:16:42.316150): bytes 117 to 137 are missing, freebie thus repeats the previous acknowledgment and then continues sending its data: Here, hub has sent yet more data, now acknowledging the data that freebie sent at 14:16:49.055735. It still hasn't sent the data in the byte range 117 to 136, so freebie resends the last acknowledgment again and continues sending data: Again hub has sent more data, still without sending the missing packet, freebie tries yet again, and then continues sending data: Finally, hub resends the missing data, with bytes from 117 to 217. freebie is now happy, and acknowledges receipt of aH the data up to 217. That's all we transmitted, so after about 1.5 seconds the two systems exchange final acknowledgments: G. Lehey FreeBSD Operating System ack 217 win 17420 (DF). ack 3255467530 win 17520 . ack 1 win 17520 (DF This connection is less than perfect. Why? You can use traceroute to find out where it's happening, but unless the place is within your ISP's network, you can't do much about it. Transport and application layers If you have got this far, the chances are that things will now work. Problems in transport layer are rare. About the only things that can still cause problems are the individual applications. We'll look at some of these in the relevant chapters. One particular problem is the Domain Name Service. This is such an integral part of the Internet Protocols that people tend to forget that it's really an application. Il you get a timeout accessing a web URL, for example, there's a good chance that DNS is causing the problem. Take a look at Chapter 21, The Domain Name Service, for some ideas. Ethereal tcpdump is a powerful tool, but the examples above show that the output isn't the easiest thing in the world to read. An alternative is ethereal, a program in the Ports Collection (/usr/ports/net/ethereal) that displays the data in much more detail, as Figure 23-1 shows. The screen is divided into three windows: G. Lehey FreeBSD Operating System In practice, you'd probably want to scale the window much larger than in this example. This image shows part of the password for a telnet login session being returned. It illustrates one of the reasons you should never use telnet to connect across the Internet. Ðèñ. 23.1. ethereal display G. Lehey FreeBSD Operating System
