General Troubleshooting Methods

One of the first signs of trouble on the network is a loss of communications by one or more hosts. If a host refuses to come up at all the first time it is added to the network, the problem may lie in one of the configuration files, or in the network interface. If a single host suddenly develops a problem, the network interface may be the cause. If the hosts on a network can communicate with each other but not with other networks, the problem could lie with the router, or it could lie in another network.

You can use the ifconfig program to obtain information on network interfaces and netstat to display routing tables and protocol statistics. Third-party network diagnostic programs provide a number of troubleshooting utilities. Refer to third-party documentation for information.

Less obvious are the causes of problems that degrade performance on the network. For example, you can use tools like ping to quantify problems like the loss of packets by a host.

Running Software Checks

If there is trouble on the network, some actions that you can take to diagnose and fix software-related problems include

1. Using the netstat command to display network information.

2. Checking the hosts database to make sure that the entries are correct and up to date.

3. If you are running RARP, checking the Ethernet addresses in the ethers database to make sure that the entries are correct and up to date.

4. Trying to connect by telnet to the local host.

5. Ensuring that the network daemon inetd is running. To do this, log in as superuser and type:

# ps -ef | grep inetd

Here is an example of output displayed if the inetd daemon is running:

root 57 1 0 Apr 04 ? 3:19 /usr/sbin/inetd -s root 4218 4198 0 17:57:23 pts/3 0:00 grep inetd

ping Command

Use the ping command to find out whether there is IP connectivity to a particular host. The basic syntax is:

/usr/sbin/ping host [timeout]

where host is the host name of the machine in question. The optional timeout argument indicates the time in seconds for ping to keep trying to reach the machine--20 seconds by default. The ping(1M) man page describes additional syntaxes and options.

When you run ping, the ICMP protocol sends a datagram to the host you specify, asking for a response. (ICMP is the protocol responsible for error handling on a TCP/IP network. See "ICMP Protocol" on page 15 for details.)

Suppose you type:

$ ping elvis

If host elvis is up, this message is displayed:

elvis is alive

indicating that elvis responded to the ICMP request. However, if elvis is down or cannot receive the ICMP packets, you receive the following response from ping:

no answer from elvis

If you suspect that a machine may be losing packets even though it is up, you can use the -s option of ping to try and detect the problem. For example, type:

$ ping -s elvis

ping continually sends packets to elvis until you send an interrupt character or a timeout occurs. The responses on your screen will resemble:

PING elvis: 56 data bytes 64 bytes from 129.144.50.21: icmp_seq=0. time=80. ms 64 bytes from 129.144.50.21: icmp_seq=1. time=0. ms 64 bytes from 129.144.50.21: icmp_seq=2. time=0. ms 64 bytes from 129.144.50.21: icmp_seq=3. time=0. ms . . . ----elvis PING Statistics---- 4 packets transmitted, 4 packets received, 0% packet loss round-trip (ms) min/avg/max = 0/20/80

The packet-loss statistic indicates whether the host has dropped packets.

If ping fails, check the status of the network reported by ifconfig and netstat, as described in "ifconfig Command" and "netstat Command" on page 93.

ifconfig Command

The ifconfig command displays information about the configuration of an interface that you specify. (Refer to the ifconfig(1M) man page for complete details.) The syntax of ifconfig is:

ifconfig interface-name [protocol_family]

If you want information about a specific interface, for example le0, type:

$ ifconfig le0

For an le0 interface, your output will resemble the following:

le0: flags=863<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 129.144.44.140 netmask ffffff00 broadcast 129.144.44.255 ether 8:0:20:8:el:fd

The flags section above shows that the interface is configured "up," capable of broadcasting, and not using "trailer" link level encapsulation. The mtu field tells you that this interface has a maximum transfer rate of 1500. Information on the second line includes the IP address of the host you are using, the netmask being currently used, and the IP broadcast address of the interface. The third line gives the machine address (Ethernet, in this case) of the host.

A useful ifconfig option is -a, which provides information on all interfaces on your network. For example, typing

ifconfig -a

produces:

le0: flags=49<UP,LOOPBACK,RUNNING> mtu 8232 inet 127.144.44.140 netmask ff000000 le0:flags=863<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 129.144.44.140 netmask ffffff00 broadcast 129.144.44.255 ether 8:0:20:8:el:fd

Output that indicates an interface is not running might mean a problem with that interface. In this case, see the ifconfig(1M) man page.

netstat Command

The netstat command generates displays that show network status and protocol statistics. You can display the status of TCP and UDP endpoints in table format, routing table information, and interface information.

netstat displays various types of network data depending on the command line option selected. These displays are the most useful for system administration. The syntax for this form is:

netstat [-m] [-n] [-s] [-i | -r] [-f address_family]

The most frequently used options for determining network status are: -s, -r, and -i. See the netstat(1M) man page for a description of the options.

Displaying Per Protocol Statistics

The netstat -s option displays per protocol statistics for the UDP, TCP, ICMP, and IP protocols. The result resembles the display shown in the example below. (Parts of the output have been truncated.) The information can indicate areas where a protocol is having problems. For example, statistical information from ICMP can indicate where this protocol has found errors.

UDP udpInDatagrams = 3928 udpInErrors = 0 udpOutDatagrams = 2455 TCP tcpRtoAlgorithm = 4 tcpRtoMin = 200 tcpRtoMax = 60000 tcpMaxConn = -1 tcpActiveOpens = 4 tcpPassiveOpens = 2 tcpAttemptFails = 3 tcpEstabResets = 1 tcpCurrEstab = 1 tcpOutSegs = 315 tcpOutDataSegs = 288 tcpOutDataBytes = 10547 tcpRetransSegs = 29 tcpRetransBytes = 8376 tcpOutAck = 27 tcpOutAckDelayed = 23 tcpOutUrg = 2 tcpOutWinUpdate = 2 tcpOutWinProbe = 0 tcpOutControl = 8 tcpOutRsts = 0 tcpOutFastRetrans = 1 tcpInSegs = 563 tcpInAckSegs = 289 tcpInAckBytes = 10549 tcpInDupAck = 27 tcpInAckUnsent = 0 tcpInInorderSegs = 254 tcpInInorderBytes = 673 tcpInUnorderSegs = 0 tcpInUnorderBytes = 0 tcpInDupSegs = 0 tcpInDupBytes = 0 tcpInPartDupSegs = 0 tcpInPartDupBytes = 0 tcpInPastWinSegs = 0 tcpInPastWinBytes = 0 tcpInWinProbe = 0 tcpInWinUpdate = 237 tcpInClosed = 0 tcpRttNoUpdate = 21 tcpRttUpdate = 266 tcpTimRetrans = 26 tcpTimRetransDrop = 0 tcpTimKeepalive = 0 tcpTimKeepaliveProbe= 0 tcpTimKeepaliveDrop = 0 IP ipForwarding = 2 ipDefaultTTL = 255 ipInReceives = 4518 ipInHdrErrors = 0 ipInAddrErrors = 0 ipInCksumErrs = 0 ipForwDatagrams = 0 ipForwProhibits = 0 ipInUnknownProtos = 0 ipInDiscards = 0 ipInDelivers = 4486 ipOutRequests = 2805

ipOutDiscards = 5 ipOutNoRoutes = 0 ipReasmTimeout = 60 ipReasmReqds = 2 ipReasmOKs = 2 ipReasmFails = 0 ipReasmDuplicates = 0 ipReasmPartDups = 0 ipFragOKs = 20 ipFragFails = 0 ipFragCreates = 116 ipRoutingDiscards = 0 tcpInErrs = 0 udpNoPorts = 33 udpInCksumErrs = 0 udpInOverflows = 6 rawipInOverflows = 0 ICMP icmpInMsgs = 0 icmpInErrors = 0 icmpInCksumErrs = 0 icmpInUnknowns = 0 icmpInDestUnreachs = 0 icmpInTimeExcds = 0 icmpInParmProbs = 0 icmpInSrcQuenchs = 0 icmpInRedirects = 0 icmpInBadRedirects = 0 icmpInEchos = 0 icmpInEchoReps = 0 icmpInTimestamps = 0 icmpInTimestampReps = 0 icmpInAddrMasks = 0 icmpInAddrMaskReps = 0 icmpInFragNeeded = 0 icmpOutMsgs = 7 icmpOutDrops = 5 icmpOutErrors = 0 icmpOutDestUnreachs = 1 icmpOutTimeExcds = 0 icmpOutParmProbs = 0 icmpOutSrcQuenchs = 6 icmpOutRedirects = 0 icmpOutEchos = 0 icmpOutEchoReps = 0 icmpOutTimestamps = 0 icmpOutTimestampReps= 0 icmpOutAddrMasks = 0 icmpOutAddrMaskReps = 0 icmpOutFragNeeded = 0 icmpInOverflows = 0 IGMP: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 membership reports sent

Displaying Network Interface Status

The -i option of netstat shows the state of the network interfaces that are

configured with the machine where you ran the command. Here is a sample display produced by netstat -i.

Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue le0 1500 b5-spd-2f-cm tatra 14093893 8492 10174659 1119 2314178 0 lo0 8232 loopback localhost 9299762 5442 12451748 0 775125 0

Using this display, you can find out how many packets a machine thinks it has transmitted and received on each network. For example, the input packet count (Ipkts) displayed for a server may increase each time a client tries to boot, while the output packet count (Opkts) remains steady. This suggests that the server is seeing the boot request packets from the client, but does not realize it is supposed to respond to them. This might be caused by an incorrect address in the hosts or ethers database.

On the other hand, if the input packet count is steady over time, it means that the machine does not see the packets at all. This suggests a different type of failure, possibly a hardware problem.

Displaying Routing Table Status

The -r option of netstat displays the IP routing table. Here is a sample display produced by netstat -r run on machine tenere.

Routing tables Destination Gateway Flags Refcnt Use Interface temp8milptp elvis UGH 0 0 irmcpeb1-ptp0 elvis UGH 0 0 route93-ptp0 speed UGH 0 0 mtvb9-ptp0 speed UGH 0 0 . mtnside speed UG 1 567 ray-net speed UG 0 0 mtnside-eng speed UG 0 36 mtnside-eng speed UG 0 558 mtnside-eng tenere U 33 190248 le0

The first column shows the destination network, the second the router through which packets are forwarded. The U flag indicates that the route is up; the G flag indicates that the route is to a gateway. The H flag indicates that the destination is a fully qualified host address, rather than a network.

The Refcnt column shows the number of active uses per route, and the Use column shows the number of packets sent per route. Finally, the Interface column shows the network interface that the route uses.

Logging Network Problems

If you suspect a routing daemon malfunction, you can log its actions, including all packet transfers. To create a log file of routing daemon actions, supply a file name when you start up the routed daemon. For example:

# /usr/sbin/in.routed /var/routerlog

---

Caution - On a busy network, this may generate almost continuous output.

---

Displaying Packet Contents

snoop captures network packets and displays their contents. Packets can be displayed as they are received, or saved to a file. For complete information, refer to the snoop(1M) man page.