APPENDIX A

Using NFS Performance-Monitoring and Benchmarking Tools

---

This appendix discusses tools that help you monitor NFS and network performance. These tools generate information that you can use to tune and improve performance. See Chapter 3, "Analyzing NFS Performance," and Chapter 4, "Configuring the Server and the Client to Maximize NFS Performance."

For more information about these tools, refer to their man pages (where applicable) and the manual Security, Performance, and Accounting Administration, published by SunSoft Press. For third-party tools, refer to the product documentation.

This chapter also describes LADDIS, an NFS file server benchmarking tool.

---

NFS Monitoring Tools

TABLE A-1 describes the tools that you can use to monitor NFS operations and performance.

TABLE A-1

Tool	Function
iostat	Reports I/O statistics, including disk I/O activity
nfsstat	Reports NFS statistics: NFS and RPC (Remote Procedure Call) interfaces to the kernel; can also be used to initialize statistical information
nfswatch	Shows NFS transactions classified by file system; nfswatch is a public domain tool with source code available on the URL: http://www.ers.ibm.com/~davy/software/ nfswatch.html
sar	Reports system activity such as CPU utilization, buffer activity, and disk and tape device activity
SharpShooter*	Pinpoints bottlenecks, balances NFS load across clients and servers; shows effect of distributed applications and balances network traffic across servers; accounts for disk usage by user or group
vmstat	Reports virtual memory statistics including disk activity
* by Network General Corporation formerly AIM Technology

For additional networking and network monitoring utilities, see the URL: http://www.alw.nih.gov/Security/prog-network.html

---

Network Monitoring Tools

Use the tools described in TABLE A-2 to monitor network performance as it relates to NFS.

TABLE A-2

Tool	Function
snoop	Displays information about specified packets on Ethernet
netstat	Displays the contents of network-related data structures
ping	Sends ICMP ECHO_REQUEST packets to network hosts
NetMetrix Load Monitor	Handles network load monitoring and characterization of load in terms of time, source, destination, protocol, and packet size
SunNet Manager	Performs network device monitoring and troubleshooting
LAN analyzers: Network General Sniffer, Novell/Excelan Lanalyzer	Performs packet analysis

snoop

The snoop command is part of the Solaris 2.x software environment. The snoop command must run by root (#) to capture packets in promiscuous mode. To capture packets in non-promiscuous mode, or to analyze packets from a captured file, you do not need to be superuser.

In promiscuous mode, the interface turns off its filter, which enables you to see all packets on the subnet, whether or not they are addressed to your system. You can observe other packets not destined for your system. Promiscuous mode is limited to root.

Using the snoop command turns a Sun system into a network sniffer, which can detect network problems. It also captures a certain number of network packets, enables you to trace the calls from each client to each server, and displays the contents of the packets. You can also save the contents of the packets to a file, which you can inspect later.

The snoop command does the following:

• Logs or displays packets selectively
• Provides accurate time stamps for checking network Remote Procedure Call (RPC) response time
• Formats packets and protocol information in a user-friendly manner

  The snoop command can display packets in a single-line summary or in expanded form. In summary form, only the data pertaining to the highest level protocol is displayed. For example, an NFS packet will have only NFS information displayed. The underlying RPC, UDP (User Datagram Protocol), IP (Internet Protocol), and network frame information is suppressed, but can be displayed if you choose either of the verbose (-v or -V) options.

The snoop command uses both the packet filter and buffer modules of the Data Link Provider Interface (DLPI) so the packets can be captured efficiently and transmitted to or received from the network.

To view or capture all traffic between any two systems, run snoop on a third system.

The snoop command is a useful tool if you are considering subnetting, since it is a packet analysis tool. You can use the output of the snoop command to drive scripts that accumulate load statistics. The program is capable of breaking the packet header in order to debug it, and to investigate the source of incompatibility problems.

The following shows some examples of how to use snoop.

Looking at Selected Packets in a Capture File (pkts)

The statistics show which client is making a read request, and the left column shows the time in seconds, with a resolution of about 4 microseconds.

When a read or write request is made, be sure the server doesn't time-out. If it does, the client has to re-send again, and the client's IP code will break up the write block into smaller UDP blocks. The default write time is .07 seconds. The time-out factor is a tunable parameter in the mount command.

# snoop -i pkts -p99,108 99 0.0027 boutique -> sunroof NFS C GETATTR FH=8E6C 100 0.0046 sunroof -> boutique NFS R GETATTR OK 101 0.0080 boutique -> sunroof NFS C RENAME FH=8E6C MTra00192 to .nfs08 102 0.0102 marmot -> viper NFS C LOOKUP FH=561E screen.r.13.i386 103 0.0072 viper -> marmot NFS R LOOKUP No such file or directory 104 0.0085 bugbomb -> sunroof RLOGIN C PORT=1023 h 105 0.0005 kandinsky -> sparky RSTAT C Get Statistics 106 0.0004 beeblebrox -> sunroof NFS C GETATTR FH=0307 107 0.0021 sparky -> kandinsky RSTAT R 108 0.0073 office -> jeremiah NFS C READ FH=2584 at 40960 for 8192

CODE EXAMPLE A-1 Output of the snoop -i pkts -p99, 108 Command

The following describes the arguments to the snoop command.

-i pkts	Displays packets previously captured in the pkts file
-p99,108	Selects packets 99 through 108 to be displayed from a capture file; the first number 99, is the first packet to be captured; the last number, 108, is the last packet to be captured; the first packet in a capture file is packet 1

· To get more information on a packet, type:

# snoop -i pkts -v 101

The command snoop -i pkts -v 101 obtains more detailed information on packet 101. The following describes the previous snoop command arguments.

-i pkts	Displays packets previously captured in the pkts file
-v	Verbose mode; prints packet headers in detail for packet 101; use this option only when you need information on selected packets

The output of the snoop -i pkts -v 101 command shown in CODE EXAMPLE A-2, which follows, shows the Ethernet, IP, UDP, RPC, and NFS headers that are captured in snoop.

ETHER: ----- Ether Header ----- ETHER: ETHER: Packet 101 arrived at 16:09:53.59 ETHER: Packet size = 210 bytes ETHER: Destination = 8:0:20:1:3d:94, Sun ETHER: Source = 8:0:69:1:5f:e, Silicon Graphics ETHER: Ethertype = 0800 (IP) ETHER: IP: ----- IP Header ----- IP: IP: Version = 4, header length = 20 bytes IP: Type of service = 00 IP: ..0. .... = routine IP: ...0 .... = normal delay IP: .... 0... = normal throughput IP: .... .0.. = normal reliability IP: Total length = 196 bytes IP: Identification 19846 IP: Flags = 0X IP: .0.. .... = may fragment IP: ..0. .... = more fragments IP: Fragment offset = 0 bytes IP: Time to live = 255 seconds/hops IP: Protocol = 17 (UDP) IP: Header checksum = 18DC IP: Source address = 129.144.40.222, boutique IP: Destination address = 129.144.40.200, sunroof IP: UDP: ----- UDP Header ----- UDP: UDP: Source port = 1023 UDP: Destination port = 2049 (Sun RPC) UDP: Length = 176 UDP: Checksum = 0 UDP: RPC: ----- SUN RPC Header ----- RPC: RPC: Transaction id = 665905 RPC: Type = 0 (Call) RPC: RPC version = 2 RPC: Program = 100003 (NFS), version = 2, procedure = 1 RPC: Credentials: Flavor = 1 (Unix), len = 32 bytes

CODE EXAMPLE A-2 Output of the snoop -i pkts -v 101 Command

RPC: Time = 06-Mar-90 07:26:58 RPC: Hostname = boutique RPC: Uid = 0, Gid = 1 RPC: Groups = 1 RPC: Verifier : Flavor = 0 (None), len = 0 bytes RPC: NFS: ----- SUN NFS ----- NFS: NFS: Proc = 11 (Rename) NFS: File handle = 000016430000000100080000305A1C47 NFS: 597A0000000800002046314AFC450000 NFS: File name = MTra00192 NFS: File handle = 000016430000000100080000305A1C47 NFS: 597A0000000800002046314AFC450000 NFS: File name = .nfs08 NFS:

CODE EXAMPLE A-2 Output of the sp -i pkts -v 101 Command (Continued)

· To view NFS packets, type:

# snoop -i pkts rpc nfs and sunroof and boutique 1 0.0000 boutique -> sunroof NFS C GETATTR FH=8E6C 2 0.0046 sunroof -> boutique NFS R GETATTR OK 3 0.0080 boutique -> sunroof NFS C RENAME FH=8E6C MTra00192 to .nfs08

This example gives a view of the NFS packets between the systems sunroof and boutique. The arguments to the previous snoop command are:.

-i pkts	Displays packets previously captured in the pkts file
rpc nfs	Displays packets for an RPC call or reply packet for the NFS protocol; the option following nfs is the name of an RPC protocol from /etc/ rpc or a program number

and         Performs a logical and operation between two boolean values;  for 
            example, sunroof boutique is the same as sunroof and 
            boutique

· To save packets to a new capture file, type:

# snoop -i pkts -o pkts.nfs rpc nfs sunroof boutique

The arguments to the previous snoop command are described below.

-i pkts	Displays packets previously captured in the pkts file
-o pkts.nfs	Saves the displayed packets in the pkts.nfs output file
rpc nfs	Displays packets for an RPC call or reply packet for the NFS protocol; the option following nfs is the name of an RPC protocol from /etc/rpc or a program number

See the snoop man page for additional details on options used with the snoop command and additional information about using snoop.

---

LADDIS

SPEC SFS 1 (0.97 LADDIS) is a NFS file server benchmark incorporated by SPEC as a member of the System-level File Server (SFS) Benchmark Suite. Two reference points are considered when reporting 097.LADDIS:

• NFS operation throughput

  The peak number of NFS operations the target server can complete in a given number of milliseconds. The larger the number of operations an NFS server can support, the more users it can serve.

• Response time

  The average time needed for an NFS client to receive a reply from a target server in response to an NFS request. The response time of an NFS server is the client's perception of how fast the server is.

LADDIS is designed so that its workload can be incrementally increased until the target server performance falls below a certain level. That point is defined as an average response time exceeding 50ms. This restriction is applied when deriving SPECnfs_A93, the maximum throughput in NFS operations per second for which the response time does not exceed 50 ms.

If throughput continues to increase with the workload, the throughput figure at 50 ms is reported. In many cases, throughput will start to fall off at a response time below the 50 ms limit. In these cases, the tables in this chapter report the response time at the point of maximum throughput.

LADDIS Benchmark

The SPEC SFS 1 (0.97 LADDIS) benchmark is a synthetic NFS workload based on an application abstraction, an NFS operation mix, and an NFS operation request rate. The workload generated by the benchmark emulates an intensive software development environment at the NFS protocol level. The LADDIS benchmark makes direct RPC calls to the server, eliminating any variation in NFS client implementation. This makes it easier to control the operation mix and workload, especially for comparing results between vendors. However, this also hides the benefits of special client implementations, such as the cache file system client implemented in the Solaris 2.3 and later software environments.

TABLE A-3 shows the NFS operations mix. These percentages indicate the relative number of calls made to each operation.

TABLE A-3

NFS Operation	Percent Mix
Lookup	34
Read	22
Write	15
GetAttr	13
ReadLink	8
ReadDir	3
Create	2
Remove	1
Statfs	1
SetAttr	1

The LADDIS benchmark for NFS file systems uses an operation mix that is 15 percent write operations. If your NFS clients generate only one to two percent write operations, LADDIS underestimates your performance. The greater the similarity between the operation mixes, the more reliable SPECnfs_A93 is as a reference.

Running the benchmark requires the server being benchmarked to have at least two NFS clients (the NFS load generators), and one or more isolated networks. The ability to support multiple networks is important because a single network may become saturated before the server maximum performance point is reached. One client is designated as the LADDIS Prime Load Generator. The Prime Load Generator controls the execution of the LADDIS load generating code on all load-generating clients. It typically controls the benchmark. In this capacity, it is responsible for collecting throughput and response time data at each of the workload points and for generating results.

To improve response time, configure your NFS server with the NVRAM-NVSIMM Prestoserve NFS accelerator. NVSIMMs provide storage directly in the high-speed memory subsystem. Using NVSIMMs results in considerably lower latency and reduces the number of disks required to attain a given level of performance.

Since there are extra data copies in and out of the NVSIMM, the ultimate peak throughput is reduced. Because NFS loads rarely sustain peak throughput, the better response time using the NVSIMMs is preferable. For information on the Prestoserve NFS accelerator, see "Prestoserve NFS Accelerator" in Chapter 4..

Interpreting LADDIS Results

Consider throughput and response time when analyzing LADDIS results. Although response time increases with throughput, a well-designed NFS server delivers a range of throughput results while maintaining a more or less constant response time.

FIGURE A-1 illustrates an idealized LADDIS baseline result. At point A, the system has minimal load. Response time is limited by the hardware characteristics of the I/O subsystem. At point D, the LADDIS benchmark reports the SPECnfs_A93 single figure of merit. Between points B and C, workload increases about five times without impacting client performance.

The more a NFS server LADDIS benchmark of average NFS response time versus NFS throughput resembles FIGURE A-1, the better the NFS server can meet the average performance requirements to handle burst activity.

FIGURE A-1 Idealized LADDIS Baseline Result, Case 1

FIGURE A-2 shows a second baseline case of average NFS response time vs. NFS throughput. Point C in FIGURE A-1 shows a response time of 20 milliseconds. However, point C in FIGURE A-2 shows a response time of 35 milliseconds.

Compared to case 1 (FIGURE A-1), a case 2 client (FIGURE A-2) can experience an extra 15 second delay for each 1000 NFS operations transacted on an NFS server throughput rate of 1000 NFS ops. For example, when executing a long listing of the ls command (ls -l) on a NFS-mounted file system, the system can easily request over 1000 NFS operations. Both cases report the same LADDIS result at point D.

FIGURE A-2 Idealized LADDIS Baseline Result, Case 2