A.1 NFS Monitoring Tools

Table A-1 describes the tools that can be used to monitor NFS operations and performance.

Table A-1

Tool	Function
iostat	Reports I/O statistics, among them disk I/O activity.
nfsstat	Reports NFS statistics: NFS and RPC (Remote Procedure Call) interfaces to the kernel. Can also be used to initialize the statistical information.
nfswatch	Shows NFS transactions classified by file system. nfswatch is a public domain tool with source code available on uunet.uu.net under the directory usr/spool/ftp/networking. The file name is nfswatch3.0.tar.Z.
sar	Reports system activity, such as CPU utilization, buffer activity, disk and tape device activity.
SharpShooter (AIM Technology)	Pinpoints bottlenecks, balances NFS load across clients and servers. Shows effect of distributed applications and balances network traffic across servers. Accounts for disk usage by user or group.
vmstat	Reports virtual memory statistics, among them, disk activity.

A.2 Network Monitoring Tools

Table A-2 describes the tools that can be used to monitor network performance as it relates to NFS.

Table A-2

Tool	Function
snoop	Displays information about specified packets on Ethernet.
netstat	Displays the contents of network-related data structures.
ping	Sends ICMP ECHO_REQUEST packets to network hosts.
NetMetrix Load Monitor	Allows network load monitoring real time or over time, and characterization of load in terms of time, source, destination, protocol and packet size.
SunNet Manager	Performs network device monitoring and troubleshooting. SunNet Manager is a management and monitoring tool.
LAN analyzers: Network General Sniffer, Novell/Excelan Lanalyzer	Performs packet analysis.

A.2.1 Snoop

The snoop command is part of the Solaris 2.x software environment. The snoop command must run by root (#) to capture packets in promiscuous mode. In order to capture packets in non-promiscuous mode or to analyze packets from a capture file, you do not need to be root.

In promiscuous mode, the interface turns off its filter and lets you see all packets on the subnet, whether or not they are addressed to your system. You can passively observe other packets not destined for your system. Promiscuous mode is limited to root.

Using the snoop command turns a Sun system into a network sniffer, able to detect network problems. The snoop command also captures a certain number of network packets, allows you to trace the calls from each client to each server, and displays the contents of the packets. You can also save the contents of the packets to a file to inspect later on.

Some of the features of the snoop command are:

• Logs or displays packets selectively
• Provides accurate timestamps for checking network RPC (for example, NIS) response time
• Formats packets and protocol information in a user-friendly manner

The snoop command can display packets in a single-line summary or in expanded form. In summary form, only the data pertaining to the highest level protocol is displayed. For example, an NFS packet will have only NFS information displayed. The underlying RPC (Remote Procedure Call), UDP (User Datagram Protocol), IP (Internet Protocol), and network frame information is suppressed, but can be displayed if you choose either of the verbose (-v or -V) options.

The snoop command uses both the packet filter and buffer modules of the Data Link Provider Interface (DLPI) to provide efficient capture of packets transmitted to or received from the network. To view or capture all traffic between any two systems, run snoop on a third system.

The snoop command is a useful tool if you are considering subnetting, since it is a packet analysis tool. The output of the snoop command can be used to drive scripts that accumulate load statistics. The program is capable of breaking the packet header out in order to debug it, and to see what might be the source of incompatibility difficulties.

A.2.1.1 Snoop Examples

Following are some examples of how to use snoop.

· To look at selected packets in a capture file, pkts

The statistics show which client is making a read request, and the left-hand column shows the time in seconds with a resolution of about 4 microseconds. Calls and corresponding responses are shown by the arrows, which do not appear on the actual screen.

When a read or write request is made, be sure the server doesn't time-out. If it does, the client has to send again, and the client's IP code will break up the write block into smaller UDP blocks. The default write time is .07 seconds. The time-out factor is a tunable parameter in the mount command.

-i pkts..Displays packets previously captured in the file, pkts.

-p99,108 Selects packets 99 through 108 to be displayed from a capture file. The first number 99, is the first packet to be captured. The last number, 108, is the last packet to be captured. The first packet in a capture file is packet 1.

· To obtain more detailed information on a packet

# snoop -i pkts -v 101 #

The command snoop -i pkts -v 101 obtains more detailed information on packet 101.

-i pkts	Displays packets previously captured in the file, pkts.
-v	Verbose mode. Prints packet headers in detail for packet 101. Use this option only when you need information on selected packets.

The output of the snoop -i pkts -v 101 command follows showing the Ethernet, IP, UDP, RPC, and NFS headers captured in snoop.

ETHER: ----- Ether Header ----- ETHER: ETHER: Packet 101 arrived at 16:09:53.59 ETHER: Packet size = 210 bytes ETHER: Destination = 8:0:20:1:3d:94, Sun ETHER: Source = 8:0:69:1:5f:e, Silicon Graphics ETHER: Ethertype = 0800 (IP) ETHER: IP: ----- IP Header ----- IP: IP: Version = 4, header length = 20 bytes IP: Type of service = 00 IP: ..0. .... = routine IP: ...0 .... = normal delay IP: .... 0... = normal throughput IP: .... .0.. = normal reliability IP: Total length = 196 bytes IP: Identification 19846 IP: Flags = 0X IP: .0.. .... = may fragment IP: ..0. .... = more fragments IP: Fragment offset = 0 bytes IP: Time to live = 255 seconds/hops IP: Protocol = 17 (UDP) IP: Header checksum = 18DC IP: Source address = 129.144.40.222, boutique IP: Destination address = 129.144.40.200, sunroof IP: UDP: ----- UDP Header ----- UDP: UDP: Source port = 1023 UDP: Destination port = 2049 (Sun RPC) UDP: Length = 176 UDP: Checksum = 0 UDP: RPC: ----- SUN RPC Header ----- RPC: RPC: Transaction id = 665905 RPC: Type = 0 (Call) RPC: RPC version = 2 RPC: Program = 100003 (NFS), version = 2, procedure = 1 RPC: Credentials: Flavor = 1 (Unix), len = 32 bytes

RPC: Time = 06-Mar-90 07:26:58 RPC: Hostname = boutique RPC: Uid = 0, Gid = 1 RPC: Groups = 1 RPC: Verifier : Flavor = 0 (None), len = 0 bytes RPC: NFS: ----- SUN NFS ----- NFS: NFS: Proc = 11 (Rename) NFS: File handle = 000016430000000100080000305A1C47 NFS: 597A0000000800002046314AFC450000 NFS: File name = MTra00192 NFS: File handle = 000016430000000100080000305A1C47 NFS: 597A0000000800002046314AFC450000 NFS: File name = .nfs08 NFS:

· To view NFS packets

This example views the NFS packets between the systems sunroof and boutique. Calls and corresponding responses are shown by arrows which do not appear on the actual screen.

-i pkts..Displays packets previously captured in the file, pkts.

rpc nfs..Displays packets for an RPC call or reply packet for the NFS protocol. The option following nfs is the name of an RPC protocol from /etc/rpc or a program number.

and            Performs a logical and operation between two boolean values. 
               For example, sunroof boutique is the same as sunroof 
               and boutique.

· To save packets to a new capture file

# snoop -i pkts -o pkts.nfs rpc nfs sunroof boutique #

-i pkts..Displays packets previously captured in the file, pkts.

-o pkts.nfs Saves the displayed packets in the output file, pkts.nfs.

rpc nfs..Displays packets for an RPC call or reply packet for the NFS protocol. The option following nfs is the name of an RPC protocol from /etc/rpc or a program number.

See the snoop man page for additional details on options used with the snoop command and additional information about using snoop.

A.3 LADDIS

SPEC SFS 1 (0.97 LADDIS) is a vendor-neutral NFS file server benchmark incorporated by SPEC as a member of the System-level File Server (SFS) Benchmark Suite. Two reference points are considered when reporting 097.LADDIS.

• NFS operation throughput The peak number of NFS operations the target server can complete in a given number of milliseconds. The larger the number of operations an NFS server can support, the more users it can serve.
• Response time The average time needed for an NFS client to receive a reply from a target server in response to an NFS request. The response time of an NFS server is the client's perception of how fast the server is.

LADDIS is designed so that its workload can be incrementally increased until the target server performance falls below a certain level. That point is defined as an average response time exceeding 50ms. This restriction is applied when deriving SPECnfs_A93, the maximum throughput in NFS operations per second for which the response time does not exceed 50 ms.

As long as throughput continues to increase with workload, the throughput figure at 50 ms is reported. In many cases, throughput will start to fall off at a response time below the 50 ms limit. In these cases, the tables in this chapter report the response time at the point of maximum throughput.

The SPEC SFS 1 (0.97 LADDIS) benchmark is a synthetic NFS workload based on an application abstraction, an NFS operation mix, and an NFS operation request rate. The workload generated by the benchmark emulates an intensive software development environment at the NFS protocol level. The LADDIS benchmark makes direct RPC calls to the server, eliminating any variation in NFS client implementation. This makes it easier to control the operation mix and workload, especially for comparing results between vendors. However, this also hides the benefits of special client implementations, such as the cache file system client implemented in the Solaris 2.3 and 2.4 software environments.

Table A-3 summarizes the percentage mix of NFS operations. These percentages indicate the relative number of calls made to each operation.

Table A-3

NFS Operation	Percent Mix
Lookup		34
Read		22
Write		15
GetAttr		13
ReadLink		8
ReadDir		3
Create		2
Remove		1
Statfs		1
SetAttr		1

The LADDIS benchmark for NFS file systems uses an operation mix that is 15 percent write operations. If your NFS clients generate only one to two percent write operations, LADDIS underestimates your performance. The greater the similarity between the operation mixes, the more reliable SPECnfs_A93 will be as a reference.

To run the benchmark requires the server being benchmarked, at least two NFS clients (the NFS load generators), and one or more isolated networks. The ability to support multiple networks is important because a single network may saturate before the server maximum performance point is reached. One client is designated as the LADDIS Prime Load Generator. The prime load generator controls the execution of the LADDIS load generating code on all load generating clients. It typically controls the benchmark, also. In this capacity, it is responsible for collecting throughput and response time data at each of the workload points and for generating results.

To improve response time, you can configure your NFS server with the NVRAM-NVSIMM Prestoserve NFS accelerator. NVSIMMs provide storage directly in the high-speed memory subsystem. Using NVSIMMs results in considerably lower latency and reduces the number of disks required to attain a given level of performance.

Since there are extra data copies in and out of the NVSIMM, the ultimate peak throughput is reduced. Because NFS loads rarely sustain peak throughput, the better response time using the NVSIMMs is preferable. For information on the Prestoserve NFS accelerator, see Section 4.6, "Prestoserve NFS Accelerator."

A.3.1 Interpreting LADDIS Results

The two factors to consider when analyzing LADDIS results are throughput and response time. Although response time increases with throughput, a well-designed NFS server delivers a range of throughput results while maintaining a more or less constant response time.

Figure A-1 illustrates an idealized LADDIS baseline result. At point A, there is little load on the system. Response time is limited by the hardware characteristics of the I/O subsystem. At point D, the LADDIS benchmark reports the SPECnfs_A93 single figure of merit. As you can see, between points B and C, workload increases about five times without impacting client performance.

The closer a NFS server LADDIS benchmark graph of average NFS response time versus NFS throughput resembles Figure A-1, the better the NFS server can meet the average performance requirements and handle burst activity.

Figure A-2 shows a second baseline case of average NFS response time vs. NFS throughput. Point C in Figure A-1 shows a response time of 20 milliseconds. However, point C in Figure A-2 shows a response time of 35 milliseconds.

Compared to case 1 (Figure A-1), a case 2 client (Figure A-2) can experience an extra 15 second delay for each 1000 NFS operations transacted on an NFS server throughput rate of 1000 NFSops. For example, when executing a long listing of the ls command (ls -l) on a NFS mounted file system can easily request over 1000 NFS operations. Both cases report the same LADDIS result at point D.