System Crashes

System crashes can occur due to hardware malfunctions, power failures, I/O problems, and software errors. If a software glitch, such as a fatal kernel error caused by an operating system bug, causes a system to crash, the system writes an image of its physical memory into a core file at the end of the swap slice of the disk. This file is a snapshot of the state of the kernel, including its program text, data, and control structures, captured at the time of the crash.

Crash Dump Files

The core file written when a UNIX system crashes can provide clues as to what caused the crash if it is examined by an experienced customer service representative who is familiar with the crash kernel debugger. However, when a UNIX system reboots after a crash, it generally overwrites any core file that may have been produced--unless you have enabled the system to save the core file in a crash dump file.

Because UNIX systems do not automatically save core files in crash dump files, if you want to save a system's crash dump files for later inspection, you must set up the system by creating a crash directory in /var/crash, reserving disk space, and editing /etc/init.d/sysetup to store crash dump files. Crash dump files can be very big, so do not retain them longer than necessary.

See Chapter 64, "Saving Crash Dumps," for instructions on how to enable a system to save crash dump files.

Message Files

After Solaris software installation, the error logging daemon, syslogd, automatically records various system warnings and errors in message files. By default, many of these system messages are stored in /var/adm (or /usr/adm) or displayed on the system console. You can direct where these messages are stored by setting up system logging. See "How to Customize System Logging" on page 1293 for more information. These messages can alert you to system problems, such as a device that is about to fail.

Because /var/adm stores large files containing messages, crash dumps, and other data, this directory can consume lots of disk space. To keep the /var/adm directory from growing too large, and to ensure that future crash

dumps can be saved, you should remove unneeded files periodically. You can automate this task by using crontab. See "How to Delete Crash Dump Files" on page 1311 and "Scheduling System Events" on page 1343 for more information on automating this task.

What to Do After a Crash

If a system crashes, making it run again may seem like your most pressing concern. However, before you reboot the system, examine the console screen for messages. These messages may provide some insight as to what caused the crash. Even if the system reboots automatically and the console messages have disappeared from the screen, you may be able to check these messages by viewing the system error log file that is generated automatically in /var/adm/messages (or /usr/adm/messages). See "How to View Crash and Boot Messages" on page 1288 for more information about viewing system error log files.

If you are having frequent crashes and are unable to determine their cause, before calling for help, gather all the information you can from the system console or the /var/adm/messages files, and have it ready for a customer service representative to examine.

What to Do If Rebooting Fails

After a crash, the system may reboot automatically. If the automatic reboot fails with a message such as:

reboot failed: help

then run fsck in single-user mode. For more information, see "Managing File Systems" in System Administration Guide, Volume I, and the fsck(1M) man page.

If the system does not reboot, or if it reboots and then crashes again, there may be a hardware problem with a disk or one of the boards.

Check your hardware connections:

• Make sure the equipment is plugged in.
• Make sure all the switches are in the proper settings and pushed all the way in.
• Look at all the connectors and cables, including the Ethernet cables.
• If all this fails, turn off the power to the system, wait 10 to 20 seconds, and then turn on the power again.

If you cannot find any obvious fault with the connections, and the system still doesn't respond, contact your local service provider.

What to Do If a System Hangs

Your system may freeze or hang rather than crash completely. If this is the case, use this checklist:

• Make sure the pointer is in the window where you are typing the commands.
• Press Control-q in case the user accidently pressed Control-s, which freezes the screen. Note that, in a windowing environment, Control-s freezes only the window, not the entire screen. If a window is frozen, try using another window.
• Press Control-\ to force a "quit" in the running program and (probably) write out a core file.
• Press Control-c to interrupt the program that may be running.
• If possible, log onto the system from another terminal or log in remotely from another system on the network. Type ps -ef and look for the hung process. If it looks like the window system is hung, find the process and kill it.
• Try becoming root and rebooting the system.
• If the system still does not respond, force a crash dump and reboot. See "Shutting Down and Booting a System" in System Administration Guide, Volume I for information on forcing a crash dump and booting.
• If the system still does not respond, turn the power off, wait a minute or so, then turn the power back on. This procedure is frequently called power cycling.

• If you cannot get the system to respond at all, contact your local service provider for help.

Quotas

Quotas enable system administrators to control the size of UFS file systems by limiting the amount of disk space and the number of inodes (which roughly corresponds to the number of files) that individual users can acquire. For this reason, quotas are especially useful on the file systems where user home directories reside. (As a rule, public and /tmp file systems probably wouldn't benefit as much from the establishment of quotas.)

Setting up quotas invovles several general steps:

1. A series of commands prepares a file system to accept quotas, ensuring that quotas will be enforced each time the system is rebooted and the file system is mounted. Entries must be added to the /etc/vfstab file, and a quotas file must be created in the top-level directory of the file system.

2. After a quota is created for one user, it can be copied as a prototype to set up other user quotas.

3. Before quotas are actually turned on, another command checks for consistency by comparing the proposed quotas to the current disk usage to make sure that there are no conflicts.

4. Finally, a command turns the quotas on for one or more entire file systems.

These steps ensure that quotas are automatically activated on a file system each time it is mounted. For specific information about these procedures, see "Setting Up Quotas" on page 1314.

Once they are in place, quotas can be changed to adjust the amount of disk space or number of inodes that users can consume. Additionally, quotas can be added or removed as system needs change. See "Changing and Removing Quotas" on page 1323 for procedures that describe how to change quotas or the amount of time that quotas can be exceeded, disable individual quotas, or remove quotas from file systems.

In addition, quota status can be monitored. Quota commands enable administrators to display information about quotas on a file system, or search for users who have exceeded their quotas. For procedures that describe how to use these commands, see "Checking Quotas" on page 1321.

Accounting Utilities

The SunOS 5.x accounting utilities are programs that collect and record data about system usage and then provide reports. The accounting utilities can be used for:

• Monitoring system usage
• Troubleshooting
• Locating and correcting performance problems
• Maintaining system security

Once they have been set up, the system accounting facilities run mostly on their own. (For instructions on setting up accounting facilities, see Chapter 67, "Setting Up and Maintaining Accounting.")

The accounting utilities provide C language programs and shell scripts that organize data into summary files and reports. These programs reside in the /usr/adm/acct and /usr/lib/acct directories. Setting up automatic accounting involves putting the scripts into crontab files so that cron can invoke them automatically.

The following is an overview of how accounting works.

1. Between system startup and shutdown, raw data about system use (such as logins, processes run, and data storage) are collected in accounting files.

2. Periodically (usually once a day), the /usr/lib/acct/runacct program processes the various accounting files and produces both cumulative summary files and daily accounting reports. The daily reports are printed by the /usr/lib/acct/prdaily program.

3. Monthly, the administrator can process and print the cumulative summary files generated by runacct by executing the monacct program. The summary reports produced by monacct provide an efficient means for billing users on a monthly or other fiscal basis.

Types of Daily Accounting

Daily accounting can help you do four types of accounting: connect accounting, process accounting, disk accounting, and fee calculations.

Connect Accounting

Connect accounting enables you to determine the following:

• The length of time a user was logged in
• How the tty lines are being used
• The number of reboots on your system
• The frequency with which the accounting software was turned off and on

To provide this information, the system stores records of time adjustments, boot times, times the accounting software was turned off and on, changes in run levels, the creation of user processes (login processes and init processes), and the deaths of processes. These records (produced from the output of system programs such as date, init, login, ttymon, and acctwtmp) are stored in the file /var/adm/wtmp. Entries in the wtmp file may contain the following information: a user's login name, a device name, a process ID, the type of entry, and a time stamp denoting when the entry was made.

Process Accounting

Process accounting enables you to keep track of the following data about each process run on your system:

• The user and group IDs of those using the process
• The beginning and elapsed times of the process
• The CPU time for the process (user time and system time)
• The amount of memory used
• The commands run
• The tty controlling the process

Every time a process dies, the exit program collects this data and writes it to /var/adm/pacct.

The pacct file has a default maximum size of 500 blocks that is enforced by the accounting shell script, ckpacct (normally run as a cron job). If ckpacct finds that /var/adm/pacct is larger than 500 blocks, it moves the file to /var/adm/pacctn, where n is the next unused incremental number.

Disk Accounting

Disk accounting enables you to gather and format the following data about the files each user has on disks:

• The name and ID of the user
• The number of blocks used by the user's files

This data is collected by the shell script /usr/lib/acct/dodisk at intervals determined by the cron command you add to the /var/spool/cron/crontabs/root file. See "How to Set Up Accounting" on page 1336 for more information about setting up dodisk.

In turn, dodisk invokes the commands acctdusg and diskusg, which gather information for each file in the system.

acctdusg gathers all the disk accounting information. Each time it is invoked, this command can process a maximum of 3000 users.

---

Caution - Information gathered by running dodisk is stored in the /var/adm/acct/nite/disktacct file. This information is overwritten the next time dodisk is run. Therefore, avoid running dodisk twice in the same day.

---

diskusg may overcharge for files that are written in random access fashion, which may create holes in the files. This is because diskusg does not read the indirect blocks of a file when determining its size. Rather, diskusg determines the size of a file by looking at the di_size value of the inode.

Calculating User Fees

If you provide special user services on a request basis, such as restoring files and remote printing, you may want to bill users by running a facility called chargefee. For example, chargefee can be used to bill a user for restoring a directory from tape. Each time a specified user logs in, the chargefee utility records an entry consisting of the login name, user ID, and the set fee in the file /var/adm/fee. Each time the runacct utility is executed, new entries are picked up and merged into the total accounting records. For instructions on setting up chargefee to bill users, see "How to Bill Users" on page 1338.

How Daily Accounting Works

Here is a step-by-step summary of how SunOS daily accounting works:

1. When the system is switched into multiuser mode, the /usr/lib/acct/startup program is executed. The startup program executes several other programs that invoke accounting.

2. The acctwtmp program adds a "boot" record to /var/adm/wtmp. In this record, the system name is shown as the login name in the wtmp record. Table 62-1 presents a summary of how the raw accounting data is gathered and where it is stored.

Table 62-1

File in /var/adm	Information	Written By	Format
wtmp	Connect sessions	login, init	utmp.h
	Changes	date
	Reboots	acctwtmp
	Shutdowns	shutacct shell
pacctn	Processes	Kernel (when the process ends) turnacct switch (creates a new file when the old one reaches 500 blocks)	acct.h
fee	Special charges	chargefee	acct.h
acct/nite/disktacct	Disk space used	dodisk	tacct.h

3. The turnacct program, invoked with the on option, begins process accounting. Specifically, turnacct executes the accton program with the argument /var/adm/pacct.

4. The remove shell script "cleans up" the saved pacct and wtmp files left in the sum directory by runacct.

5. The login and init programs record connect sessions by writing records into /var/adm/wtmp. Any date changes (using date with an argument) are also written to /var/adm/wtmp. Reboots and shutdowns using acctwtmp are also rec'orded in /var/adm/wtmp.

6. When a process ends, the kernel writes one record per process, in the form of acct.h, in the /var/adm/pacct file.

7. runacct is executed by cron each night. runacct processes the accounting files: /var/adm/pacctn, /var/adm/wtmp, /var/adm/fee, and /var/adm/acct/nite/disktacct, to produce command summaries and usage summaries by login.

8. The /usr/lib/acct/prdaily program is executed on a daily basis by runacct to write the daily accounting information collected by runacct (in ASCII format) in /var/adm/acct/sum/rprt.MMDD.

9. The monacct program should be executed on a monthly basis (or at intervals determined by you, such as the end of every fiscal period). The monacct program creates a report based on data stored in the sum directory that has been updated daily by runacct. After creating the report, monacct "cleans up" the sum directory to prepare the directory's files for the new runacct data.

Two programs track disk usage by login: acctdusg and diskusg. They are invoked by the shell script dodisk.

Every hour, cron executes the ckpacct program to check the size of /var/adm/pacct. If the file grows past 500 blocks (default), the turnacct switch is executed. (The program moves the pacct file and creates a new one.) The advantage of having several smaller pacct files becomes apparent when trying to restart runacct if a failure occurs when processing these records.

If the system is shut down using shutdown, the shutacct program is executed automatically. The shutacct program writes a reason record into /var/adm/wtmp and turns off process accounting.

Daily Accounting Reports

The runacct shell script generates four basic reports upon each invocation. These reports cover the areas of connect accounting, usage by login on a daily basis, command usage reported by daily and monthly totals, and a report of the last time users were logged in. The four basic reports generated are:

• Daily Report - Shows line utilization by tty number.
• Daily Usage Report - Indicates usage of system resources by users (listed in order of UID).

• Daily Command Summary - Indicates usage of system resources by commands, listed in descending order of use of memory (in other words, the command that used the most memory is listed first). This same information is reported for the month with the monthly command summary.
• Last Login - Shows the last time each user logged in (arranged in chronological order).

The following sections describe the reports and the meaning of the data presented in each one.

Daily Report

This report gives information about each terminal line used. A sample daily report appears below.

Jun 26 09:53 1994 DAILY REPORT FOR sfxbs Page 1 from Thu Jun 25 17:45:22 1994 to Fri Jun 26 09:51:25 1994 1 runacct 1 acctcon TOTAL DURATION IS 966 MINUTES LINE MINUTES PERCENT # SESS # ON # OFF term/23 25 3 7 7 3 term/22 157 16 6 6 3 TOTALS 183 -- 13 13 7 ------------------------------------------------------

The from and to lines specify the time period reflected in the report--the period from the time the last accounting report was generated until the time the current accounting report was generated. It is followed by a log of system reboots, shutdowns, power failure recoveries, and any other record dumped into /var/adm/wtmp by the acctwtmp program. For more, see acct(1M).

The second part of the report is a breakdown of line utilization. The TOTAL DURATION tells how long the system was in multiuser state (accessible through the terminal lines). The columns are described in Table 62-2.

Table 62-2

Column	Description
LINE	The terminal line or access port.
MINUTES	The total number of minutes that the line was in use during the accounting period.
PERCENT	The total number of MINUTES the line was in use, divided into the TOTAL DURATION.
# SESS	The number of times this port was accessed for a login session.
# ON	Identical to SESS. (This column does not have much meaning anymore. It used to list the number of times that a port was used to log in a user.)
# OFF	This column reflects the number of times a user logs out and any interrupts that occur on that line. Generally, interrupts occur on a port when ttymon is first invoked when the system is brought to multiuser state. If the # OFF exceeds the # ON by a large factor, the multiplexer, modem, or cable is probably going bad, or there is a bad connection somewhere. The most common cause of this is an unconnected cable dangling from the multiplexer.

During real time, you should monitor /var/adm/wtmp because it is the file from which the connect accounting is geared. If the wtmp file grows rapidly, execute acctcon -l file < /var/adm/wtmp to see which tty line is the noisiest. If the interrupting is occurring frequently, general system performance will be affected. Additionally, wtmp may become corrupted. To correct this, see "How to Fix a wtmp File" on page 1339.

Daily Usage Report

The daily usage report gives a breakdown of system resource utilization by user. A sample of this type of report appears below.

Jun 29 09:53 1994 DAILY USAGE REPORT FOR sfxbs Page 1 LOGIN CPU (MINS) KCORE-MINS CONNECT(MINS) DISK # OF # OF # DISK FEE UID NAME PRIME NPRIME PRIME NPRIME PRIME NPRIME BLOCKS PROCS SESS SAMPLES 0 TOTAL 5 12 6 16 131 51 0 1114 13 0 0 0 root 2 8 1 11 0 0 0 519 0 0 0 3 sys 0 1 0 1 0 0 0 45 0 0 0 4 adm 0 2 0 1 0 0 0 213 0 0 0 5 uucp 0 0 0 0 0 0 0 53 0 0 0 999 rly 3 1 5 2 111 37 0 269 1 0 0 7987 jan 0 0 0 1 20 14 0 15 6 0 0

The data provided in the daily usage report is described in Table 62-3.

Table 62-3

Column	Description
UID	User identification number.
LOGIN NAME	Login name of the user. Identifies a user who has multiple login names.
CPU-MINS	Amount of time, in minutes, that the user's process used the central processing unit. Divided into PRIME and NPRIME (non-prime) utilization. The accounting system's version of this data is located in the file /etc/acct/holidays.
KCORE-MINS	A cumulative measure of the amount of memory in kilobyte segments per minute that a process uses while running. Divided into PRIME and NPRIME utilization.
CONNECT-MINS	Amount of time a user was logged into the system, or "real time." Divided into PRIME and NPRIME use. If these numbers are high while the # OF PROCS is low, you can conclude that the user logs in first thing in the morning and hardly touches the terminal the rest of the day.

Table 62-3 (Continued)

Column	Description
DISK BLOCKS	Output from the acctdusg program, which runs and merges disk accounting programs and total accounting record (daytacct). (For accounting purposes, a block is 512 bytes.)
# OF PROCS	Number of processes invoked by the user. If large numbers appear, a user may have a shell procedure that has run out of control.
# OF SESS	Number of times a user logged on to the system.
# DISK SAMPLES	Number of times disk accounting was run to obtain the average number of DISK BLOCKS.
FEE	Often unused field that represents the total accumulation of units charged against the user by chargefee.

Daily Command Summary

The daily command summary report shows the system resource use by command. With this report, you can identify the most heavily used commands and, based on how those commands use system resources, gain insight on how best to tune the system. The format of the daily and monthly reports are virtually the same; however, the daily summary reports only on the current accounting period while the monthly summary reports on the start of the fiscal period to the current date. In other words, the monthly report reflects the data accumulated since the last invocation of monacct.

These reports are sorted by TOTAL KCOREMIN, which is an arbitrary gauge but often a good one for calculating drain on a system.

A sample daily command summary appears below.

Jun 29 09:52 1994 DAILY COMMAND SUMMARY Page 1 TOTAL COMMAND SUMMARY PRIME PRIME PRIME COMMAND NUMBER TOTAL TOTAL TOTAL MEAN MEAN HOG CHARS BLOCKS NAME CMDS KCOREMIN CPU-MIN REAL-MIN SIZE-K CPU-MIN FACTOR TRNSFD READ TOTALS 1114 2.44 16.69 136.33 0.15 0.01 0.12 4541666 1926 sh 227 1.01 2.45 54.99 0.41 0.01 0.04 111025 173 vi 12 0.35 0.62 44.23 0.55 0.05 0.01 151448 60 sed 143 0.09 0.82 1.48 0.10 0.01 0.55 14505 35 sadc 13 0.08 0.19 1.45 0.44 0.01 0.13 829088 19 more 3 0.04 0.07 2.17 0.59 0.02 0.03 30560 1 cut 14 0.03 0.09 0.28 0.37 0.01 0.33 154 13 uudemon. 76 0.03 0.66 2.30 0.05 0.01 0.29 43661 13 uuxqt 29 0.03 0.30 0.72 0.08 0.01 0.42 80765 35 mail 4 0.02 0.06 0.09 0.37 0.01 0.60 4540 9 ckstr 21 0.02 0.11 0.13 0.17 0.01 0.85 0 4 awk 13 0.02 0.12 0.21 0.15 0.01 0.54 444 2 ps 2 0.02 0.10 0.13 0.17 0.05 0.77 8060 21 find 9 0.02 3.35 5.73 0.00 0.37 0.58 355269 760 sar 1 0.01 0.19 0.24 0.08 0.19 0.80 564224 4 acctdisk 2 0.01 0.01 0.06 1.02 0.01 0.22 0 9 mv 24 0.01 0.14 0.17 0.10 0.01 0.81 3024 36 . . .

The data provided, by column, in the daily command summary is described in Table 62-4.

Table 62-4

Column	Description
COMMAND NAME	Name of the command. Unfortunately, all shell procedures are lumped together under the name sh because only object modules are reported by the process accounting system. It's a good idea to monitor the frequency of programs called a.out or core or any other unexpected name. acctcom can be used to determine who executed an oddly named command and if root privileges were used.
PRIME NUMBER CMNDS	Total number of invocations of this particular command during prime time.
TOTAL KCOREMIN	Total cumulative measurement of the Kbyte segments of memory used by a process per minute of run time.
PRIME TOTAL CPU-MIN:	Total processing time this program has accumulated during prime time.
PRIME TOTAL REAL-MIN	Total real-time (wall-clock) minutes this program has accumulated.
MEAN SIZE-K	Mean of the TOTAL KCOREMIN over the number of invocations reflected by NUMBER CMDS.
MEAN CPU-MIN	Mean derived between the NUMBER CMDS and TOTAL CPU-MIN.
HOG FACTOR	Total CPU time divided by elapsed time. Shows the ratio of system availability to system use, providing a relative measure of total available CPU time consumed by the process during its execution.
CHARS TRNSFD	Total count of the number of characters pushed around by the read and write system calls. May be negative due to overflow.
BLOCKS READ	Total count of the physical block reads and writes that a process performed.

Monthly Command Summary

The monthly command summary is similar to the daily command summary. The only difference is that the monthly command summary shows totals accumulated since the last invocation of monacct. A sample report appears below.

TOTAL COMMAND SUMMARY COMMAND NUMBER TOTAL TOTAL TOTAL MEAN MEAN HOG CHARS BLOCKS NAME CMDS KCOREMIN CPU-MIN REAL-MIN SIZE-K CPUMIN FACTOR TRNSFD READ TOTALS 301314 300607.70 4301.59 703979.81 69.88 0.01 0.01 6967631360 10596385 troff 480 58171.37 616.15 1551.26 94.41 1.28 0.40 650669248 194926 rnews 5143 29845.12 312.20 1196.93 95.59 0.06 0.26 1722128384 2375741 uucico 2710 16625.01 212.95 52619.21 78.07 0.08 0.00 228750872 475343 nroff 1613 15463.20 206.54 986.06 74.87 0.13 0.21 377563304 277957 vi 3040 14641.63 157.77 14700.13 92.80 0.05 0.01 116621132 206025 expire 14 13424.81 104.90 265.67 127.98 7.49 0.39 76292096 145456 comp 3483 12140.64 60.22 423.54 201.62 0.02 0.14 9584838 372601 ad_d 71 10179.20 50.02 1158.31 203.52 0.70 0.04 11385054 19489 as 2312 9221.59 44.40 285.52 207.68 0.02 0.16 35988945 221113 gone 474 8723.46 219.93 12099.01 39.67 0.46 0.02 10657346 19397 i10 299 8372.60 44.45 454.21 188.34 0.15 0.10 60169932 78664 find 760 8310.97 196.91 728.39 42.21 0.26 0.27 58966910 710074 ld 2288 8232.84 61.19 425.57 134.55 0.03 0.14 228701168 279530 fgrep 832 7585.34 62.62 199.11 121.14 0.08 0.31 22119268 37196 sh 56314 7538.40 337.60 291655.70 22.33 0.01 0.00 93262128 612892 du 624 5049.58 126.32 217.59 39.97 0.20 0.58 16096269 215297 ls 12690 4765.60 75.71 541.53 62.95 0.01 0.14 65759473 207920 vnews 52 4235.71 28.11 959.74 150.70 0.54 0.03 28291679 28285 . . .

See "Daily Command Summary" on page 1256 for a description of the data.

Last Login Report

This report gives the date when a particular login was last used. You can use this information to find unused logins and login directories that may be archived and deleted. A sample report appears below.

Feb 13 04:40 1994 LAST LOGIN Page 1 00-00-00 **rje** 88-01-01 jlr 88-02-09 cec42 88-02-13 cec20 00-00-00 **rje** 88-01-13 crom 88-02-10 jgd 88-02-13 cec22 00-00-00 3bnet 88-01-14 usg 88-02-10 wbr 88-02-13 cec23 00-00-00 adm 88-01-17 cec11 88-02-11 cec30 88-02-13 cec24 00-00-00 daemon 88-01-17 cec38 88-02-11 cec41 88-02-13 cec25 00-00-00 notes 88-01-17 cec40 88-02-11 cec43 88-02-13 cec26 00-00-00 oas 88-01-18 cec60 88-02-11 cec53 88-02-13 cec27 00-00-00 pds 88-01-19 cec35 88-02-11 cec54 88-02-13 cec3 00-00-00 polaris 88-01-19 cec37 88-02-11 cec55 88-02-13 cec31 00-00-00 rje 88-01-22 dmk 88-02-11 cec56 88-02-13 cec32 00-00-00 shqer 88-01-26 ask 88-02-11 cec57 88-02-13 cec4 00-00-00 sys 88-01-26 cec39 88-02-11 cec58 88-02-13 cec6 00-00-00 trouble 88-01-27 sync 88-02-11 jwg 88-02-13 cec7 00-00-00 usors 88-02-02 pkl 88-02-11 skt 88-02-13 cec8 00-00-00 uucp 88-02-03 ibm 88-02-11 tfm 88-02-13 commlp 00-00-00 wna 88-02-03 slk 88-02-12 cec21 88-02-13 djs 87-07-06 lp 88-02-04 cec59 88-02-12 cec28 88-02-13 epic 87-07-30 dgn 88-02-05 cec33 88-02-12 cec29 88-02-13 jab 87-08-19 blg 88-02-05 cec34 88-02-12 csp 88-02-13 jcs 87-12-08 emna 88-02-05 cec36 88-02-12 drc 88-02-13 mak 88-01-14 s 88-02-05 cec51 88-02-12 emw 88-02-13 dn 88-01-09 rib 88-02-05 dfh 88-02-12 je 88-02-13 mlp 88-01-25 dmf 88-02-05 fsh 88-02-12 kab 88-02-13 nbh 88-01-25 emda 88-02-05 pkw 88-02-12 rap 88-02-13 rah . . .

Looking at the pacct File With acctcom

At any time, you can examine the contents of the /var/adm/pacctn files, or any file with records in the acct.h format, by using the acctcom program. If you don't specify any files and don't provide any standard input when you run this command, acctcom reads the pacct file. Each record read by

acctcom represents information about a dead process (active processes may be examined by running the ps command). The default output of acctcom provides the following information:

• Command name (# sign if it was executed with root privileges)
• User
• tty name (listed as ? if unknown)
• Starting time
• Ending time
• Real time (in seconds)
• CPU time (in seconds)
• Mean size (in Kbytes)

The following information can be obtained by using options to acctcom:

• State of the fork/exec flag (1 for fork without exec)
• System exit status
• Hog factor
• Total kcore minutes
• CPU factor
• Characters transferred
• Blocks read

Table 62-5 describes the acctcom options.

Table 62-5 acctcom

Option	Description
-a	Show some average statistics about the processes selected. (The statistics are printed after the output is recorded.)
-b	Read the files backward, showing latest commands first. (This has no effect if reading standard input.)
-f	Print the fork/exec flag and system exit status columns. (The output is an octal number.)
-h	Instead of mean memory size, show the hog factor, which is the fraction of total available CPU time consumed by the process during its execution. Hog factor = total_CPU_time/elapsed_time.
-i	Print columns containing the I/O counts in the output.
-k	Show total kcore minutes instead of memory size.
-m	Show mean core size (this is the default).

Table 62-5 acctcom(Continued)

Option	Description
-q	Don't print output records, just print average statistics.
-r	Show CPU factor: user_time/(system_time + user_time).
-t	Show separate system and user CPU times.
-v	Exclude column headings from the output.
-C sec	Show only processes with total CPU time (system plus user) exceeding sec seconds.
-e time	Show processes existing at or before time, given in the format hr[:min[:sec]].
-E time	Show processes starting at or before time, given in the format hr[:min[:sec]]. Using the same time for both -S and -E shows processes that existed at the time.
-g group	Show only processes belonging to group.
-H factor	Show only processes that exceed factor, where factor is the "hog factor" (see the -h option).
-I chars	Show only processes transferring more characters than the cutoff number specified by chars.
-l line	Show only processes belonging to the terminal /dev/line.
-n pattern	Show only commands matching pattern (a regular expression except that "+" means one or more occurrences).
-o ofile	Instead of printing the records, copy them in acct.h format to ofile.
-O sec	Show only processes with CPU system time exceeding sec seconds.
-s time	Show processes existing at or after time, given in the format hr[:min[:sec]].
-S time	Show processes starting at or after time, given in the format hr[:min[:sec]].
-u user	Show only processes belonging to user.

Accounting Files

The /var/adm directory structure contains the active data collection files and is owned by the adm login (currently user ID of 4).

Table 62-6 /var/adm

File	Description
dtmp	Output from the acctdusg program
fee	Output from the chargefee program, ASCII tacct records
pacct	Active process accounting file
pacctn	Process accounting files switched using turnacct
Spacctn.MMDD	Process accounting files for MMDD during execution of runacct

The /var/adm/acct directory contains the nite, sum, and fiscal directories, which contain the actual data collection files. For example, the nite directory contains files that are reused daily by the runacct procedure. A brief summary of the files in the /var/adm/acct/nite directory follows.

Table 62-7 /var/adm/acct/nite

File	Description
active	Used by runacct to record progress and print warning and error messages
active.MMDD	Same as active after runacct detects an error
cms	ASCII total command summary used by prdaily
ctacct.MMDD	Connect accounting records in tacct.h format
ctmp	Output of acctcon1 program, connect session records in ctmp.h format (acctcon1 and acctcon2 are provided for compatibility purposes)
daycms	ASCII daily command summary used by prdaily
daytacct	Total accounting records for one day in tacct.h format
disktacct	Disk accounting records in tacct.h format, created by the dodisk procedure
fd2log	Diagnostic output during execution of runacct

Table 62-7 /var/adm/acct/nite(Continued)

File	Description
lastdate	Last day runacct executed (in date +%m%d format)
lock	Used to control serial use of runacct
lineuse	tty line usage report used by prdaily
log	Diagnostic output from acctcon
log.MMDD	Same as log after runacct detects an error
owtmp	Previous day's wtmp file
reboots	Beginning and ending dates from wtmp and a listing of reboots
statefile	Used to record current state during execution of runacct
tmpwtmp	wtmp file corrected by wtmpfix
wtmperror	Place for wtmpfix error messages
wtmperror.MMDD	Same as wtmperror after runacct detects an error
wtmp.MMDD	runacct's copy of the wtmp file

The sum directory contains the cumulative summary files updated by runacct and used by monacct. A brief summary of the files in the /var/adm/acct/sum directory is in Table 62-8.

Table 62-8 /var/adm/acct/sum

File	Description
cms	Total command summary file for current fiscal period in internal summary format
cmsprev	Command summary file without latest update
daycms	Command summary file for the day's usage in internal summary format
loginlog	Record of last date each user logged on; created by lastlogin and used in the prdaily program
rprt.MMDD	Saved output of prdaily program

Table 62-8 /var/adm/acct/sum(Continued)

File	Description
tacct	Cumulative total accounting file for current fiscal period
tacctprev	Same as tacct without latest update
tacct.MMDD	Total accounting file for MMDD

The fiscal directory contains periodic summary files created by monacct. A brief description of the files in the /var/adm/acct/fiscal directory is in Table 62-9.

Table 62-9 /var/adm/acct/fiscal

File	Description
cmsn	Total command summary file for fiscal period n in internal summary format
fiscrptn	Report similar to rprtn for fiscal period n
tacctn	Total accounting file for fiscal period n

Fixing Corrupted Files and wtmp Errors

Unfortunately, the UNIX accounting system is not foolproof. Occasionally, a file will become corrupted or lost. Some of the files can simply be ignored or restored from the backup. However, certain files must be fixed to maintain the integrity of the accounting system.

The wtmp files seem to cause the most problems in the day-to-day operation of the accounting system. When the date is changed and the system is in multiuser mode, a set of date change records is written into /var/adm/wtmp. The wtmpfix utility is designed to adjust the time stamps in the wtmp records when a date change is encountered. However, some combinations of date changes and reboots will slip through wtmpfix and cause acctcon to fail. For instructions on correcting wtmp problems, see "How to Fix a wtmp File" on page 1339.

Executing Routine Tasks Automatically

Many routine system events can be set up to execute automatically. Some of these tasks need to occur repetitively, at regular intervals. Other tasks need to run only once, perhaps during off hours such as evenings or weekends.

This section contains information about two commands, crontab and at, that enable you to schedule routine commands to execute automatically, avoiding peak hours or repeating commands according to a fixed schedule. crontab schedules repetitive commands, while at schedules commands that execute once.

Scheduling Repetitive Jobs: crontab

You can schedule routine system administration commands to execute daily, weekly, or monthly by using the crontab commands.

Daily crontab system administration tasks might include:

• Removing junk files more than a few days old from temporary directories
• Executing accounting summary commands
• Taking snapshots of the system by using df and ps commands
• Performing daily security monitoring
• Running system backups

Weekly crontab system administration tasks might include:

• Rebuilding the catman database for use by man -k
• Running fsck -n to list any disk problems

Monthly crontab system administration tasks might include:

• Listing files not used that month
• Producing monthly accounting reports

Additionally, users can schedule crontab commands to execute other routine system tasks, such as sending reminders and removing backup files.

How the cron Daemon Handles Scheduling

The cron daemon handles the automatic scheduling of crontab commands. Its function is to check the /usr/spool/cron/crontab directory (or the /var/spool/cron/crontab directory, depending on your system

configuration) for the presence of crontab files, normally every 15 minutes. It checks for new crontab files or changes to existing ones, reads the execution times listed within the files, and submits the commands for execution at the proper times.

In much the same way, the cron daemon controls the scheduling of at files, which are stored in the /usr/spool/cron/atjobs directory.

Inside a crontab File

The cron daemon schedules system events according to commands found within each crontab file. A crontab file consists of commands, one per line, that will be executed at regular intervals. The beginning of each line contains date and time information that tells the cron daemon when to execute the command.

For example, a crontab file named root is supplied during SunOS software installation. Its contents include these command lines:

0 20 * * 0,4 /etc/cron.d/logchecker 5 4 * * 6 /usr/lib/newsyslog 15 3 * * * /usr/lib/fs/nfs/nfsfind

The first command line instructs the system to run logchecker at 10 p.m. on Sundays and Thursdays. The second command line schedules the system to run newsyslog at 4:05 a.m. every Sunday. The third command line orders the system to execute nfsfind daily at 3:15 a.m.

For more information about the syntax of lines within a crontab file, see "Syntax of crontab File Entries" on page 1344.

The crontab files are stored in /usr/spool/cron/crontabs (or /var/spool/cron/crontabs). Several crontab files besides root are provided during SunOS software installation (see Table 62-10).

Table 62-10 crontab

crontab File	Function
adm	Accounting
lp	Printing
root	General system functions and file system cleanup
sys	Performance collection

Other crontab files are named after the user accounts in which they are created, such as bob, mary, smith, or jones.

Besides the default crontab file, users can create crontab files to schedule their own system events. To access crontab files belonging to root or other users, root privileges are required.

Procedures explaining how to create, edit, display, and remove crontab files are described in "Commands for Scheduling System Events" on page 1344.

crontab Command Security

For additional security over the scheduling of system events, you can set up files that control access to the crontab command, permitting only specified users to create, edit, display, or remove their crontab jobs. Two files control access to crontab: /etc/cron.d/cron.deny and /etc/cron.d/cron.allow. Each of these files consists of a list of user names, one per line. These files work together to control user access to crontab commands.

Whenever a user tries to access crontab, the crontab facility finds and reads the cron.allow file first, if one exists. crontab then reads the cron.deny file.

• If the cron.allow file exists, only the users listed in this file can create, edit, display, or remove crontab files.

• If cron.allow doesn't exist, all users may access crontab commands except for those listed in cron.deny.
• If neither file exists, only root can run crontab.

Root privileges are required to edit or create cron.deny and cron.allow.

By default, no cron.allow file is created during SunOS software installation. However, a cron.deny file is created, containing the following user names:

daemon bin smtp nuucp listen nobody noaccess

None of these user names can access crontab commands. You can edit this file to add other user names who will be denied access to crontab.

If you create a cron.allow file, only these users can access crontab commands.

Scheduling a Single Job: at

at allows you to schedule a job for execution at a later time. The job may consist of a single command or a script.

Like crontab, at allows you to schedule the automatic completion of routine commands. However, unlike crontab files, at files execute their commands once, and then are removed from their directory. Therefore, at is most useful for running simple commands or scripts that direct output into separate files for later examination.

Submitting an at job involves entering a command, following the at command syntax to specify options that schedule the time your job will be executed. For more information about submitting at jobs, see "at Command Description" on page 1356.

The at command stores the command or script that you entered, along with a copy of your current environment variables, in either /usr/spool/cron/atjobs or /var/spool/cron/atjobs. As a file name, your at job file is given a long number specifying its location in the at queue, followed by the .a extension, such as 793962000.a.

The cron daemon periodically executes the atrun program, usually at 15-minute intervals. atrun then executes at jobs at their scheduled times. After your at job has been executed, its file is removed from the atjobs directory.

at Command Security

You can set up a file to control access to the at command, permitting only specified users to create, remove, or display queue information about their at jobs. The file that controls access to at, /etc/cron.d/at.deny, consists of a list of user names, one per line. The users listed in this file cannot access at commands.

The at.deny file, created during SunOS software installation, contains the following user names:

daemon bin smtp nuucp listen nobody noaccess

With root privileges, you can edit this file to add other user names whose at access you want to restrict.