Chapter 9 Managing System Accounting (Tasks)

This chapter describes how to set up and maintain system accounting.

This is a list of the overview information in this chapter.

• What is System Accounting?

• Setting Up System Accounting

For information on using extended accounting, see Chapter 4, Extended Accounting (Overview), in System Administration Guide: Solaris Containers-Resource Management and Solaris Zones.

For information on the step-by-step procedures that are associated with system accounting, see System Accounting (Task Map).

For reference information about the various system accounting reports, see Chapter 10, System Accounting (Reference).

What's New in System Accounting

This section describes new or changed features in system accounting in the Solaris release. For a complete listing of new Solaris features and a description of Solaris releases, see Solaris 10 What’s New.

Solaris Process Accounting and Statistics Improvements

Solaris 10: Changes have been made to the internals of the load averaging, cpu usr/sys/idle, and accounting functions. Microstate accounting has replaced the old accounting mechanism and is enabled by default all of the time. As a result, you might notice slightly different process usage and timing statistics.

Switching to microstate accounting provides substantially more accurate data about user processes and the amount of time they spend in various states. In addition, this information is used to generate more accurate load averages and statistics from the /proc file system. For more information, see the proc(4) man page.

What is System Accounting?

System accounting software in the Solaris OS is a set of programs that enables you to collect and record data about user connect time, CPU time charged to processes, and disk usage. Once you collect this data, you can generate reports and charge fees for system usage.

You can use system accounting on a daily or monthly basis. Or, you can tack disk usage per user.

You can use the accounting programs to perform these tasks:

• Monitor system usage

• Locate and correct performance problems

• Maintain system security

After you set up the system accounting programs, they run mostly on their own.

How System Accounting Works

Automatic accounting is set up by first putting the accounting startup script into root's crontab file. The accounting startup script can then be started automatically by the cron command.

The following overview describes the system accounting process.

1. Between system startup and shutdown, raw data about system use (such as user logins, running processes, and data storage) are collected in accounting files.

2. Periodically (usually once a day), the /usr/lib/acct/runacct script processes the various accounting files and produces both cumulative summary files and daily accounting reports. Then, the /usr/lib/acct/prdaily script prints the daily reports.

   For more information about the runacct script, see runacct Script.

3. Monthly, you can process and print the cumulative runacct summary files by executing the monacct script. The summary reports produced by the monacct script provide an efficient means for billing users on a monthly or other fiscal basis.

System Accounting Components

The system accounting software provides C language programs and shell scripts that organize data into summary files and reports. These programs reside in the /usr/lib/acct directory. The accounting reports reside in the /var/adm/acct directory.

Daily accounting can help you perform four types of auditing:

• Connect accounting

• Process accounting

• Disk accounting

• Fee calculations

Connect Accounting

Connect accounting enables you to determine the following information:

• The length of time a user was logged in

• How the tty lines are being used

• The number of reboots on your system

• How many times the accounting software was turned off and on

To provide this information on connect sessions, the system stores the following data

• Record of time adjustments

• Boot times

• Number of times the accounting software was turned off and on

• Changes in run levels

• The creation of user processes (login processes and init processes)

• The terminations of processes

These records are produced from the output of system programs such as date, init, login, ttymon, and acctwtmp. They are stored in the /var/adm/wtmpx file.

Entries in the wtmpx file can contain the following information:

• Login name

• Device name

• Process ID

• Entry type

• Time stamp that denotes when the entry was made

Process Accounting

Process accounting enables you to keep track of the following data about each process that runs on your system:

• User IDs and group IDs of users using the process

• Beginning times and elapsed times of the process

• CPU time for the process (user time and system time)

• Amount of memory used by the process

• Commands run by the process

• The tty that controls the process

Every time a process terminates, the exit program collects this information and writes it to the /var/adm/pacct file.

Disk Accounting

Disk accounting enables you to gather and format the following data about the files each user has on disks:

• User name and user ID of the user

• Number of blocks that are used by the user's files

This data is collected by the /usr/lib/acct/dodisk shell script at intervals that are determined by the entry you add to the /var/spool/cron/crontabs/root file. In turn, the dodisk script invokes the acctdisk and acctdusg commands. These commands gather disk usage by login name.

Information gathered by running the dodisk script is stored in the /var/adm/acct/nite/disktacct file. This information is overwritten the next time the dodisk script is run. Therefore, avoid running the dodisk script twice in the same day.

The acctdusg command might overcharge for files that are written randomly, which can create holes in the files. This problem occurs because the acctdusg command does not read the indirect blocks of a file when determining the file size. Rather, the acctdusg command determines the file size by checking the current file size value in the file's inode.

Fee Calculations

The chargefee utility stores charges for special services that are provided to a user in the /var/adm/fee file. A special service, for example, is file restoration. Each entry in the file consists of a user login name, user ID, and the fee. This file is checked by the runacct script every day, and new entries are merged into the accounting records. For instructions on running the chargefee script to bill users, see How to Bill Users.

How Daily Accounting Works

Here is a step-by-step summary of how daily accounting works:

1. When the system is switched into multiuser mode, the /usr/lib/acct/startup program is executed. The startup program executes several other programs that invoke daily accounting.

2. The acctwtmp program adds a “boot” record to the /var/adm/wtmpx file. In this record, the system name is shown as the user name in the wtmpx record. The following table summarizes how the raw accounting data is gathered and where it is stored.

   File in /var/adm	Information Stored	Written By	Format
   wtmpx	Connect sessions	login, init	Binary
   	Changes	date	Binary
   	Reboots	acctwtmp	Binary
   	Shutdowns	shutacct	Binary
   pacctn	Processes	Kernel (when the process ends)	Binary
   		turnacct switch (which creates a new file when the old file reaches 500 blocks)	Binary
   fee	Special charges	chargefee	ASCII
   acct/nite/disktacct	Disk space used	dodisk	Binary

3. The turnacct script, invoked with the -on option, begins process accounting. Specifically, the turnacct script executes the accton program with the /var/adm/pacct argument.

4. The remove shell script “cleans up” the saved pacct and wtmpx files that are left in the sum directory by the runacct script.

5. The login and init programs record connect sessions by writing records into the /var/adm/wtmpx file. Date changes (using date with an argument) are also written to the /var/adm/wtmpx file. Reboots and shutdowns using the acctwtmp command are also recorded in the /var/adm/wtmpx file.

6. When a process ends, the kernel writes one record per process, using the acct.h format, in the /var/adm/pacct file.

   Every hour, the cron command executes the ckpacct script to check the size of the /var/adm/pacct file. If the file grows beyond 500 blocks (default), the turnacct switch command is executed. (The program moves the pacct file to the pacctn file and creates a new file.) The advantage of having several smaller pacct files becomes apparent when you try to restart the runacct script if a failure occurs when processing these records.

7. The runacct script is executed by the cron command each night. The runacct script processes the accounting files to produce command summaries and usage summaries by user name. These accounting files are processed: /var/adm/pacctn, /var/adm/wtmpx, /var/adm/fee, and /var/adm/acct/nite/disktacct.

8. The /usr/lib/acct/prdaily script is executed on a daily basis by the runacct script to write the daily accounting information in the /var/adm/acct/sum/rprtMMDD files.

9. The monacct script should be executed on a monthly basis (or at intervals you determine, such as at the end of every fiscal period). The monacct script creates a report that is based on data stored in the sum directory that has been updated daily by the runacct script. After creating the report, the monacct script “cleans up” the sum directory to prepare the directory's files for the new runacct data.

What Happens if the System Shuts Down

If the system is shut down by using the shutdown command, the shutacct script is executed automatically. The shutacct script writes a reason record into the /var/adm/wtmpx file and turns off process accounting.

System Accounting (Task Map)

Setting Up System Accounting

You can set up system accounting to run while the system is in multiuser mode (Run Level 2). Generally, this task involves these steps:

1. Creating the /etc/rc0.d/K22acct and /etc/rc2.d/S22acct startup scripts

2. Modifying the /var/spool/cron/crontabs/adm and /var/spool/cron/crontabs/root crontab files

The following table describes the default accounting scripts.

You can choose which accounting scripts run by default. After these entries have been added to the crontab files, system accounting should run automatically.

How to Set Up System Accounting

1. Become superuser or assume an equivalent role.

   Roles contain authorizations and privileged commands. For more information about roles, see Configuring RBAC (Task Map) in System Administration Guide: Security Services.

2. If necessary, install the SUNWaccr and SUNWaccu packages on your system by using the pkgadd command.

3. Install /etc/init.d/acct as the startup script for Run Level 2.

   # ln /etc/init.d/acct /etc/rc2.d/S22acct

4. Install /etc/init.d/acct as the stop script for Run Level 0.

   # ln /etc/init.d/acct /etc/rc0.d/K22acct

5. Add the following lines to the adm crontab file to start the ckpacct, runacct, and monacct scripts automatically.

   # EDITOR=vi; export EDITOR # crontab -e adm 0 * * * * /usr/lib/acct/ckpacct 30 2 * * * /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log 30 7 1 * * /usr/lib/acct/monacct

6. Add the following line to the root crontab file to start the dodisk script automatically.

   # crontab -e 30 22 * * 4 /usr/lib/acct/dodisk

7. Edit /etc/acct/holidays to include national holidays and local holidays.

   For more information, see the holidays(4) man page and the example that follows.

8. Reboot the system, or start system accounting manually by typing:

   # /etc/init.d/acct start

Example 9–1 Setting Up Accounting (adm crontab)

This modified adm crontab contains entries for the ckpacct, runacct, and monacct scripts.

#ident  "@(#)adm        1.5     92/07/14 SMI"    /* SVr4.0 1.2   */
#
# The adm crontab file should contain startup of performance 
# collection if the profiling and performance feature has been 
# installed.
0 * * * * /usr/lib/acct/ckpacct
30 2 * * * /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log
30 7 1 * * /usr/lib/acct/monacct

Example 9–2 Setting Up Accounting (root crontab)

This modified root crontab contains entries for the dodisk program.

#ident  "@(#)root       1.19    98/07/06 SMI"   /* SVr4.0 1.1.3.1       */
#
# The root crontab should be used to perform accounting data collection.
#
#
10 3 * * * /usr/sbin/logadm
15 3 * * 0 /usr/lib/fs/nfs/nfsfind
30 3 * * * [ -x /usr/lib/gss/gsscred_clean ] && /usr/lib/gss/gsscred_clean
30 22 * * 4 /usr/lib/acct/dodisk

Example 9–3 Setting Up Accounting (/etc/acct/holidays)

The following example shows a sample /etc/acct/holidays file.

* @(#)holidays	January 1, 2004
*
* Prime/Nonprime Table for UNIX Accounting System
*
* Curr	Prime	Non-Prime
* Year	Start	Start
*
  2004	0800	1800
*
* only the first column (month/day) is significant.
*
* month/day	Company
* 		Holiday
*
1/1		New Years Day
7/4		Indep. Day
12/25		Christmas

Billing Users

If you provide special user services by request, you might want to bill users by running the chargefee utility. Special services include restoring files or remote printing. The chargefee utility records charges in the /var/adm/fee file. Each time the runacct utility is executed, new entries are merged into the total accounting records.

See the acctsh(1M) man page for more information.

How to Bill Users

1. Become superuser or assume an equivalent role.

   Roles contain authorizations and privileged commands. For more information about roles, see Configuring RBAC (Task Map) in System Administration Guide: Security Services.

2. Charge a user for special services.

   # /usr/lib/acct/chargefee username amount

   username

   Is the user account you want to bill.

   amount

   Specifies the number of units to bill the user. This value is an arbitrary unit that you set to charge users based on some task such as printing or restoring a file. You would have to write a script that invokes the chargefee utility and charges a user for a specific task.

Example 9–4 Billing Users

In the following example, the user print_customer is charged 10 units.

# /usr/lib/acct/chargefee print_customer 10

Maintaining Accounting Information

This section describes how to fix corrupted system accounting files and how to restart the runacct script.

Fixing Corrupted Files and wtmpx Errors

Unfortunately, system accounting is not foolproof. Occasionally, a file becomes corrupted or lost. Some files can simply be ignored or restored from backup. However, certain files must be fixed to maintain the integrity of system accounting.

The wtmpx files seem to cause the most problems in the daily operation of system accounting. When the date is changed manually and the system is in multiuser mode, a set of date change records is written to the /var/adm/wtmpx file. The wtmpfix utility is designed to adjust the time stamps in the wtmp records when a date change is encountered. However, some combinations of date changes and reboots slip through the wtmpfix utility and cause the acctcon program to fail.

How to Fix a Corrupted wtmpx File

1. Become superuser or assume an equivalent role.

   Roles contain authorizations and privileged commands. For more information about roles, see Configuring RBAC (Task Map) in System Administration Guide: Security Services.

2. Change to the /var/adm directory.

3. Convert the wtmpx file from binary format to ASCII format.

   # /usr/lib/acct/fwtmp < wtmpx > wtmpx.ascii

4. Edit the wtmpx.ascii file to delete the corrupted records.

5. Convert the wtmpx.ascii file back to a binary file.

   # /usr/lib/acct/fwtmp -ic < wtmpx.ascii > wtmpx

   See the fwtmp(1M) man page for more information.

Fixing tacct Errors

The integrity of the /var/adm/acct/sum/tacct file is important if you are charging users for system resources. Occasionally, unusual tacct records appear with negative numbers, duplicate user IDs, or a user ID of 65535. First, check the /var/adm/acct/sum/tacctprev file by using the prtacct script to print the file. If the contents look all right, patch the latest /var/adm/acct/sum/tacctMMDD file. Then, re-create the /var/adm/acct/sum/tacct file. The following steps outline a simple patch procedure.

How to Fix tacct Errors

1. Become superuser or assume an equivalent role.

   Roles contain authorizations and privileged commands. For more information about roles, see Configuring RBAC (Task Map) in System Administration Guide: Security Services.

2. Change to the /var/adm/acct/sum directory.

3. Convert the tacctMMDD file from binary format to ASCII format.

   # /usr/lib/acct/acctmerg -v < tacctMMDD > xtacct

   MMDD is pair of two-digit numbers that represent the month and day.

4. Edit the xtacct file, removing corrupted records and writing duplicate records to another file.

5. Convert the xtacct file from ASCII format to binary format.

   # /usr/lib/acct/acctmerg -i < xtacct > tacctMMDD

6. Merge the files tacctprev and tacct.MMDD into the tacct file.

   # /usr/lib/acct/acctmerg < tacctprev tacctMMDD > tacct

Restarting the runacct Script

The runacct script can fail for several reasons.

The following are the most common reasons:

• A system crash

• The /var directory is running out of space

• A corrupted wtmpx file

If the active.MMDD file exists, check it first for error messages. If the active and lock files exist, check the fd2log file for any relevant messages.

Run without arguments, the runacct script assumes that this invocation is the first invocation of the day. The argument MMDD is necessary if the runacct script is being restarted and specifies the month and day for which the runacct script reruns the accounting. The entry point for processing is based on the contents of the statefile file. To override the statefile file, include the desired state on the command line. For a description of the available states, see the runacct(1M) man page.

When you run the runacct program manually, be sure to run it as user adm.

How to Restart the runacct Script

1. Change directories to the /var/adm/acct/nite directory.

   $ cd /var/adm/acct/nite

2. Remove the lastdate file and any lock* files, if any.

   $ rm lastdate lock*

   The lastdate file contains the date that the runacct program was last run. Restarting the runacct script in the next step re-creates this file.

3. Restart the runacct script.

   $ /usr/lib/acct/runacct MMDD [state] 2> /var/adm/acct/nite/fd2log &

   MMDD

   Is the month and day specified by two-digit numbers.

   state

   Specifies a state, or starting point, where the runacct script processing should begin.

Stopping and Disabling System Accounting

You can temporarily stop system accounting or permanently disable it.

How to Temporarily Stop System Accounting

1. Become superuser or assume an equivalent role.

   Roles contain authorizations and privileged commands. For more information about roles, see Configuring RBAC (Task Map) in System Administration Guide: Security Services.

2. Edit the adm crontab file to stop the ckpacct, runacct, and monacct programs from running by commenting out the appropriate lines.

   # EDITOR=vi; export EDITOR # crontab -e adm #0 * * * * /usr/lib/acct/ckpacct #30 2 * * * /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log #30 7 1 * * /usr/lib/acct/monacct

3. Edit the root crontab file to stop the dodisk program from running by commenting out the appropriate line.

   # crontab -e #30 22 * * 4 /usr/lib/acct/dodisk

4. Stop the system accounting program.

   # /etc/init.d/acct stop

5. (Optional) Remove the newly added comment symbols from the crontab files.

6. Restart the system accounting program to re-enable system accounting.

   # /etc/init.d/acct start

How to Permanently Disable System Accounting

1. Become superuser or assume an equivalent role.

   Roles contain authorizations and privileged commands. For more information about roles, see Configuring RBAC (Task Map) in System Administration Guide: Security Services.

2. Edit the adm crontab file and delete the entries for the ckpacct, runacct, and monacct programs.

   # EDITOR=vi; export EDITOR # crontab -e adm

3. Edit the root crontab file and delete the entries for the dodisk program.

   # crontab -e

4. Remove the startup script for Run Level 2.

   # unlink /etc/rc2.d/S22acct

5. Remove the stop script for Run Level 0.

   # unlink /etc/rc0.d/K22acct

6. Stop the system accounting program.

   # /etc/init.d/acct stop