Chapter 8 Installing the Accounting and Reporting Console

This chapter describes the steps to install the Accounting and Reporting console software (ARCo):

• Setting Up the Database Software

• Install the Accounting and Reporting Software

Setting Up the Database Software

Database software must be installed and configured properly before you can install and use the Accounting and Reporting Console. This section describes how to set up the database software using either a PostgreSQL database or an Oracle database.

Set up the PostgreSQL Database Software

Detailed information on the PostgreSQL database software can be found in the Postgres documentation.

How to Start the Database Server

Once the PostgreSQL software is installed, you can start the database server.

Before You Begin

Download, compile, and install the PostgreSQL database software. Create a user account to own the database processes. Usually, this user is postgres. Add the PostgreSQL bin directory and necessary LD_LIBRARY_PATH settings to your environment.

Steps

1. Create a home directory for the postgres user.

   In this example, the home directory is /space/postgres/data.

   % mkdir -p /space/postgres/data % useradd -d /space/postgres postgres % chown postgres /space/postgres/data % su - postgres

2. Continue as described in the PostgreSQL documentation to set up a database.

   > initdb -D /space/postgres/data creating directory /space/postgres/data... ok creating directory /space/postgres/data/base... ok creating directory /space/postgres/data/global... ok creating directory /space/postgres/data/pg_xlog... ok creating directory /space/postgres/data/pg_clog... ok creating template1 database in /space/postgres/data/base/1... ok creating configuration files... ok initializing pg_shadow... ok enabling unlimited row size for system tables... ok initializing pg_depend... ok creating system views... ok loading pg_description... ok creating conversions... ok setting privileges on built-in objects... ok vacuuming database template1... ok copying template1 to template0... ok Success. You can now start the database server using: postmaster -D /space/postgres/data or pg_ctl -D /space/postgres/data -l logfile start

3. Make the following changes to the pg_hba.conf file.

   This change permits unrestricted and password free access to the database superuser postgres but requires md5 encrypted passwords for all other database users. Replace nnn.nnn.nnn with your subnet address without the trailing 0. You also can add access rules on a per-host basis by adding similar lines with host IP addresses.

   # TYPE  DATABASE  USER  IP-ADDRESS     IP-MASK         METHOD
   local   all       postgres                             trust
   local   all       all                                  md5
   # IPv4-style local connections:
   #host    all       all   nnn.nnn.nnn.0  255.255.255.0  md5
   

4. Make the following changes to the postgresql.conf file, to enable TCP/IP access from other hosts.

   ---

   Note –

   Ensure that the value of shared_buffers is at least twice the value of max_connections.

   ---

   tcpip_socket = true
   max_connections = 40   (increase if necessary)

5. Start the database.

   In this example, -i enables TCP/IP communication, and -S is for “silent” mode.

   > postmaster -S -i

6. Verify the installation.

   As the postgres user, try the following commands:

   % su - postgres > createuser -P test_user Enter password for new user: Enter it again: Shall the new user be allowed to create databases? (y/n) y Shall the new user be allowed to create more new users? (y/n) n CREATE USER > createdb -O test_user -E UNICODE test CREATE DATABASE

7. Execute commands as the database super user.

   > psql test Welcome to psql 7.3, the PostgreSQL interactive terminal. Type: \copyright for distribution terms \h for help with SQL commands \? for help on internal slash commands \g or terminate with semicolon to execute query \q to quit

   test=# create table test (x int, y text); CREATE TABLE test=# insert into test values (1, 'one'); INSERT 16982 1 test=# insert into test values (2, 'two'); INSERT 16983 1 test=# select * from test; x | y ---+------ 1 | one 2 | two (2 rows) test=# \q > psql -U test_user test Password: Welcome to psql 7.4.1, the PostgreSQL interactive terminal. Type: \copyright for distribution terms \h for help with SQL commands \? for help on internal slash commands \g or terminate with semicolon to execute query \q to quit test=>

8. After you have successfully tested your database, you should proceed to the next task, How to Set Up a PostgreSQL Database.

How to Set Up a PostgreSQL Database

Steps

1. Log in as the database superuser, for example, postgres.

   # su - postgres

2. Create the owner of the database.

   > createuser -P arco_write Enter password for new user: Enter it again: Shall the new user be allowed to create databases? (y/n) y Shall the new user be allowed to create more new users? (y/n) n CREATE USER

3. Create the database for accounting and reporting.

   > createdb -O arco_write arco CREATE DATABASE

4. Create a database user for reading the database.

   If you do not use the arco_read user, you will need to modify the privileges script that is used in the next step.

   > createuser -P arco_read Enter password for new user: Enter it again: Shall the new user be allowed to create databases? (y/n) n Shall the new user be allowed to create more new users? (y/n) n CREATE USER

How to Set Up an Oracle Database

Steps

1. Ask your database administrator for an instance of an Oracle database.

   You need two database users for this instance, arco_write and arco_read. The arco_write user must be able to create or alter tables, views, and indexes. Privileges for the arco_read user are set later by an SQL script.

2. Ask your database administrator for the connection string to the database.

Install the Accounting and Reporting Software

How to Set Up dbWriter

Steps

1. Extract the accounting and reporting software.

   • The tar Method

     # cd sge-root

     Type the following command as one string (the command is too long to fit on one line in this guide), with a space between the -dc and the path to the tar file.

     # gunzip -dc cdrom_mount_point/N1_Grid_Engine_6u4/ARCo/tar/n1ge-6_0-arco-1_0.tar.gz | tar xvpf -

   • The pkgadd Method

     At the command prompt, type the following command and respond to the script questions.

     # cd cdrom_mount_point/N1_Grid_Engine_6u4/ARCo/Packages # pkgadd -d . SUNWsgeea

2. As the administrative user, set your environment variables.

   • If you are using a Bourne shell or Korn shell, type the following command:

     $ . sge-root/default/common/settings.sh

   • If you are using a C shell, type the following command:

     % source sge-root/default/common/settings.csh

3. Change the global configuration to enable reporting.

   For details on how to enable reporting, see Report Statistics (ARCo) in N1 Grid Engine 6 Administration Guide.

   % qconf -mconf <......> reporting_params accounting=true \ reporting=true flush_time=00:00:15 joblog=true \ sharelog=00:00:00<......>

   By default, no report variables are activated. You can use the qconf command to enable statistics gathering on specific variables, as shown in the following example.

   % qconf -me global hostname global <......> report_variables cpu,np_load_avg,mem_free,virtual_free <......>

4. Install the dbWriter software.

   ---

   Note –

   With N1GE6 update 4 the dbwriter needs to be installed as user root.

   ---

   # cd sge-root/dbwriter % cd sge-root/dbwriter % su password: # ./inst_dbwriter # ./inst_dbwriter Welcome to the Grid Engine installation --------------------------------------- Grid Engine dbWriter installation --------------------------------- The dbWriter installation will take approximately 5 minutes Hit <RETURN> to continue >>

5. Specify the location of sge-root, and the name of your cell or cells.

   Hit <RETURN> to continue >> Checking $SGE_ROOT directory ---------------------------- The Grid Engine root directory is: $SGE_ROOT = /opt/n1ge6 If this directory is not correct (e.g. it may contain an automounter prefix) enter the correct path to this directory or hit <RETURN> to use default [/opt/n1ge6] >> Your $SGE_ROOT directory: /opt/n1ge6 Hit <RETURN> to continue >> Grid Engine cells ----------------- Grid Engine supports multiple cells. If you are not planning to run multiple Grid Engine clusters or if you don't know yet what is a Grid Engine cell it is safe to keep the default cell name default If you want to install multiple cells you can enter a cell name now. The environment variable $SGE_CELL=<your_cell_name> will be set for all further Grid Engine commands. Enter cell name [default] >> Using cell >default<. Hit <RETURN> to continue >>

6. Specify the location of your Java^TM Software Development Kit.

   Version 1.4.2 or higher is required. If your JAVA_HOME environment variable is set, the script will use that as a default value.

   Please enter the path to your java 1.4 installation [/usr/java] >>/opt/j2sdk1.4.2_06

7. Specify the connection parameters for the reporting database.

   Setup your database connection parameters ----------------------------------------- Enter your database type ( o = Oracle, p = PostgreSQL ) [p] >> p Please enter the name of your postgres db host []>> my-host.my-domain Please enter the port of your postgres db [5432] >> Please enter the name of your postgres database [arco] >>

8. Enter the user name and password of the database user.

   Please enter the name of the database user [arco_write] >> Please enter the password of the database user >> Please retype the password >>

9. Enter the name of the database schema.

   • If the JDBC driver is found, you will be asked for the name of the schema.

     For PostgreSQL this value is normally public. For oracle this should be the username of the database owner (arco_write).

     Please enter the name of the database schema [public] >> Search for the jdbc driver org.postgresql.Driver in directory /opt/n1ge6/dbwriter/lib .......... found in /opt/n1ge6/dbwriter/lib/pg73jdbc2.jar OK, jdbc driver found

   • If no corresponding JDBC driver is found, you are asked to copy the JAR file of the database driver into the sge-root/dbwriter/lib directory.

     After you copy the JAR file to the correct location, press RETURN, and the search repeats.

     Error: jdbc driver org.postgresql.Driver not found in any jar file of directory /opt/n1ge6/dbwriter/lib Please copy a jdbc driver for your database into this directory! Press enter to continue >>

10. Verify that the database has been set up correctly.

    Should the connection to the database be tested? (y/n) [y] >>

    • If the test succeeds, you will be notified.

      Test db connection to 'jdbc:postgresql://my-host.my-domain:5432/arco' ... OK

    • If the test fails, you are given the option to repeat the set up procedure.

      Test db connection to 'jdbc:postgresql://my-host.my-domain:5432/arco' ... Failed Do you want to repeat database connection setup? (y/n) [y] >>

11. Database model initialization or upgrade

    During the next step, the database tables are created or updated and the user arco_read is granted limited access to the database tables. The ARCo web application connects to the database using the user arco_readuser.

    Query db version ... no sge tables found New version of the database model is needed Should the database model be upgraded to version 1? (y/n) [y] >> The ARCo web application connects to the database with a user which has restricted access. The name of this database user is needed to grant him access to the sge tables. Please enter the name of this database user [arco_read] >> Upgrade to database model version 1 ... Install version 6.0 (id=0) ------- Create table sge_job Create index sge_job_idx0 Create index sge_job_idx1 create table sge_job_usage Create table sge_job_log Create table sge_job_request Create table sge_job_request Create table sge_queue Create index sge_queue_idx0 Create table sge_queue_values Create index sge_queue_values_idx0 Create table sge_host Create index sge_host_idx0 Create table sge_host_values Create index sge_host_values_idx0 Create table sge_department Create index sge_department_idx0 Create table sge_department_values Create index sge_department_values_idx0 Create table sge_project Create index sge_project_idx0 Create table sge_project_values Create index sge_project_values_idx0 Create table sge_user Create table sge_user_values Create index sge_user_values_idx0 Create table sge_group Create index sge_group_idx0 Creat table sge_group_values Create index sge_group_values_idx0 Create table sge_share_log Create view view_accounting Create view view_job_times Create view view_jobs_completed Create view view_job_log Create view view_department_values Create view view_group_values Create view_host_values Create view view_project_values Create view view_queue_values Create view view_user_values commiting changes version 6.0 (id=0) successfully installed Install version 6.0u1 (id=1) ------- Create table sge_version Update view view_job_times Update version table commiting changes version 6.0u1 (id=1) successfully installed OK

12. Specify how often the dbWriter program should check the grid engine system log files for new data.

    Please enter the interval between two dbwriter runs in seconds [60] >>

13. Specify the spool directory for the dbwriter. The dbwriter log files are stored in this directory. At startup the dbwriter process writes its pid in the file dbwriter.pid in this directory.

    Please enter the path of the dbwriter spool directory [$SGE_ROOT/$SGE_CELL/spool/dbwriter]>>

14. Enter the location of the file containing the rules for calculating derived values.

    Based on the accounting and reporting data, the dbWriter program calculates some derived values. The derived values are reporting information that are calculated from one or more raw values. The dbWriter program also allows you to delete data from the reporting database, when that data is not longer needed. The rules governing the derived value calculations and data deletion are stored in a file. The dbWriter program distribution contains an example file for a PostgreSQL database and an example file for an Oracle database. The example files are $SGE_ROOT/dbwriter/database/postgres/dbwriter.xml and $SGE_ROOT/dbwriter/database/oracle/dbwriter.xml

    You can accept the default file location, or you can specify a alternative path.

    Please enter the file with the derived value rules [$SGE_ROOT/dbwriter/database/postgres/dbwriter.xml] >>

15. Enter the level of logging that the dbWriter program should use.

    The dbWriter can run with different debug levels Possible values: WARNING INFO CONFIG FINE FINER FINEST Please enter the debug level of the dbwriter [INFO] >>

16. Verify the settings.

    If you answer n, you are given the option to repeat the set up.

    All parameters are now collected -------------------------------- SGE_ROOT=/opt/n1ge6 SGE_CELL=default JAVA_HOME=/usr/java (java version "1.4.2") DB_URL=jdbc:postgresql://my-host.my-domain:5432/arco DB_USER=arco_write INTERVAL=60 REPORTING_FILE=/opt/n1ge6/default/common/reporting DERIVED_FILE=/opt/n1ge6/dbwriter/database/postgres/dbwriter.xml DEBUG_LEVEL=INFO Are this settings correct? (y/n) [y] >> y

    The installation scripts creates a start script, sge-root/dbwriter/bin/sgedbwriter. Your configuration parameters are stored in sge-root/cell/common/dbwriter.conf. If any configuration changes are necessary, you can edit the dbwriter.conf file or repeat the installation script.

    Create configuration file for dbWriter in /opt/n1ge6/default/common

17. The installation script prompts you to choose whether or not to install a boot time startup script for the dbwriter:

    dbwriter startup script ----------------------- We can install the startup script that will start dbwriter at machine boot (y/n) [y] >> y Installing startup script /etc/rc2.d/S96sgedbwriter Installation of dbWriter completed

18. Start the dbWriter program.

    # /etc/init.d/sgedbwriter start

How to Install Sun Web Console

Steps

1. Check whether Sun Web Console is already available on your system (this is usually the case for Solaris 10 and on newer Solaris 9 versions). As root, you can check using the following command:

   # smcwebserver -V Version 2.2.1

   ARCo for N1GE6 update 4 requires Sun Web Console 2.2.1 or higher.

2. If you need to install the console, extract the web console package under a temporary directory.

   # cd /tmp # umask 022 # mkdir swc # cd swc

   Type the following command as one string (the command is too long to fit on one line in this guide), with a space between the xvf and the path to the tar file.

   # tar xvf cdrom_mount_point/N1_Grid_Engine_6u4/SunWebConsole/tar/swc_sparc_2.2.1.tar

3. If the user noaccess and the group noaccess do not exist in your password file or NIS passwd map, add them by typing:

   # groupadd -g 60002 noaccess # useradd -u 60002 -g 60002 -d /tmp -s /bin/sh -c "No Access User" noaccess

   ---

   Note –

   On some operating systems, using /bin/csh as the default shell may cause the installation to fail. If ARCo will not install, and you see the error message Server not started! No management applications registered try setting the default shell as /bin/sh for noaccess.

   ---

4. If you are running SuSE 9.0, create symbolic links for each of the /etc/rc#.d directories.

   # ln -s /etc/rc.d/rc0.d /etc/rc0.d # ln -s /etc/rc.d/rc1.d /etc/rc1.d # ln -s /etc/rc.d/rc2.d /etc/rc2.d

5. Run the Sun Web Console setup script.

   # ./setup -n <....> Installation complete. Starting Sun(TM) Web Console Version 2.2.1... See /var/log/webconsole/console_debug_log for server logging information

   The web console is installed but not started until after the ARCo console installation. Once the console is installed, you can stop, start, or restart the console at any time, using the following commands:

   # /usr/sadm/bin/smcwebserver start # /usr/sadm/bin/smcwebserver stop # /usr/sadm/bin/smcwebserver restart

How to Install the Accounting and Reporting Console

Steps

1. Change directory to sge-root/reporting.

   # cd sge-root/reporting

2. Use the inst_reporting script to install the software.

   # ./inst_reporting Welcome to the N1 SGE reporting module installation ---------------------------------------------------- The installation will take approximately 5 minutes Hit <RETURN> to continue >>

3. The install script checks for the existence of the SGE_ROOT directory and SGE_CELL name. If you already created these parameters, the script continues.

4. Enter the path to your Java Software Development Kit.

   Version 1.4.2 or higher is required. If your JAVA_HOME environment variable is set, the script will use that as a default value.

   Please enter the path to your java 1.4 installation [] >> /opt/j2sdk1.4.2_06

5. Specify the directory where the accounting and reporting software can store its results.

   If this directory does not exist, it will be created for you.

   Spool directory --------------- In the spool directory the N1 SGE reporting module will store all queries and results Please enter the path to the spool directory [/var/spool/arco] >>

6. Specify the parameters for the database connection.

   Database Setup -------------- Enter your database type ( o = Oracle, p = PostgreSQL ) [p] >> o Please enter the name of your oracle db host [] >> my-host Please enter the port of your oracle db [1521] >> Please enter the name of your oracle database [arco] >>

7. Specify an accounting and reporting database user.

   Do not use the same database user as you did for the dbwriter application. For security reasons, it is recommended that the database user for accounting and reporting only have read permission for the database tables.

   Please enter the name of the database user [arco_read] >> Please enter the password of the database user >> Please retype the password >> Please enter the name of the database schema [arco_write] >> Search for the jdbc driver oracle.jdbc.driver.OracleDriver in directory /opt/n1ge/reporting/WEB-INF/lib ... found in /opt/n1ge/reporting/WEB-INF/lib/classes12.jar Should the connection to the database be tested? (y/n) [y] >> y Test db connection to 'jdbc:oracle:thin:@my-system:1521:arco' ... OK h150

   The name of the database schema depends on your database. For a Postgres database, the database schema name should be public. For Oracle the name of the database schema should be equal to the name of the user account which is used by the dbwriter (arco_write).

   Please enter the name of the database schema [arco_write] >> Searching for the jdbc driver oracle.jdbc.driver.OracleDriver OK, jdbc driver found Should the connection to the database be tested? (y/n) [y] >> y Test db connection to 'jdbc:oracle:thin:@my-system:1521:arco' ... OK

8. Enter the login names of users who are allowed to store the queries and results.

   Configure users with write access Users: sge-admin Enter a login name of a user (Press enter to finish) >> user2 Users: sge-admin user2 Enter a login name of a user (Press enter to finish) >> ---------------------------------

   ---

   Note –

   After installation, you can add or delete authorized users by editing the config.xml file. See the information at the end of this chapter.

   ---

9. Verify the information.

   All parameters are now collected -------------------------------- SPOOL_DIR=/var/spool/arco DB_URL=jdbc:oracle:thin://my-system:1521/arco DB_USER=arco_read ARCO_WRITE_USERS=user1 user2 Are this settings correct? (y/n) [y] >> y Shutting down Sun(TM) Web Console Version 2.2.1...

10. If a previous version of ARCo had been installed you will be asked to delete it:

    Found old versions of the N1 SGE reporting module ------------------------------------------------- Should the application com.sun.grid.arco_6u1 be deleted? (y/n) [n] >> y Unregistering com.sun.grid.arco_6u1.

11. Create the query directory.

    If the query directory already exists, it will be created. The example queries will be installed. Existing queries will not be replaced.sge-root/reporting/database/example_queries/queries.

    Directory /var/spool/arco does not exist, create it? (y/n) [y] >> y Create query directory /var/spool/arco Create query directory /var/spool/arco/queries Copy query Accounting_per_Department.xml ... OK Copy query Accounting_per_Project.xml ... OK Copy query Accounting_per_User.xml ... OK Copy query Average_Job_Turnaround_Time.xml ... OK Copy query Average_Job_Wait_Time.xml ... OK Copy query Host_Load.xml ... OK Copy query Job_Log.xml ... OK Copy query Number_of_Jobs_completed.xml ... OK Copy query Queue_Consumables.xml ... OK Copy query Statistics.xml ... OK Copy query Wallclock_time.xml ... OK Create query directory /var/spool/arco/results Register the N1 SGE reporting module in the webconsole Registering com.sun.grid.arco_6u4. Starting Sun(TM) Web Console Version 2.2.1... See /var/log/webconsole/console_debug_log for server logging information

12. Check the log file for error or warning messages.

    # more /var/log/webconsole/console_debug_log

    The accounting and reporting logs are written to the /var/log/webconsole/console_debug_log file. The default log level is INFO, but you can modify the log level from the command line:

    # smreg add -p -e arco_logging_level=FINE

    The new log take effect the next time the console is started or restarted. The possible log levels are WARNING, INFO, FINE, FINER, and FINEST.

13. Connect to the Sun Web Console by accessing the following URL in your browser. Replace hostname with the name of your master host.

    https://hostname:6789

14. Login with your UNIX account.

15. Select N1 Grid Engine 6 Accounting and Reporting Console.

How to Add Authorized ARCo Users

During the installation of the ARCo reporting module, the installer is asked to enter a list of users who should have write permissions to the ARCo system. Only those users are allowed save modifications on ARCo.

Steps

1. Depending on the web console version you are using, the list of authorized users is stored in /var/opt/webconsole/webapps/reporting/config.xml or /usr/share/webconsole/webapps/reporting/config.xml.

2. After editing this file you have to restart the sun web console:

   # smcwebserver restart