Chapter 2 Compiling and Running OpenMP Programs

This chapter describes compiler and runtime options affecting programs that utilize the OpenMP API.

To run an OpenMP program in a multithreaded environment, you must set the number of threads in the the program to be greater than one. The number of threads is controlled by the OMP_SET_NUM_THREADS environment variable, which must be set greater than one prior to running the program. The number of threads can also be set by calling omp_set_num_threads() in the program with a value greater than one, or by using the num_threads clause with a PARALLEL directive.

The latest information regarding Sun Studio compilers and OpenMP can be found on the Sun Developer Network portal, http://developers.sun.com/sunstudio

2.1 Compiler Options To Use

To enable explicit parallelization with OpenMP directives, compile your program with the cc, CC, or f95 option flag -xopenmp. (The f95 compiler accepts both -xopenmp and -openmp as synonyms.)

The -xopenmp flag accepts the following optional keyword sub-options.

-xopenmp=parallel

-xopenmp=noopt

-xopenmp=stubs

-xopenmp=none

Additional Notes:

• If you do not specify —xopenmp on the command line, the compiler assumes —xopenmp=none (disabling recognition of OpenMP directives).

• If you specify —xopenmp but without a keyword sub-option, the compiler assumes —xopenmp=parallel.

• Specifying -xopenmp=parallel or noopt will define the _OPENMP preprocessor token to be YYYYMM (specifically 200805L for C/C++ and 200805 for Fortran 95).

• When debugging OpenMP programs with dbx, compile with -xopenmp=noopt -g

• The default optimization level for -xopenmp might change in future releases. Compilation warning messages can be avoided by specifying an appropriate optimization level explicitly.

• With Fortran 95, -xopenmp , -xopenmp=parallel, -xopenmp=noopt will add -stackvar automatically.

• When compiling and linking an OpenMP program in separate steps, include -xopenmp on each of the compile and the link steps.

• Use the -xvpara C/C++/Fortran option to display compiler parallelization messages.

• For best performance and functionality on Solaris platforms, make sure that the latest OpenMP runtime library, libmtsk.so, is installed on the running system.

2.2 OpenMP Environment Variables

The OpenMP specification defines a number of environment variables that control the execution of OpenMP programs. These are summarized in the following table. For details, refer to the OpenMP API Version 3.0 specifications. See also 3.8 Environment Variables

Sun Studio supports additional multiprocessing environment variables that affect execution of OpenMP programs and are not part of the OpenMP specifications. These are summarized in the following table.

   int sunw_mp_register_warn (void (*func)(void *));

setenv SUNW_MP_THR_IDLE SPIN 
setenv SUNW_MP_THR_IDLE SLEEP 
setenv SUNW_MP_THR_IDLE SLEEP(2s) 
setenv SUNW_MP_THR_IDLE SLEEP(20ms) 
setenv SUNW_MP_THR_IDLE SLEEP(150mc)

2.3 Processor Binding

With processor binding, the programmer instructs the operating system that a thread in the program should run on the same processor throughout the execution of the program.

Processor binding, when used along with static scheduling, benefits applications that exhibit a certain data reuse pattern where data accessed by a thread in a parallel or worksharing region will be in the local cache from a previous invocation of a parallel or worksharing region.

From the hardware point of view, a computer system is composed of one or more physical processors. From the operating system point of view, each of these physical processors maps to one or more virtual processors onto which threads in a program can be run. If n virtual processors are available, then n threads can be scheduled to run at the same time. Depending on the system, a virtual processor may be a processor, a core, etc.

For example, each UltraSPARC IV physical processor has two cores; from the Solaris OS point of view, each of these cores is a virtual processor onto which a thread can be scheduled to run. The UltraSPARC T1 physical processor, on the other hand, has eight cores, and each core can run four simultaneous processing threads; from the Solaris OS point of view, there are 32 virtual processors onto which threads can be scheduled to run. On the Solaris Operating System, the number of virtual processors can be determined by using the psrinfo(1M) command. On Linux systems, the file /proc/cpuinfo provides information about available processors.

When the operating system binds threads to processors, they are in effect bound to specific virtual processors, not physical processors.

Set the SUNW_MP_PROCBIND environment variable to bind threads in an OpenMP program to specific virtual processors. The value specified for SUNW_MP_PROCBIND can be one of the following:

• The string "TRUE" or "FALSE" (or lower case "true" or "false"). For example, % setenv SUNW_MP_PROCBIND "false"

• A non-negative integer. For example, % setenv SUNW_MP_PROCBIND "2"

• A list of two or more non-negative integers separated by one or more spaces. For example, % setenv SUNW_MP_PROCBIND "0 2 4 6"

• Two non-negative integers, n1 and n2, separated by a minus ("-"); n1 must be less than or equal to n2. For example, % setenv SUNW_MP_PROCBIND "0-6"

Note that the non-negative integers referred to above denote logical identifiers (IDs). Logical IDs may be different from virtual processor IDs. The difference will be explained below.

Virtual Processor IDs:

Each virtual processor in a system has a unique processor ID. On Solaris platforms, you can use the psrinfo(1M) command to display information about the processors in a system, including their processor IDs. You can use psrinfo -pv to list all physical processors in the system and the virtual processors that are associated with each physical processor. Moreover, you can use the prtdiag(1M) command to display system configuration and diagnostic information.

Virtual processor IDs may be sequential or there may be gaps in the IDs. For example, on a Sun Fire 4810 with 8 UltraSPARC IV processors (16 cores), the virtual processor IDs may be: 0, 1, 2, 3, 8, 9, 10, 11, 512, 513, 514, 515, 520, 521, 522, 523.

Logical IDs:

As mentioned above, the non-negative integers specified for SUNW_MP_PROCBIND are logical IDs. Logical IDs are consecutive integers that start with 0. If the number of virtual processors available in the system is n, then their logical IDs are 0, 1, ..., n-1, in the order presented by psrinfo(1M). The following Korn shell script can be used to display the mapping from virtual processor IDs to logical IDs.

#!/bin/ksh

NUMV=`psrinfo | fgrep "on-line" | wc -l `
set -A VID  `psrinfo | cut -f1 `

echo "Total number of on-line virtual processors = $NUMV"
echo

let "I=0"
let "J=0"
while [[ $I -lt $NUMV ]]
do
  echo "Virtual processor ID ${VID[I]} maps to logical ID ${J}"
  let "I=I+1"
  let "J=J+1"
done

On systems where a single physical processor maps to several virtual processors, it may be useful to know which logical IDs correspond to virtual processors that belong to the same physical processor. The following Korn shell script can be used with later Solaris releases to display this information.

#!/bin/ksh

NUMV= `psrinfo | grep "on-line" | wc -l `
set -A VLIST  `psrinfo | cut -f1 `
set -A CHECKLIST  `psrinfo | cut -f1 `

let "I=0"

while [ $I -lt $NUMV ]
do
  let "COUNT=0"
  SAMELIST="$I"

  let "J=I+1"

  while [ $J -lt $NUMV ]
  do
    if [ ${CHECKLIST[J]} -ne -1 ]
    then
      if [  `psrinfo -p ${VLIST[I]} ${VLIST[J]} ` = 1 ]
      then
        SAMELIST="$SAMELIST $J"
        let "CHECKLIST[J]=-1"
        let "COUNT=COUNT+1"
      fi
    fi
    let "J=J+1"
  done

  if [ $COUNT -gt 0 ]
  then
    echo "The following logical IDs belong to the same physical processor:"
    echo "$SAMELIST"
    echo " "
  fi

  let "I=I+1"
done

Interpreting the Value Specified for SUNW_MP_PROCBIND:

If the value specified for SUNW_MP_PROCBIND is TRUE, then the threads will be bound to virtual processors in a round-robin fashion, starting with the processor whose logical ID is 0. (Specifying TRUE is equivalent to specifying the value 0 for SUNW_MP_PROCBIND.)

If the value specified for SUNW_MP_PROCBIND is a non-negative integer, then that integer denotes the starting logical ID of the virtual processor to which threads should be bound. Threads will be bound to virtual processors in a round-robin fashion, starting with the processor with the specified logical ID, and wrapping around to the processor with logical ID 0, after binding to the processor with logical ID n-1.

If the value specified for SUNW_MP_PROCBIND is a list of two or more non-negative integers, then threads will be bound in a round-robin fashion to virtual processors with the specified logical IDs. Processors with logical IDs other than those specified will not be used.

If the value specified for SUNW_MP_PROCBIND is two non-negative integers separated by a minus ("-"), then threads will be bound in a round-robin fashion to virtual processors in the range that begins with the first logical ID and ends with the second logical ID. Processors with logical IDs other than those included in the range will not be used.

If the value specified for SUNW_MP_PROCBIND does not conform to one of the forms described above, or if an invalid logical ID is given, then an error message will be emitted and execution of the program will terminate.

Note that the number of threads created by the microtasking library, libmtsk, depends on environment variables, API calls in the user’s program, and the num_threads clause. SUNW_MP_PROCBIND specifies the logical IDs of virtual processors to which the threads should be bound. Threads will be bound to that set of processors in a round-robin fashion. If the number of threads used in the program is less than the number of logical IDs specified by SUNW_MP_PROCBIND, then some virtual processors will not be used by the program. If the number of threads is greater than the number of logical IDs specified by SUNW_MP_PROCBIND, them some virtual processors will have more than one thread bound to them.

Interaction with OS Processor Sets

A processor set can be specified using the psrset utility on Solaris platforms, or the taskset command on Linux platforms. SUNW_MP_PROCBIND does not take processor sets into account. If the programmer uses processor sets, then it is their responsibility to ensure that the setting of SUNW_MP_PROCBIND is consistent with the processor set used. Otherwise, the setting of SUNW_MP_PROCBIND will override the processor set setting on Linux systems, while on Solaris systems an error message will be issued.

2.4 Stacks and Stack Sizes

The executing program maintains a main stack for the initial thread executing the program, as well as distinct stacks for each helper thread. Stacks are temporary memory address spaces used to hold arguments and automatic variables during invocation of a subprogram or function reference.

In general, the default main stack size is 8 megabytes. Compiling Fortran programs with the f95 -stackvar option forces the allocation of local variables and arrays on the stack as if they were automatic variables. Use of -stackvar with OpenMP programs is implied However, this may lead to stack overflow if not enough memory is allocated for the stack.

Use the limit C-shell command, or the ulimit ksh/sh command, to display or set the size of the main stack.

Each helper thread of an OpenMP program has its own thread stack. This stack mimics the initial (or main) thread stack but is unique to the thread. The thread’s PRIVATE arrays and variables (local to the thread) are allocated on the thread stack. The default size is 4 megabytes on 32-bit SPARC V8 and x86 platforms, and 8 megabytes on 64-bit SPARC V9 and x86 platforms. The size of the helper thread stack is set with the OMP_STACKSIZE environment variable.

demo% setenv OMP_STACKSIZE 16384   <-Set thread stack size to 16 Mb (C shell)

demo$ OMP_STACKSIZE=16384          <-Same, using Bourne/Korn shell
demo$ export OMP_STACKSIZE

Finding the best stack size might have to be determined by trial and error. If the stack size is too small for a thread to run it may cause silent data corruption in neighboring threads, or segmentation faults. If you are unsure about stack overflows, compile your Fortran, C, or C++ programs with the -xcheck=stkovf compiler option to force a segmentation fault on stack overflow. This stops the program before any data corruption can occur.

2.5 Checking the Correctness of OpenMP Programs

The following are various mechanisms and tools that are available for checking the correctness of an OpenMP program.

• You can use the -xvpara C/C++/Fortran option to display compiler parallelization messages

• You can use the Sun Studio dbx tool to debug C, C++, and Fortran OpenMP programs. An OpenMP program should first be prepared for debugging with dbx by compiling it with the options -xopenmp=noopt -g . dbx allows the programmer to single-step into a PARALLEL region, set breakpoints in the body of an OpenMP construct, as well as print the values of shared, private, and other. variables for a given thread.

• You can use the SUNW_MP_WARN environment variable to enable runtime error checking and the issuing of warning messages by the OpenMP runtime library.

• You can check your OpenMP program for data races and deadlocks by using the Sun Studio Thread Analyzer tool. Refer to the Thread Analyzer manual and the tha(1) man page for details.