Overview of the XIL Test Suite

1

The XIL Test Suite enables you, as a system developer, to verify that new functionality (devices and molecules) that you have added to the XIL Imaging Library is performing accurately. The XIL Test Suite provides a set of test programs to verify XIL functionality and provides a set of functions that enable you to write your own test programs.

System developers may find the XIL Test Suite useful during certain phases of the development process; however, using the XIL Test Suite is not a complete testing solution. In general, it is most useful for developers who are writing XIL molecules or who are writing handlers for compute devices such as accelerators. In these cases, you might choose to test your code against reference data.

The usefulness of the XIL Test Suite to your development process depends on the product you are developing:

If you are writing a compute device handler, you can use the XIL Test Suite to verify that your version of the XIL atomic functions is equivalent to the XIL implementation.
If you are writing XIL molecules, you can write additional test programs that would invoke your molecule and test it against the corresponding atomic functions in the XIL Imaging Library.
If you are developing an input device handler, the XIL Test Suite is not useful in directly testing it.

If you are developing a storage device handler, the XIL Test Suite cannot directly test it, but it can test an associated compute handler. If the tests for the operators that use the storage handler pass for a variety of images, it is an indication that the storage device handler is performing properly.

For more information on these XIL device types, consult the XIL Device Porting and Extensibility Guide.

dl>

Test Programs

The XIL Test Suite test programs were designed to test XIL Imaging Library functions using a wide range of input parameters. Stored versions of data, referred to as reference data, were produced when running the test programs to verify the XIL functions. Reference data are images and colormaps, for example. Because reference data requires large amounts of disk space for storage, reference signatures were created from the reference data. These reference signatures are provided with the XIL Test Suite for use in regression testing. For information about how the XIL Test Suite uses test programs and references, see Chapter 2, "Running Test Programs in the XIL Test Suite."

Because of the low-level nature of the XIL Imaging Library, it is critical that the functions in this library perform as expected. Therefore, the code for test programs was reviewed to ensure accuracy, and reference data was verified by visual inspection.

Reference Data

You use reference data to verify the accuracy of the development library. Again, reference data is stored versions of data produced by tests programs and has been verified as accurate. Therefore, reference data can be used to debug development code.

Reference data is not provided but can be generated, as discussed in Chapter 2, "Running Test Programs in the XIL Test Suite." The primary disadvantage of using reference data is the large amount of disk space required if images are saved for all useful permutations of image sizes, pixel types, and other parameters. The disk space used by these images can be tens of gigabytes.

Also, to verify the accuracy of development code, you can specify a tolerance range for comparisons when you run the XIL Test Suite. With a specified tolerance range, you can successfully verify data even if there are minor

discrepancies caused by floating point precision effects. These discrepancies occur mostly in operations such as convolution and

compression/decompression.

New options have been added to the XIL Test Suite for improvement of image comparison tolerance. An absolute and relative tolerance between the current and reference images can be defined.

It is also possible to save discrepancies between current and reference images for later visual inspection of discrepancies or for printing the differences between them.

Reference Signatures

Reference signatures are used for regression testing. Reference signatures represent a Cyclic Redundancy Codes (CRC) checksum derived from reference data. For more information on the CRC algorithm used in the XIL Test Suite, see the document entitled "A Painless Guide to CRC Error Detection Algorithms" by Ross Williams. You can locate this document on the web at ftp://ftp.rocksoft.com/clients/rocksoft. A reference signature uniquely identifies the data, just as a person's signature uniquely identifies that person. Reference signatures are used to avoid storing a large amount of data.

Regression testing is performed by producing a signature for the functions being tested and comparing that signature to the corresponding stored reference signature. If differences are found, errors are assumed to be in the functions being tested.

Benchmarking

The XIL Test Suite includes functions that allow you to create benchmarking programs to measure the performance of XIL functions. Performance is measured in frames per second. Chapter 3, "Writing Test Programs in the XIL Test Suite Environment" includes a discussion of how to write benchmarking programs.

What Is Tested?

To thoroughly verify each function, the XIL Test Suite provides tests that are appropriate to the function and to the platform on which you are running. The following three sections, "Test Conditions," "Deferred Execution," and "Architecture/Platform Testing" discuss XIL function testing.

Test Conditions

The XIL Test Suite uses a set of general and function-specific test conditions to test each function. In the general case, all functions are tested under permutations of the following conditions when possible and appropriate:

All pixel types
Various image sizes
Various numbers of bands
Various image origins
With and without regions of interest (ROIs)
Images as parent and child images
With random (noise) and ordered images as the source

Many of the functions are tested with function-specific test conditions. For example, the geometric operations test for additional parameters, such as rotation angles, scale factors, and interpolation types.

Deferred Execution

In general, testing of the deferred execution mechanism is no different from testing a particular function, except that you verify images after a group of function calls instead of after individual function calls. Consult the XIL Programmer's Guide for information on deferred execution.

Because the current release of the XIL Library is multithreaded and supports tiled images, control options are added to support testing XIL's threading and tiling performance.

Architecture/Platform Testing

You can verify XIL functions on all supported platforms and architectures. To verify XIL functions on a particular platform, you must run test programs on the platform you wish to verify. By doing this, you verify the consistency of test cases from platform to platform.

iv>

The Parts of the XIL Test Suite

The XIL Test Suite has three parts:

Xilch, the master control program, or harness, that runs all of the tests in the test suite.
libts, the test suite library that provides utility functions and functions to generate and verify data. Also, you can write your own test program using these functions.
A set of test programs, which are dynamically linked with functions in libts, that verify the accuracy of the XIL functions being tested.

Figure 1-1 shows the test suite's parts.

Figure 1-1

Test programs consist of libts functions. These functions are used to exercise the XIL function that is being verified. You can invoke a test program through the Xilch harness, or you can run the test program by itself.

The Xilch harness accepts a variety of options that allow you to precisely control what is being tested or created. See Chapter 2, "Running Test Programs in the XIL Test Suite," for a detailed discussion about how the XIL Test Suite works.