ref: 466cf20d3524b8e42edc333a6d2df2a01e99a95b
dir: /sys/src/cmd/gs/doc/Testing.htm/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <html> <head> <title>Using and writing Ghostscript testing scripts</title> <!-- $Id: Testing.htm,v 1.37 2005/10/20 19:46:23 ray Exp $ --> <link rel="stylesheet" type="text/css" href="gs.css" title="Ghostscript Style"> </head> <body> <!-- [1.0 begin visible header] ============================================ --> <!-- [1.1 begin headline] ================================================== --> <h1>Using and writing Ghostscript testing scripts</h1> <!-- [1.1 end headline] ==================================================== --> <!-- [1.2 begin table of contents] ========================================= --> <!-- [1.2 end table of contents] =========================================== --> <h2>Table of contents</h2> <blockquote><ul> <li><a href="#General_overview">General overview</a> <li><a href="#Running_tests">Running tests</a> <ul> <li><a href="#Individual_tests">Individual tests</a> <li><a href="#Regression_testing">Regression testing</a> </ul> <li><a href="#Writing_new_tests">Writing new tests</a> </ul></blockquote> <!-- [1.3 begin hint] ====================================================== --> <p> This document describes how to use the scripts located in the <a href="../toolbin/tests">toolbin/tests</a> directory, and conventions for writing new testing scripts. <p>For other information, see the <a href="Readme.htm">Ghostscript overview</a> and the documentation related to <a href="Maintain.htm">maintaining Ghostscript</a>. <!-- [1.3 end hint] ======================================================== --> <hr> <!-- [1.0 end visible header] ============================================== --> <!-- [2.0 begin contents] ================================================== --> <h2><a name="General_overview"></a>General overview</h2> <p> The test scripts discussed here are written in Python, a language whose implementation is freely available from <a class="offsite" href="http://www.python.org"><tt>http://www.python.org</tt></a>. The scripts require Python version 2.1 or later. <h2><a name="Running_tests"></a>Running tests</h2> <p> On Unix and Linux systems, test scripts written in Python can be executed directly simply by typing their name into the shell, e.g., <blockquote><pre> toolbin/tests/check_source.py </pre></blockquote> <p> On other systems, it may be necessary to invoke Python explicitly, e.g., <blockquote><pre> python toolbin/tests/check_source.py </pre></blockquote> <p> The test scripts will print information about any failures that occur. <h3><a name="Individual_tests"></a>Individual tests</h3> <p> The individual test scripts are named <b><tt>toolbin/tests/check_</tt></b><em>xxx</em><b><tt>.py</tt></b> (if they do not run Ghostscript) or <b><tt>toolbin/tests/gscheck_</tt></b><em>xxx</em><b><tt>.py</tt></b> (if they do run Ghostscript). Any script with such a name can be run individually, and is also normally run as part of regression testing (described in the next section). <p> We don't list the individual test scripts here, because any such documentation would inevitably be out of date most of the time. Each of these scripts contains documentation about what it tests: we suggest you read the documentation in the scripts. <h3><a name="Regression_testing"></a>Regression testing</h3> <p> We run a nightly regression test to discover any obvious problems caused by code checked in the previous day. Here is a list of the scripts and supporting files that make up the regression test. <h4>Top-level scripts</h4> <dl> <dl> <dt><a href="../toolbin/tests/dump_testdb"><tt>dump_testdb</tt></a> <dd>A debugging script that will print the contents of the database defined by gsconf.testdatadb. </dl> <dl> <dt><a href="../toolbin/tests/make_testdb"><tt>make_testdb</tt></a> <dd>This script creates an initial test database. It uses gsconf.baselinegs to create raster data from the test files and computes their MD5 sums and stores them in the gsconf.testdatadb database. </dl> <dl> <dt><a href="../toolbin/tests/make_two_versions"><tt>make_two_versions</tt></a> <dd>A helper script to make two versions of a particular file for visual diffing or manual analysis. When a test fails nightly regression, this is generally the first investigative step. </dl> <dl> <dt><a href="../toolbin/tests/make_two_pdfversions"><tt>make_two_pdfversions</tt></a> <dd>Same as above, except for pdfwrite regressions. </dl> <dl> <dt><a href="../toolbin/tests/testdiff"><tt>testdiff</tt></a> <dd>this script provides the difference between two sets of regression results. if end date is omitted, the current date will be used. </dl> <dl> <dt><a href="../toolbin/tests/revert_baseline"><tt>revert_baseline</tt></a> <dd>In cases where a baseline has been accidentally updated, this script will revert the database entry to the MD5 sum computed with gsconf.baselinegs. </dl> <dl> <dt><a href="../toolbin/tests/revert_pdfbaseline"><tt>revert_pdfbaseline</tt></a> <dd>Same as above, except for pdfwrite baselines. </dl> <dl> <dt><a href="../toolbin/tests/run_nightly"><tt>run_nightly</tt></a> <dd>This is the control script (usually invoked by cron) that controls the nightly test run. It's responsible for checking the latest code out of CVS, building a new Ghostscript to compare with and launching the test suite via run_regression. </dl> <dl> <dt><a href="../toolbin/tests/run_regression"><tt>run_regression</tt></a> <dd>This script runs the full gamut of regression tests using files from gsconf.comparedir. It differentiates files by extension and controls what tests get run and with what options. </dl> <dl> <dt><a href="../toolbin/tests/update_baseline"><tt>update_baseline</tt></a> <dd>This script is invoked to update the MD5 sum in the test database when a nightly regression is really a progression. Generally after noticing that the output from make_two_versions is acceptable or better, this script is run to log the changes to the database. </dl> <dl> <dt><a href="../toolbin/tests/update_pdfbaseline"><tt>update_pdfbaseline</tt></a> <dd>Same as above, except for pdfwrite baselines. </dl> <dl> <dt><a href="../toolbin/tests/update_specific"><tt>update_specific</tt></a> <tt><em>cvs-date-spec</em></tt> <dd>Update a series of baselines to a specific datestamp; accepts a series of "flag filename" lines on stdin, where flag is either N or P to specify whether the normal or pdfwrite baselines should be updated. <dl> <dt><a href="../toolbin/tests/get_baselines"><tt>get_baselines</tt></a> <dd>This script returns the change log for the test database starting at a given date. </dl> </dl> <h4>Support files</h4> <dl> <dl> <dt><a href="../toolbin/tests/fuzzy.c"><tt>fuzzy.c</tt></a> <dd>A fuzzy comparison tool appropriate for tests where exact binary matches aren't appropriate. </dl> <dl> <dt><a href="../toolbin/tests/cmpi.py"><tt>cmpi.py</tt></a> <dd>An image comparison tool for exploring the differences between two raster files. It can switch between the two, display only the differences and localize, highlight and jump between areas where differences occur. This can be a useful way of examining regression difference to decide if they are progressions or regressions. Requires python imaging and python Tkinter. </dl> <dl> <dt><a href="../toolbin/tests/gsconf.py"><tt>gsconf.py</tt></a> <dd>This is the configuration framework for the scripts above. It reads configuration files and makes available those configuration options to the rest of the testing framework. </dl> <dl> <dt><a href="../toolbin/tests/testing.cfg.example"><tt>testing.cfg.example</tt></a> <dd>This is an example configuration file for the scripts above. It controls where files are found, where Ghostscript executables are and the location of the test database. Most test configuration will be in this file. It must be copied to testing.cfg in order for the tests to find it. </dl> <dl> <dt><a href="../toolbin/tests/testing.cfg"><tt>testing.cfg</tt></a> <dd>This is the configuration file for the scripts above. It controls where files are found, where Ghostscript executables are and the location of the test database. Most test configuration will be in this file. The file testing.cfg.example above can be used as a template. </dl> <dl> <dt><a href="../toolbin/tests/gstestgs.py"><tt>gstestgs.py</tt></a> <dd>This provides classes for running tests that actually execute Ghostcript. </dl> <dl> <dt><a href="../toolbin/tests/gsparamsets.py"><tt>gsparamsets.py</tt></a> <dd>What parameters tests get run with by default are stored in this file. Between this configuration information and the information in gsconf.py all configurable testing parameters should be covered. </dl> <dl> <dt><a href="../toolbin/tests/gsutil.py"><tt>gsutil.py</tt></a> <dd>Various utility routines used by the regression test scripts. </dl> <dl> <dt><a href="../toolbin/tests/gssum.py"><tt>gssum.py</tt></a> <dd>Helper functions that compute, compare and store MD5 sums. </dl> <dl> <dt><a href="../toolbin/tests/rasterdb.py"><tt>rasterdb.py</tt></a> <dd>Hepler functions for accessing the raster database. </dl> </dl> <h2><a name="Writing_new_tests"></a>Writing new tests</h2> <p> Some of Ghostscript's test scripts follow a set of conventions that allow them to be run either stand-alone or as part of a suite; in particular, they can be run as part of the nightly regression test suite. In this section, we provide pointers to documentation on how to write new tests that follow these conventions, since that will make them the most useful. <p> The test scripts are based on Python's <b><tt>unittest</tt></b> module. We suggest that if you are not familiar with this module, you read the documentation, which is available at <a class="offsite" href="http://www.python.org/doc/current/lib/module-unittest.html">http://www.python.org/doc/current/lib/module-unittest.html</a>. <p> Ghostscript specializes the <b><tt>unittest</tt></b> module by defining subclasses, which all individual tests use in place of those in <b><tt>unittest</tt></b>. These subclasses are defined in <a href="../toolbin/tests/gstestutils.py">toolbin/tests/gstestutils.py</a>. <p> Since code documentation separate from the code itself is always out of date, we have decided to maintain the primary documentation for writing new tests in <a href="../toolbin/tests/gstestutils.py">gstestutils.py</a> rather than here in a separate document. Please read that file for more information. <!-- [2.0 end contents] ==================================================== --> <!-- [3.0 begin visible trailer] =========================================== --> <hr> <p> <small>Copyright © 2002 artofcode LLC. All rights reserved.</small> <p> <small>This file is part of AFPL Ghostscript. See the <a href="Public.htm">Aladdin Free Public License</a> (the "License") for full details of the terms of using, copying, modifying, and redistributing AFPL Ghostscript.</small> <p> <small>Ghostscript version 8.53, 20 October 2005 <!-- [3.0 end visible trailer] ============================================= --> </body> </html>