215 lines
10 KiB
Plaintext
Executable File
215 lines
10 KiB
Plaintext
Executable File
/*!
|
|
|
|
\mainpage Visual and Interactive Test Automation for SDL 2.0
|
|
|
|
\section license_sec License
|
|
Check the file \c LICENSE.txt for licensing information.
|
|
|
|
\section intro_sec Introduction
|
|
The goal of this GSoC project is to automate the testing of testsprite2.
|
|
testsprite2 takes 26 parameters which have thousands of valid combinations and is
|
|
used to validate SDL's window, mouse and rendering behaviour. By having a test
|
|
harness that runs testsprite2 with various command line argument strings and
|
|
validates the output for each run, we can make testing an easier task for
|
|
maintainers, contributors and testers. The test harness can be used by a continuous
|
|
integration system (like buildbot or jenkins) to validate SDL after checkins.
|
|
|
|
SDL Homepage: http://libsdl.org/
|
|
|
|
\section build_sec Building
|
|
|
|
\subsection build_linux Building on Linux/Cygwin
|
|
<tt>./autogen.sh; ./configure; make;</tt>
|
|
|
|
\subsection build_windows Building on Windows
|
|
Use the Visual Studio solution under \c SDL/VisualC/visualtest.
|
|
|
|
\section docs_sec Documentation
|
|
Documentation is available via Doxygen. To build the documentation, cd to the
|
|
SDL/visualtest/docs directory and run \c doxygen. A good starting point for
|
|
exploring the documentation is \c SDL/visualtest/docs/html/index.html
|
|
|
|
\section usage_sec Usage
|
|
To see all the options supported by the test harness, just run \c testharness
|
|
with no arguments.
|
|
|
|
At the moment the following options are supported:
|
|
\li \c sutapp - Path to the system under test (SUT) application
|
|
\li \c sutargs - Launch the SUT with the specified arguments string
|
|
\li \c timeout - The maximum time after which the SUT process will be killed;
|
|
passed as hh:mm:ss; default 00:01:00
|
|
\li \c variator - Which variator to use; see \ref variators_sec
|
|
\li \c num-variations - The number of variations to run for; taken to be
|
|
1 for the random variator and ALL for the exhaustive variator by default
|
|
\li \c no-launch - Just print the arguments string for each variation without
|
|
launching the SUT or performing any actions
|
|
\li \c parameter-config - A config file that describes the command line parameters
|
|
supported by the SUT; see \ref paramconfig_sec or the sample *.parameters files
|
|
for more details
|
|
\li \c action-config - A config file with a list of actions to be performed while
|
|
the SUT is running; see \ref actionconfig_sec or the sample *.actions files
|
|
\li \c output-dir - Path to the directory where screenshots should be saved; is
|
|
created if it doesn't exist; taken to be "./output" by default
|
|
\li \c verify-dir - Path to the directory with the verification images; taken to
|
|
be "./verify" by default
|
|
|
|
Paths can be relative or absolute.
|
|
|
|
Alternatively, the options can be passed as a config file for convenience:
|
|
|
|
<tt>testharness \-\-config testsprite2_sample.config</tt>
|
|
|
|
For a sample, take a look at the *.config files in this repository.
|
|
|
|
We can also pass a config file and override certain options as necessary:
|
|
<tt>testharness \-\-config testsprite2_sample.config \-\-num-variations 10</tt>
|
|
|
|
Note: You may find it convenient to copy the SUT executable along with any
|
|
resources to the test harness directory. Also note that testsprite2 and its
|
|
resources (icon.bmp) are automatically copied when using the Visual Studio
|
|
solution.
|
|
|
|
\subsection usageexamples_subsec Usage examples:
|
|
|
|
Passing a custom arguments string:
|
|
<tt>testharness \-\-sutapp testsprite2 \-\-sutargs "\-\-cyclecolor \-\-blend mod
|
|
\-\-iterations 2" \-\-action-config xyz.actions</tt>
|
|
|
|
Using the random variator:
|
|
<tt>testharness \-\-sutapp testsprite2 \-\-variator random \-\-num-variations 5
|
|
\-\-parameter-config xyz.parameters \-\-action-config xyz.actions</tt>
|
|
|
|
\subsection config_subsec Config Files
|
|
Config files are an alternate way to pass parameters to the test harness. We
|
|
describe the paramters in a config file and pass that to the test harness using
|
|
the \-\-config option. The config file consists of lines of the form "x=y" where
|
|
x is an option and y is it's value. For boolean options, we simply give the name
|
|
of the option to indicate that it is to be passed to the testharness.
|
|
|
|
The hash '#' character can be used to start a comment from that point to the end
|
|
of the line.
|
|
|
|
\section paramconfig_sec The SUT Parameters File
|
|
To generate variations we need to describe the parameters the will be passed to
|
|
the SUT. This description is given in a parameters file. Each line of the parameters
|
|
file (except the blank lines) represents one command line option with five
|
|
comma separated fields:
|
|
<tt>name, type, values, required, categories</tt>
|
|
|
|
\li \c name is the name of the option, e.g., \c \-\-cyclecolor.
|
|
\li \c type can have one of three values - integer, boolean and enum.
|
|
\li \c values - for integer options this is the valid range of values the option
|
|
can take, i.e., [min max]. For enum options this is a list of strings that
|
|
the option can take, e.g., [val1 val2 val3]. For boolean options this field
|
|
is ignored.
|
|
\li \c required - true if the option is required, false otherwise.
|
|
\li \c categories - a list of categories that the option belongs to. For example,
|
|
[video mouse audio]
|
|
|
|
Just like with config files, hash characters can be used to start comments.
|
|
|
|
\subsection additionalnotes_subsec Additional Notes
|
|
|
|
\li If you want to have an option that always takes a certain value, use an enum
|
|
with only one value.
|
|
\li Currently there isn't any way to turn an option off, i.e., all options will
|
|
be included in the command line options string that is generated using the
|
|
config. If you don't want an option to be passed to the SUT, remove it from
|
|
the config file or comment it out.
|
|
|
|
\section variators_sec Variators
|
|
Variators are the mechanism by which we generate strings of command line arguments
|
|
to test the SUT with. A variator is quite simply an iterator that iterates through
|
|
different variations of command line options. There are two variators supported at
|
|
the moment:
|
|
\li \b Exhaustive - Generate all possible combinations of command line arguments
|
|
that are valid.
|
|
\li \b Random - Generate a random variation each time the variator is called.
|
|
|
|
As an example, let's try a simple .parameters file:\n
|
|
<tt>
|
|
\-\-blend, enum, [add mod], false, [] \n
|
|
\-\-fullscreen, boolean, [], false, []
|
|
</tt>
|
|
|
|
The exhaustive variator would generate the following four variations:\n
|
|
<tt>
|
|
\-\-blend add \n
|
|
\-\-blend mod \n
|
|
\-\-blend add \-\-fullscreen \n
|
|
\-\-blend mod \-\-fullscreen \n
|
|
</tt>
|
|
|
|
The random variator would simply generate a random variation like the following:\n
|
|
<tt>\-\-blend mod</tt>
|
|
|
|
\section actionconfig_sec The Actions File
|
|
Once the SUT process has been launched, automated testing happens using a mechanism
|
|
called actions. A list of actions is read from a file and each action is performed
|
|
on the SUT process sequentially. Each line in the actions file describes an action.
|
|
The format for an action is <tt>hh:mm:ss ACTION_NAME additional parameters</tt>.
|
|
There are five actions supported at the moment:
|
|
\li \b SCREENSHOT - Takes a screenshot of each window owned by the SUT process. The
|
|
images are saved as \c [hash]_[i].bmp where \c [hash] is the 32 character long
|
|
hexadecimal MD5 hash of the arguments string that was passed to the SUT while
|
|
launching it and \c i is the window number. i = 1 is an exceptional case
|
|
where the \c _[i] is dropped and the filename is simply \c [hash].bmp\n
|
|
Note: The screenshots are only of the window's client area.
|
|
\li \b VERIFY - Verifies the screenshots taken by the last SCREENSHOT action by
|
|
comparing them against a verification image. Each \c [hash]_i.bmp image output
|
|
by the SCREENSHOT action is compared against a \c [hash].bmp image in the
|
|
verify-dir.
|
|
\li \b QUIT - Gracefully quits the SUT process. On Windows this means sending a
|
|
WM_CLOSE message to each window owned by the SUT process. On Linux it means
|
|
sending a SIGQUIT signal to the SUT process.
|
|
\li \b KILL - Forcefully kills the SUT process. This is useful when the SUT process
|
|
doesn't respond to the QUIT action.
|
|
\li <b>LAUNCH [/path/to/executable] [args]</b> - Runs an executable with \c [args]
|
|
as the arguments string.
|
|
|
|
Just like with config files, hash characters can be used to start comments.
|
|
|
|
\section contint_sec Continuous Integration (CI)
|
|
One of the goals of the project was to create a test harness that integrates
|
|
with CI systems to provide automated visual and interactive testing to SDL.
|
|
|
|
At the moment the test harness can be run in two modes that are useful for CI:
|
|
\li Crash testing mode - launch the SUT with every variation and all parameters,
|
|
report to the CI if there's a crash
|
|
\li Visual testing mode - launch and visually verify the SUT for a smaller subset
|
|
of the parameters
|
|
|
|
Look at the launch_harness.sh/launch_harness.cmd for an example scripts that run the
|
|
test harness for all variations with all parameters and report an error on a crash.
|
|
The script uses the testsprite2_crashtest config, so remember to copy those files
|
|
over to the test harness executable directory along with the script.
|
|
|
|
\section todo_sec TODOs
|
|
\li Allow specifying a clipping box along with the VERIFY action, i.e., hh:mm:ss
|
|
VERIFY x, y, w, h
|
|
\li Add support for spaces between the equals sign in test harness config files
|
|
\li Implement the SCREENSHOT action on Linux
|
|
\li Add a pairwise variator
|
|
\li Add actions to inject keyboard/mouse events
|
|
\li Add actions to manipulate the SUT window, e.g., minimize, restore, resize
|
|
\li Add support to load and save screenshots as .pngs instead of .bmps
|
|
|
|
\section issues_sec Known Issues
|
|
\li The QUIT action does not work on a testsprite2 process with multiple windows.
|
|
This appears to be an issue with testsprite2.
|
|
\li The SCREENSHOT action doesn't capture the testsprite2 window correctly if the
|
|
--fullscreen option is supplied. It works with --fullscreen-desktop, however.
|
|
|
|
\section moreinfo_sec More Information
|
|
|
|
Author Contact Info:\n
|
|
Apoorv Upreti \c \<apoorvupreti@gmail.com\>
|
|
|
|
Other useful links:
|
|
- Project Repository: https://bitbucket.org/nerdap/sdlvisualtest
|
|
- Project Wiki: https://github.com/nerdap/autotestsprite2/wiki
|
|
- Project Blog: http://nerdap.github.io
|
|
- Verification images for testsprite2_blendmodes: https://www.dropbox.com/s/nm02aem76m812ng/testsprite2_blendmodes.zip
|
|
- Verification images for testsprite2_geometry: https://www.dropbox.com/s/csypwryopaslpaf/testsprite2_geometry.zip
|
|
*/
|