| Home > Hacking Invenio > Test Suite Strategy |
This documents presents guidelines for unit testing and regression testing homogenisation throughout all Invenio modules.
Testing is an important coding activity. Most authors believe that writing test cases should take between 10% and 30% of the project time. But, even with such a large fraction, don't put too much belief on such a testing. It cannot find bugs that aren't tested for. So, while testing is an important activity inherent to safe software development practices, it cannot become a substitute for pro-active bug hunting, source code inspection, and bugfree-driven development approach from the start.
Testing should happen alongside with coding. If you write a function, immediately load it into your toplevel, evaluate its definition, and call it for a couple of arguments to make sure the function works as expected. If not, then change the function definition, re-evaluate it, re-call it, etc. Dynamic languages with interactive toplevel such as Common Lisp or Python makes this easy for you. Dynamic redefinition capabilities (full in Common Lisp, partial in Python) are very programmer-friendly in this respect. If your test cases are interesting to be kept, then keep them in a test file. (It's almost all the time a good idea to store them in the test file, since you cannot predict whether you won't want to change something in the future.) We'll see below how to store your tests in a test file.
When testing, it is nice to know some rules of thumb, like: check your edge cases (e.g. null array), check atypical input values (e.g. laaarge array instead of typically 5-6 elements only), check your termination conditions, ask whether your arguments have already been safe-proofed or whether it is in your mandate to check them, write a test case for each `if-else' branch of the code to explore all the possibilites, etc. Another interesting rule of thumb is the bug frequency distribution. Experience has shown that the bugs tend to cluster. If you discover a bug, there are chances that other bugs are in the neighborhood. The famous 80/20 rule of thumb applies here too: about 80% of bugs are located in about 20% of the code. Another rule of thumb: if you find a bug caused by some coding practice pattern thay may be used elsewhere too, look and fix other pattern instances.
In a nutshell, the best advice to write bug-free code is: think ahead. Try to prepare in advance for unusual usage scenarios, to foresee problems before they happen. Don't rely on typical input and typical usage scenarios. Things have a tendency to become atypical one day. Recall that testing is necessary, but not sufficient, to write good code. Therefore, think ahead!
Core functionality, such as the hit set intersection for the search engine, or the text input manipulating functions of the BibConvert language, should come with a couple of test cases to assure proper behaviour of the core functionality. The test cases should cover typical input (e.g. hit set corresponding to the query for ``ellis''), as well as the edge cases (e.g. empty/full hit set) and other unusual situations (e.g. non-UTF-8 accented input for BibConvert functions to test a situation of different number of bytes per char).
The test cases should be written for most important core functionality. Not every function or class in the code is to be throughly tested. Common sense will tell.
Unit test cases are free of side-effects. Users should be able to run them on production database without any harm to their data. This is because the tests test ``units'' of the code, not the application as such. If the behaviour of the function you would like to test depends on the status of the database, or some other parameters that cannot be passed to the function itself, the unit testing framework is not suitable for this kind of situation and you should use the regression testing framework instead (see below).
For more information on Pythonic unit testing, see the documentation to the unittest module at http://docs.python.org/lib/module-unittest.html. For a tutorial, see for example http://diveintopython.org/unit_testing/.
Each core file that is located in the lib directory (such as the
webbasketlib.py in the example above) should come with a
testing file where the test cases are stored. The test file is to be
named identically as the lib file it tests, but with the suffix
_unit_tests (in our example,
webbasketlib_unit_tests.py).
The test cases are written using Pythonic unittest TestCase class. An example for testing search engine query parameter washing function:
$ cat /opt/invenio/lib/python/invenio/search_engine_unit_tests.py
[...]
from invenio.websearch import search_engine
from invenio.miscutil.testutils import InvenioTestCase
class TestWashQueryParameters(InvenioTestCase):
"""Test for washing of search query parameters."""
def test_wash_url_argument(self):
"""search engine washing of URL arguments"""
self.assertEqual(1, search_engine.wash_url_argument(['1'],'int'))
self.assertEqual("1", search_engine.wash_url_argument(['1'],'str'))
self.assertEqual(['1'], search_engine.wash_url_argument(['1'],'list'))
self.assertEqual(0, search_engine.wash_url_argument('ellis','int'))
self.assertEqual("ellis", search_engine.wash_url_argument('ellis','str'))
self.assertEqual(["ellis"], search_engine.wash_url_argument('ellis','list'))
self.assertEqual(0, search_engine.wash_url_argument(['ellis'],'int'))
self.assertEqual("ellis", search_engine.wash_url_argument(['ellis'],'str'))
self.assertEqual(["ellis"], search_engine.wash_url_argument(['ellis'],'list'))
[...]
In addition, each test file is supposed to define a
TEST_SUITE variable that will return test suite with all
the tests available in this file:
$ cat /opt/invenio/lib/python/invenio/search_engine_unit_tests.py
[...]
TEST_SUITE = make_test_suite(TestWashQueryParameters,
TestStripAccents,
TestQueryParser,)
[...]
In this way, all the test cases defined in the file
search_engine_unit_tests.py will be executed when the global
testcase executable is called.
Note that it may be time-consuming to run all the tests in one go.
If you are interested in running tests only on a certain file (say
search_engine_unit_tests.py), then launch:
$ python /opt/invenio/lib/python/invenio/search_engine_unit_tests.py
For full-scale examples, you may follow
search_engine_unit_tests.py and other _unit_tests.py
files in the source distribution.
Invenio test suite can be run at any time via:
This command will run all available unit tests provided with CDS Invenio.$ /opt/invenio/bin/inveniocfg --run-unit-tests
The informative output is of the form:
In case of problems you will see failures like:Invenio v0.3.2.20040519 test suite results: =========================================== search engine washing of query patterns ... ok search engine washing of URL arguments ... ok search engine stripping of accented letters ... ok bibindex engine list union ... ok ---------------------------------------------------------------------- Ran 4 tests in 0.121s OK
Invenio v0.3.2.20040519 test suite results:
===========================================
search engine washing of query patterns ... FAIL
search engine washing of URL arguments ... ok
search engine stripping of accented letters ... ok
bibindex engine list union ... ok
======================================================================
FAIL: search engine washing of query patterns
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/invenio/lib/python/invenio/search_engine_unit_tests.py", line 25, in test_wash_pattern
self.assertEqual("ell*", search_engine.wash_pattern('ell*'))
File "/usr/lib/python2.3/unittest.py", line 302, in failUnlessEqual
raise self.failureException, \
AssertionError: 'ell*' != 'ell'
----------------------------------------------------------------------
Ran 4 tests in 0.091s
FAILED (failures=1)
The test suite compliance should be checked before each CVS commit. (And, obviously, double-checked before each Invenio release.)
In addition to the above-mentioned unit testing of important functions, a regression testing should ensure that the overall application functionality is behaving well and is not altered by code changes. This is especially important if a bug had been previously found. Then a regression test case should be written to assure that it will never reappear. (It also helps to scan the neighborhood of the bug, or the whole codebase for occurrences of the same kind of bug, see the 80/20 thumb rule cited above.)
Moreover, the regression test suite should be used when the functionality of the item we would like to test depends on extra-parametrical status, such as the database content. Also, the regression framework is suitable for testing the overall behaviour of web pages. (In extreme programming, the regression testing is called acceptance testing, the name that evolved from previous functionality testing.)
Within the framework of the regression test suite, we have liberty to alter database content, unlike that of the unit testing framework. We can also simulate the web browser in order to test web applications. (Note that for real browser tests, we have yet another web test suite described below.)
As an example of a regression test, we can test whether the web pages are alive; whether searching for Ellis in the demo site produces indeed 12 records; whether searching for aoeuidhtns produces no hits but the box of nearest terms, and with which content; whether accessing the Theses collection page search prompts an Apache password prompt; whether the admin interface is really accessible only to admins or also to guests, etc.
For more information on regression testing, see for example http://c2.com/cgi/wiki?RegressionTesting.
Regression tests are written per application (or sub-module) in
files named like websearch_regression_unit_tests.py or
websubmitadmin_regression_tests.py.
When writing regression tests, you can assume that the site is in the fresh demo mode (Atlantis Institute of Fictive Science). You can also safely write not only database-read-only tests, but you can also safely insert/update/delete into/from the database whatever values you need for testing. Users are warned prior to running the regression test suite about its possibly destructive side-effects. (See below.) Therefore you can create users, create user groups, attach users to groups to test the group joining process etc, as needed.
For testing web pages using GET arguments, you can take advantage of the following helper function:
$ cat /opt/invenio/lib/python/invenio/testutils.py [...] def test_web_page_content(url, username="guest", expected_text="