7. Tests

7.1. Unit tests

The CubicWeb framework provides the cubicweb.devtools.testlib.CubicWebTC test base class .

Tests shall be put into the mycube/test directory. Additional test data shall go into mycube/test/data.

It is much advised to write tests concerning entities methods, actions, hooks and operations, security. The CubicWebTC base class has convenience methods to help test all of this.

In the realm of views, automatic tests check that views are valid XHTML. See Automatic views testing for details.

Most unit tests need a live database to work against. This is achieved by CubicWeb using automatically sqlite (bundled with Python, see http://docs.python.org/library/sqlite3.html) as a backend.

The database is stored in the mycube/test/tmpdb, mycube/test/tmpdb-template files. If it does not (yet) exist, it will be built automatically when the test suite starts.

Warning

Whenever the schema changes (new entities, attributes, relations) one must delete these two files. Changes concerned only with entity or relation type properties (constraints, cardinalities, permissions) and generally dealt with using the sync_schema_props_perms() function of the migration environment do not need a database regeneration step.

7.1.1. Unit test by example

We start with an example extracted from the keyword cube (available from https://forge.extranet.logilab.fr/cubicweb/cubes/keyword).

from cubicweb.devtools.testlib import CubicWebTC
from cubicweb import ValidationError

class ClassificationHooksTC(CubicWebTC):

    def setup_database(self):
        with self.admin_access.repo_cnx() as cnx:
            group_etype = cnx.find('CWEType', name='CWGroup').one()
            c1 = cnx.create_entity('Classification', name=u'classif1',
                                   classifies=group_etype)
            user_etype = cnx.find('CWEType', name='CWUser').one()
            c2 = cnx.create_entity('Classification', name=u'classif2',
                                   classifies=user_etype)
            self.kw1eid = cnx.create_entity('Keyword', name=u'kwgroup', included_in=c1).eid
            cnx.commit()

    def test_cannot_create_cycles(self):
        with self.admin_access.repo_cnx() as cnx:
            kw1 = cnx.entity_from_eid(self.kw1eid)
            # direct obvious cycle
            with self.assertRaises(ValidationError):
                kw1.cw_set(subkeyword_of=kw1)
            cnx.rollback()
            # testing indirect cycles
            kw3 = cnx.execute('INSERT Keyword SK: SK name "kwgroup2", SK included_in C, '
                              'SK subkeyword_of K WHERE C name "classif1", K eid %(k)s'
                              {'k': kw1}).get_entity(0,0)
            kw3.cw_set(reverse_subkeyword_of=kw1)
            self.assertRaises(ValidationError, cnx.commit)

The test class defines a setup_database() method which populates the database with initial data. Each test of the class runs with this pre-populated database.

The test case itself checks that an Operation does its job of preventing cycles amongst Keyword entities.

The create_entity method of connection (or request) objects allows to create an entity. You can link this entity to other entities, by specifying as argument, the relation name, and the entity to link, as value. In the above example, the Classification entity is linked to a CWEtype via the relation classifies. Conversely, if you are creating a CWEtype entity, you can link it to a Classification entity, by adding reverse_classifies as argument.

Note

the commit() method is not called automatically. You have to call it explicitly if needed (notably to test operations). It is a good practice to regenerate entities with entity_from_eid() after a commit to avoid request cache effects.

You can see an example of security tests in the Step 1: configuring security into the schema.

It is possible to have these tests run continuously using apycot.

7.1.1.1. Managing connections or users

Since unit tests are done with the SQLITE backend and this does not support multiple connections at a time, you must be careful when simulating security, changing users.

By default, tests run with a user with admin privileges. Connections using these credentials are accessible through the admin_access object of the test classes.

The repo_cnx() method returns a connection object that can be used as a context manager:

# admin_access is a pre-cooked session wrapping object
# it is built with:
# self.admin_access = self.new_access('admin')
with self.admin_access.repo_cnx() as cnx:
    cnx.execute(...)
    self.create_user(cnx, login='user1')
    cnx.commit()

user1access = self.new_access('user1')
with user1access.web_request() as req:
    req.execute(...)
    req.cnx.commit()

On exit of the context manager, a rollback is issued, which releases the connection. Don’t forget to issue the cnx.commit() calls!

Warning

Do not use references kept to the entities created with a connection from another one!

7.1.2. Email notifications tests

When running tests, potentially generated e-mails are not really sent but are found in the list MAILBOX of module cubicweb.devtools.testlib.

You can test your notifications by analyzing the contents of this list, which contains objects with two attributes:

  • recipients, the list of recipients

  • msg, email.Message object

Let us look at a simple example from the blog cube.

from cubicweb.devtools.testlib import CubicWebTC, MAILBOX

class BlogTestsCubicWebTC(CubicWebTC):
    """test blog specific behaviours"""

    def test_notifications(self):
        with self.admin_access.web_request() as req:
            cubicweb_blog = req.create_entity('Blog', title=u'cubicweb',
                                description=u'cubicweb is beautiful')
            blog_entry_1 = req.create_entity('BlogEntry', title=u'hop',
                                             content=u'cubicweb hop')
            blog_entry_1.cw_set(entry_of=cubicweb_blog)
            blog_entry_2 = req.create_entity('BlogEntry', title=u'yes',
                                             content=u'cubicweb yes')
            blog_entry_2.cw_set(entry_of=cubicweb_blog)
            self.assertEqual(len(MAILBOX), 0)
            req.cnx.commit()
            self.assertEqual(len(MAILBOX), 2)
            mail = MAILBOX[0]
            self.assertEqual(mail.subject, '[data] hop')
            mail = MAILBOX[1]
            self.assertEqual(mail.subject, '[data] yes')

7.1.3. Visible actions tests

It is easy to write unit tests to test actions which are visible to a user or to a category of users. Let’s take an example in the conference cube.

class ConferenceActionsTC(CubicWebTC):

    def setup_database(self):
        with self.admin_access.repo_cnx() as cnx:
            self.confeid = cnx.create_entity('Conference',
                                             title=u'my conf',
                                             url_id=u'conf',
                                             start_on=date(2010, 1, 27),
                                             end_on = date(2010, 1, 29),
                                             call_open=True,
                                             reverse_is_chair_at=chair,
                                             reverse_is_reviewer_at=reviewer).eid

    def test_admin(self):
        with self.admin_access.web_request() as req:
            rset = req.find('Conference').one()
            self.assertListEqual(self.pactions(req, rset),
                                  [('workflow', workflow.WorkflowActions),
                                   ('edit', confactions.ModifyAction),
                                   ('managepermission', actions.ManagePermissionsAction),
                                   ('addrelated', actions.AddRelatedActions),
                                   ('delete', actions.DeleteAction),
                                   ('generate_badge_action', badges.GenerateBadgeAction),
                                   ('addtalkinconf', confactions.AddTalkInConferenceAction)
                                   ])
            self.assertListEqual(self.action_submenu(req, rset, 'addrelated'),
                                  [(u'add Track in_conf Conference object',
                                    u'http://testing.fr/cubicweb/add/Track'
                                    u'?__linkto=in_conf%%3A%(conf)s%%3Asubject&'
                                    u'__redirectpath=conference%%2Fconf&'
                                    u'__redirectvid=' % {'conf': self.confeid}),
                                   ])

You just have to execute a rql query corresponding to the view you want to test, and to compare the result of pactions() with the list of actions that must be visible in the interface. This is a list of tuples. The first element is the action’s __regid__, the second the action’s class.

To test actions in a submenu, you just have to test the result of action_submenu() method. The last parameter of the method is the action’s category. The result is a list of tuples. The first element is the action’s title, and the second element the action’s url.

7.2. Automatic views testing

This is done automatically with the cubicweb.devtools.testlib.AutomaticWebTest class. At cube creation time, the mycube/test/test_mycube.py file contains such a test. The code here has to be uncommented to be usable, without further modification.

The auto_populate method uses a smart algorithm to create pseudo-random data in the database, thus enabling the views to be invoked and tested.

Depending on the schema, hooks and operations constraints, it is not always possible for the automatic auto_populate to proceed.

It is possible of course to completely redefine auto_populate. A lighter solution is to give hints (fill some class attributes) about what entities and relations have to be skipped by the auto_populate mechanism. These are:

  • no_auto_populate, may contain a list of entity types to skip

  • ignored_relations, may contain a list of relation types to skip

  • application_rql, may contain a list of rql expressions that auto_populate cannot guess by itself; these must yield resultsets against which views may be selected.

Warning

Take care to not let the imported AutomaticWebTest in your test module namespace, else both your subclass and this parent class will be run.

7.3. Cache heavy database setup

Some test suites require a complex setup of the database that takes seconds (or even minutes) to complete. Doing the whole setup for each individual test makes the whole run very slow. The CubicWebTC class offer a simple way to prepare a specific database once for multiple tests. The test_db_id class attribute of your CubicWebTC subclass must be set to a unique identifier and the pre_setup_database() class method must build the cached content. As the pre_setup_database() method is not garanteed to be called every time a test method is run, you must not set any class attribute to be used during test there. Databases for each test_db_id are automatically created if not already in cache. Clearing the cache is up to the user. Cache files are found in the data/database subdirectory of your test directory.

Warning

Take care to always have the same pre_setup_database() function for all classes with a given test_db_id otherwise your tests will have unpredictable results depending on the first encountered one.

7.4. Testing on a real-life database

The CubicWebTC class uses the cubicweb.devtools.ApptestConfiguration configuration class to setup its testing environment (database driver, user password, application home, and so on). The cubicweb.devtools module also provides a RealDatabaseConfiguration class that will read a regular cubicweb sources file to fetch all this information but will also prevent the database to be initalized and reset between tests.

For a test class to use a specific configuration, you have to set the _config class attribute on the class as in:

from cubicweb.devtools import RealDatabaseConfiguration
from cubicweb.devtools.testlib import CubicWebTC

class BlogRealDatabaseTC(CubicWebTC):
    _config = RealDatabaseConfiguration('blog',
                                        sourcefile='/path/to/realdb_sources')

    def test_blog_rss(self):
        with self.admin_access.web_request() as req:
            rset = req.execute('Any B ORDERBY D DESC WHERE B is BlogEntry, '
                               'B created_by U, U login "logilab", B creation_date D')
            self.view('rss', rset, req=req)

7.5. Testing with other cubes

Sometimes a small component cannot be tested all by itself, so one needs to specify other cubes to be used as part of the the unit test suite. This is handled by the bootstrap_cubes file located under mycube/test/data. One example from the preview cube:

card, file, preview

The format is:

  • possibly several empy lines or lines starting with # (comment lines)

  • one line containing a comma-separated list of cube names.

It is also possible to add a schema.py file in mycube/test/data, which will be used by the testing framework, therefore making new entity types and relations available to the tests.

7.6. Literate programming

CubicWeb provides some literate programming capabilities. The cubicweb-ctl tool shell command accepts different file formats. If your file ends with .txt or .rst, the file will be parsed by doctest.testfile with CubicWeb’s Migration API enabled in it.

Create a scenario.txt file in the test/ directory and fill with some content. Refer to the doctest.testfile documentation.

Then, you can run it directly by:

$ cubicweb-ctl shell <cube_instance> test/scenario.txt

When your scenario file is ready, put it in a new test case to be able to run it automatically.

from os.path import dirname, join
from logilab.common.testlib import unittest_main
from cubicweb.devtools.testlib import CubicWebTC

class AcceptanceTC(CubicWebTC):

        def test_scenario(self):
                self.assertDocTestFile(join(dirname(__file__), 'scenario.txt'))

if __name__ == '__main__':
        unittest_main()

7.6.1. Skipping a scenario

If you want to set up initial conditions that you can’t put in your unit test case, you have to use a KeyboardInterrupt exception only because of the way doctest module will catch all the exceptions internally.

>>> if condition_not_met:
...     raise KeyboardInterrupt('please, check your fixture.')

7.6.2. Passing paramaters

Using extra arguments to parametrize your scenario is possible by prepending them by double dashes.

Please refer to the cubicweb-ctl shell –help usage.

Important

Your scenario file must be utf-8 encoded.

7.7. Test APIS

7.7.1. Using Pytest

The pytest utility (shipping with logilab-common, which is a mandatory dependency of CubicWeb) extends the Python unittest functionality and is the preferred way to run the CubicWeb test suites. Bare unittests also work the usual way.

To use it, you may:

  • just launch pytest in your cube to execute all tests (it will discover them automatically)

  • launch pytest unittest_foo.py to execute one test file

  • launch pytest unittest_foo.py bar to execute all test methods and all test cases whose name contains bar

Additionally, the -x option tells pytest to exit at the first error or failure. The -i option tells pytest to drop into pdb whenever an exception occurs in a test.

When the -x option has been used and the run stopped on a test, it is possible, after having fixed the test, to relaunch pytest with the -R option to tell it to start testing again from where it previously failed.

7.7.2. Using the TestCase base class

The base class of CubicWebTC is logilab.common.testlib.TestCase, which provides a lot of convenient assertion methods.

class logilab.common.testlib.TestCase(methodName: str = 'runTest')[source]

A unittest.TestCase extension with some additional methods.

Create an instance of the class that will use the named test method when executed. Raises a ValueError if the instance does not have a method with the specified name.

classmethod datapath(*fname: str) str[source]

joins the object’s datadir and fname

innerSkip(msg: Optional[str] = None) mypy_extensions.NoReturn[source]

mark a generative test as skipped for the <msg> reason

maxDiff = None
optval(option, default=None)[source]

return the option value or default if the option is not define

set_description(descr)[source]

sets the current test’s description. This can be useful for generative tests because it allows to specify a description per yield

shortDescription() Optional[Any][source]

override default unittest shortDescription to handle correctly generative tests

7.7.3. CubicWebTC API

class cubicweb.devtools.testlib.CubicWebTC(*args, **kwargs)[source]

abstract class for test using an apptest environment

attributes:

  • vreg, the vregistry

  • schema, self.vreg.schema

  • config, cubicweb configuration

  • cnx, repoapi connection to the repository using an admin user

  • session, server side session associated to cnx

  • app, the cubicweb publisher (for web testing)

  • repo, the repository object

  • admlogin, login of the admin user

  • admpassword, password of the admin user

  • shell, create and use shell environment

  • anonymous_allowed: flag telling if anonymous browsing should be allowed

Create an instance of the class that will use the named test method when executed. Raises a ValueError if the instance does not have a method with the specified name.

admin_request_from_url(url)[source]

parses url and builds the corresponding CW-web request

req.form will be setup using the url’s query string

property app

return a cubicweb publisher

assertSentEmail(subject, recipients=None, nb_msgs=None)[source]

test recipients in system mailbox for given email subject

Parameters
  • subject – email subject to find in mailbox

  • recipients – list of email recipients

  • nb_msgs – expected number of entries

Returns

list of matched emails

configcls

alias of cubicweb.devtools.ApptestConfiguration

create_user(req, login=None, groups=('users',), password=None, email=None, commit=True, **kwargs)[source]

create and return a new user entity

ctrl_publish(req, ctrl='edit', rset=None)[source]

call the publish method of the edit controller

expect_redirect(callback, req)[source]

call the given callback with req as argument, expecting to get a Redirect exception

expect_redirect_handle_request(req, path='edit')[source]

call the publish method of the application publisher, expecting to get a Redirect exception

static fake_form(formid, field_dict=None, entity_field_dicts=())[source]

Build _cw.form dictionnary to fake posting of some standard cubicweb form

  • formid, the form id, usually form’s __regid__

  • field_dict, dictionary of name:value for fields that are not tied to an entity

  • entity_field_dicts, list of (entity, dictionary) where dictionary contains name:value for fields that are not tied to the given entity

http_publish(url, data=None)[source]

like url_publish, except this returns a http response, even in case of errors. You may give form parameters using the data argument.

classmethod init_config(config)[source]

configuration initialization hooks.

You may only want to override here the configuraton logic.

Otherwise, consider to use a different ApptestConfiguration defined in the configcls class attribute.

This method will be called by the database handler once the config has been properly bootstrapped.

list_actions_for(rset)[source]

returns the list of actions that can be applied on rset

list_boxes_for(rset)[source]

returns the list of boxes that can be applied on rset

list_startup_views()[source]

returns the list of startup views

list_views_for(rset)[source]

returns the list of views that can be applied on rset

new_access(login)[source]

provide a new RepoAccess object for a given user

The access is automatically closed at the end of the test.

classmethod pre_setup_database(cnx, config)[source]

add your pre database setup code by overriding this method

Do not forget to set the cls.test_db_id value to enable caching of the result.

remote_calling(fname, *args, **kwargs)[source]

remote json call simulation

requestcls

alias of cubicweb.devtools.fake.FakeRequest

property schema

return the application schema

setUp()[source]

Hook method for setting up the test fixture before exercising it.

classmethod setUpClass()[source]

Hook method for setting up class fixture before running tests in the class.

setup_database()[source]

add your database setup code by overriding this method

tearDown()[source]

Hook method for deconstructing the test fixture after testing it.

temporary_permissions(*perm_overrides, **perm_kwoverrides)[source]

Set custom schema permissions within context.

There are two ways to call this method, which may be used together :

  • using positional argument(s):

    rdef = self.schema['CWUser'].rdef('login')
    with self.temporary_permissions((rdef, {'read': ()})):
        ...
    
  • using named argument(s):

    with self.temporary_permissions(CWUser={'read': ()}):
        ...
    

Usually the former will be preferred to override permissions on a relation definition, while the latter is well suited for entity types.

The allowed keys in the permission dictionary depend on the schema type (entity type / relation definition). Resulting permissions will be similar to orig_permissions.update(partial_perms).

url_publish(url, data=None)[source]

takes url, uses application’s app_resolver to find the appropriate controller and result set, then publishes the result.

To simulate post of www-form-encoded data, give a data dictionary containing desired key/value associations.

This should pretty much correspond to what occurs in a real CW server except the apache-rewriter component is not called.

view(vid, rset=None, req=None, template='main-template', **kwargs)[source]

This method tests the view vid on rset using template

If no error occurred while rendering the view, the HTML is analyzed and parsed.

Returns

an instance of cubicweb.devtools.htmlparser.PageInfo encapsulation the generated HTML

7.8. What you need to know about request and session

../../../_images/request_session.png

First, remember to think that some code run on a client side, some other on the repository side. More precisely:

  • client side: web interface, raw repoapi connection (cubicweb-ctl shell for instance);

  • repository side: RQL query execution, that may trigger hooks and operation.

The client interacts with the repository through a repoapi connection.

Note

These distinctions are going to disappear in cubicweb 3.21 (if not before).

A repoapi connection is tied to a session in the repository. The connection and request objects are inaccessible from repository code / the session object is inaccessible from client code (theoretically at least).

The web interface provides a request class. That request object provides access to all cubicweb resources, eg:

  • the registry (which itself provides access to the schema and the configuration);

  • an underlying repoapi connection (when using req.execute, you actually call the repoapi);

  • other specific resources depending on the client type (url generation according to base url, form parameters, etc.).

A session provides an api similar to a request regarding RQL execution and access to global resources (registry and all), but also has the following responsibilities:

  • handle transaction data, that will live during the time of a single transaction. This includes the database connections that will be used to execute RQL queries.

  • handle persistent data that may be used across different (web) requests

  • security and hooks control (not possible through a request)

7.8.1. The _cw attribute

The _cw attribute available on every application object provides access to all cubicweb resources, i.e.:

  • For code running on the client side (eg web interface view), _cw is a request instance.

  • For code running on the repository side (hooks and operation), _cw is a Connection or Session instance.

Beware some views may be called with a session (e.g. notifications) or with a request.

7.8.2. Request, session and transaction

In the web interface, an HTTP request is handled by a single request, which will be thrown away once the response is sent.

The web publisher handles the transaction:

  • commit / rollback is done automatically

  • you should not commit / rollback explicitly, except if you really need it

Let’s detail the process:

  1. an incoming RQL query comes from a client to the web stack

  2. the web stack opens an authenticated database connection for the request, which is associated to a user session

  3. the query is executed (through the repository connection)

  4. this query may trigger hooks. Hooks and operations may execute some rql queries through cnx.execute.

  5. the repository gets the result of the query in 1. If it was a RQL read query, the database connection is released. If it was a write query, the connection is then tied to the session until the transaction is commited or rolled back.

  6. results are sent back to the client

This implies several things:

  • when using a request, or code executed in hooks, this database connection handling is totally transparent

  • however, take care when writing tests: you are usually faking / testing both the server and the client side, so you have to decide when to use RepoAccess.client_cnx or RepoAccess.repo_cnx. Ask yourself “where will the code I want to test be running, client or repository side?”. The response is usually: use a repo (since the “client connection” concept is going away in a couple of releases).