Checkbox Whitelist Files

When creating a test suite for a particular purpose, it will be necessary to specify which tests to run and which order they should run in. For this purpose Checkbox provides the concept of Whitelists.

Whitelist Format

A whitelist is a text file containing a line-seperated sequence of patterns, each representing one or more ‘jobs’. These patterns are in the Python regular expression syntax. Comments may be included in the file by starting the line with ‘#’.

Minimal Whitelist File

In order to be useful a whitelist file needs to include a particular subset of jobs which provide Checkbox with all of the information it needs to run tests properly. These include jobs which attach hardware information and resource jobs which provide other jobs with information of the environment they a re running in (available hardware, available packages etc). To make this easy to do a single job exists whose purpose is to execute all of these other jobs:

miscellanea/submission-resources

This should be included as the first job in any whitelist.

Job Categories

In order to allow Checkbox to display jobs by category in the UI it is necessary to include a particular local job which itself generates jobs which belong to that category. This job will normally look like __<category>__ where <category> is the name of the job file which contains the job. This is indicated again by the prefix of the job (before the / in the job name). As a quick example, the job graphics/glxgears is contained in graphics.txt. Therefore we should include the __graphics__ job so that the graphics/glxgears job shows correctly under the category. The __graphics__ job itself looks like:

name: __graphics__
plugin: local
_description: Graphics tests
command:
  shopt -s extglob
  cat $CHECKBOX_SHARE/jobs/graphics.txt?(.in)

Checkbox will interpret this job as a request to display any job in graphics.txt (or its untranslated version graphics.txt.in) under the heading shown in the description of this job (in this case ‘Graphics tests’).

Tutorial

To compound what we discussed before, below is a brief tutorial which walks through assembling a basic whitelist file.

1. First we need to create a file, let’s name it tutorial.whitelist. Whitelists don’t have to end with the .whitelist suffix but this is the convention used to help identify them. 2. We start by adding the one job that is required for all whitelists, as explained above in the section ‘Minimal Whitelist File’, so our whitelist file looks like:

miscellanea/submission-resources

3. Next we should choose some jobs that we want to run. This all depends on your specific use-case of course, but I’ve selected a few jobs that will help clearly illustrate more of the concepts involved in whitelists. These jobs will give us a whitelist file that looks like:

   miscellanea/submission-resources
   cpu/clocktest
   ethernet/multi_nic
   ethernet/multi_nic_eth0
   graphics/glxgears

If we run this whitelist now then all of these jobs will be executed and a
valid test submission will be created, but we can still improve it in a couple
of ways.

4. The first way is by adding the necessary jobs to allow the Checkbox UI to group the jobs into specific categories. To do this we need to add a job with a name like __<category>__ for each category. We have three categories in our whitelist file - cpu, ethernet and graphics. The category of the job is the prefix of the job name prior to the /. So now our whitelist file looks like:

  miscellanea/submission-resources
  __cpu__
  __ethernet__
  __graphics__
  cpu/clocktest
  ethernet/multi_nic
  ethernet/multi_nic_eth0
  graphics/glxgears

Now the Checkbox UI will group the jobs into these categories.

5. Although it’s not immediately apparent there is another problem with this whitelist. The ethernet/multi_nic tests are only able to include one job for the ethernet port ‘eth0’. It would be better if we included all of the jobs generated by ‘ethernet/multi_nic’, no matter how many ethernet ports are present on the system under test. The best way to do this is to write the pattern so that it matches all of the possible job names. We can take advantage of the Python regular expression syntax and use the \d special character to match any decimal number. After doing this the whitelist file will look like this:

miscellanea/submission-resources
__cpu__
__ethernet__
__graphics__
cpu/clocktest
ethernet/multi_nic
ethernet/multi_nic_eth\d
graphics/glxgears
comments powered by Disqus