plainbox.impl.secure.providers.v1 – Implementation of V1 provider

class plainbox.impl.secure.providers.v1.AbsolutePathValidator[source]

Bases: plainbox.impl.secure.config.IValidator

A validator that checks that the value is an absolute path

class plainbox.impl.secure.providers.v1.ExistingDirectoryValidator[source]

Bases: plainbox.impl.secure.config.IValidator

A validator that checks that the value points to an existing directory

class plainbox.impl.secure.providers.v1.IQNValidator[source]

Bases: plainbox.impl.secure.config.PatternValidator

A validator for provider name.

Provider names use a RFC3720 IQN-like identifiers composed of the follwing parts:

  • year
  • (dot separating the next section)
  • domain name
  • (colon separating the next section)
  • identifier

Each of the fields has an informal definition below:

year:
four digit number
domain name:
identifiers separated by dots, at least one dot has to be present
identifier:
[a-z][a-z0-9-]*
class plainbox.impl.secure.providers.v1.Provider1(name, namespace, version, description, secure, gettext_domain, units_dir, jobs_dir, whitelists_dir, data_dir, bin_dir, locale_dir, base_dir, *, validate=False, validation_kwargs=None, check=True, context=None)[source]

Bases: plainbox.abc.IProvider1

A v1 provider implementation.

A provider is a container of jobs and whitelists. It provides additional meta-data and knows about location of essential directories to both load structured data and provide runtime information for job execution.

Providers are normally loaded with Provider1PlugIn, due to the number of fields involved in basic initialization.

CHECKBOX_SHARE

required value of CHECKBOX_SHARE environment variable.

Note

This variable is only required by one script. It would be nice to remove this later on.

base_dir

path of the directory with (perhaps) all of jobs_dir, whitelists_dir, data_dir, bin_dir, locale_dir. This may be None

bin_dir

absolute path of the bin directory

Note

The programs in that directory may not work without setting PYTHONPATH and CHECKBOX_SHARE.

build_bin_dir

absolute path of the build/bin directory

This value may be None. It depends on location/base_dir being set.

build_dir

absolute path of the build directory

This value may be None. It depends on location/base_dir being set.

build_mo_dir

absolute path of the build/mo directory

This value may be None. It depends on location/base_dir being set.

classify

Exposed ProviderContentClassifier.classify()

content_collection

Exposed ProviderContentEnumerator.content_collection()

data_dir

absolute path of the data directory

description

description of this provider

executable_list

List of all the executables

extra_PYTHONPATH

additional entry for PYTHONPATH, if needed.

This entry is required for CheckBox scripts to import the correct CheckBox python libraries.

Note

The result may be None

fake

Bridge to .content_collection.fake_plugins that’s shorter to type.

classmethod from_definition(definition, secure, *, validate=False, validation_kwargs=None, check=True, context=None)[source]

Initialize a provider from Provider1Definition object

Parameters:
  • definition – A Provider1Definition object to use as reference
  • secure – Value of the secure flag. This cannot be expressed by a definition object.
  • validate – Enable job validation. Incorrect job definitions will not be loaded and will abort the process of loading of the remainder of the jobs. This is ON by default to prevent broken job definitions from being used. This is a keyword-only argument.
  • validation_kwargs – Keyword arguments to pass to the JobDefinition.validate(). Note, this is a single argument. This is a keyword-only argument.

This method simplifies initialization of a Provider1 object where the caller already has a Provider1Definition object. Depending on the value of definition.location all of the directories are either None or initialized to a good (typical) value relative to location

The only value that you may want to adjust, for working with source providers, is locale_dir, by default it would be location/locale but manage.py i18n creates location/build/mo

get_translated_data(msgid)[source]

Get a localized piece of data

Parameters:msgid – data to translate
Returns:translated data obtained from the provider if msgid is not False (empty string and None both are) and this provider has a gettext_domain defined for it, msgid itself otherwise.
gettext_domain

the name of the gettext domain associated with this provider

This value may be empty, in such case provider data cannot be localized for the user environment.

has_dedicated_namespace

Flag set if namespace was defined by a dedicated field.

id_map

A mapping from unit identifier to list of units with that identifier.

Note

Typically the list will be one element long but invalid providers may break that guarantee. Code defensively if you can.

job_list

A sorted list of loaded job definition units.

jobs_dir

absolute path of the jobs directory

locale_dir

absolute path of the directory with locale data

The value is applicable as argument bindtextdomain()

name

name of this provider

namespace

namespace component of the provider name

This property defines the namespace in which all provider jobs are defined in. Jobs within one namespace do not need to be fully qualified by prefixing their partial identifier with provider namespace (so all stays ‘as-is’). Jobs that need to interact with other provider namespaces need to use the fully qualified job identifier instead.

The identifier is defined as the part of the provider name, up to the colon. This effectively gives organizations flat namespace within one year-domain pair and allows to create private namespaces by using sub-domains.

path_map

A mapping from filename path to a list of units stored in that file.

Note

For .pxu files this will enumerate all units stored there. For other things it will typically be just the FileUnit.

po_dir

absolute path of the po/ directory

This value may be None. It depends on location/base_dir set.

problem_list

list of problems encountered by the loading process

secure

flag indicating that this provider was loaded from the secure portion of PROVIDERPATH and thus can be used with the plainbox-trusted-launcher-1.

src_dir

absolute path of the src/ directory

This value may be None. It depends on location/base_dir set.

tr_description()[source]

Get the translated version of description()

unit_list

List of loaded units.

This list may contain units of various types. You should not assume all of them are JobDefinition instances. You may use filtering to obtain units of a given type.

>>> [unit for unit in provider.unit_list
...  if unit.Meta.name == 'job']
[...]
units_dir

absolute path of the units directory

version

version of this provider

whitelist_list

List of loaded whitelists.

Warning

WhiteList is currently deprecated. You should never need to access them in any new code. They are entirely replaced by TestPlan. This property is provided for completeness and it will be removed once whitelists classes are no longer used.

whitelists_dir

absolute path of the whitelist directory

class plainbox.impl.secure.providers.v1.Provider1Definition[source]

Bases: plainbox.impl.secure.config.Config

A Config-like class for parsing plainbox provider definition files

Note

The location attribute is special, if set, it defines the base directory of all the other directory attributes. If location is unset, then all the directory attributes default to None (that is, there is no directory of that type). This is actually a convention that is implemented in Provider1PlugIn. Here, all the attributes can be Unset and their validators only check values other than Unset.

class Meta

Bases: plainbox.impl.secure.config.ConfigMetaData

filename_list = []
parametric_section_list = []
section_list = []
variable_list = [<Variable name:'relocatable'>, <Variable name:'location'>, <Variable name:'name'>, <Variable name:'namespace'>, <Variable name:'version'>, <Variable name:'description'>, <Variable name:'gettext_domain'>, <Variable name:'units_dir'>, <Variable name:'jobs_dir'>, <Variable name:'whitelists_dir'>, <Variable name:'data_dir'>, <Variable name:'bin_dir'>, <Variable name:'locale_dir'>]
Provider1Definition.bin_dir

Pathname of the directory with provider executables

Provider1Definition.data_dir

Pathname of the directory with provider data

Provider1Definition.description

Description of the provider

Provider1Definition.effective_bin_dir

effective value of bin_dir

The effective value is bin_dir() itself, unless it is Unset. If it is Unset the effective value is the implicit_bin_dir(), if that value would be valid. The effective value may be None.

Provider1Definition.effective_data_dir

effective value of data_dir

The effective value is data_dir() itself, unless it is Unset. If it is Unset the effective value is the implicit_data_dir(), if that value would be valid. The effective value may be None.

Provider1Definition.effective_gettext_domain

effective value of gettext_domian

The effective value is gettex_domain() itself, unless it is Unset. If it is Unset the effective value None.

Provider1Definition.effective_jobs_dir

effective value of jobs_dir

The effective value is jobs_dir() itself, unless it is Unset. If it is Unset the effective value is the implicit_jobs_dir(), if that value would be valid. The effective value may be None.

Provider1Definition.effective_locale_dir

effective value of locale_dir

The effective value is locale_dir() itself, unless it is Unset. If it is Unset the effective value is the implicit_locale_dir(), if that value would be valid. The effective value may be None.

Provider1Definition.effective_units_dir

effective value of units_dir

The effective value is units_dir() itself, unless it is Unset. If it is Unset the effective value is the implicit_units_dir(), if that value would be valid. The effective value may be None.

Provider1Definition.effective_whitelists_dir

effective value of whitelists_dir

The effective value is whitelists_dir() itself, unless it is Unset. If it is Unset the effective value is the implicit_whitelists_dir(), if that value would be valid. The effective value may be None.

Provider1Definition.filename_list

list of pathnames to files that were loaded by the last call to read()

Provider1Definition.get()

Get an instance of this Config class with all the configuration loaded from default locations. The locations are determined by Meta.filename_list attribute.

Returns:fresh Config instance
Provider1Definition.get_parser_obj()

Get a ConfigParser-like object with the same data.

Returns:A PlainBoxConfigParser object with all of the data copied from this Config object.

Since PlainBoxConfigParser is a subclass of configparser.ConfigParser it has a number of useful utility methods. By using this function one can obtain a ConfigParser-like object and work with it directly.

Provider1Definition.gettext_domain

Name of the gettext domain for translations

Provider1Definition.implicit_bin_dir

implicit value of bin_dir (if Unset)

The implicit value is only defined if location is not Unset. It is the ‘bin’ subdirectory of the directory that location points to.

Provider1Definition.implicit_build_locale_dir

implicit value of locale_dir (if Unset) as laid out in the source tree

This value is only applicable to source layouts, where the built translation catalogs are in the build/mo directory.

Provider1Definition.implicit_data_dir

implicit value of data_dir (if Unset)

The implicit value is only defined if location is not Unset. It is the ‘data’ subdirectory of the directory that location points to.

Provider1Definition.implicit_jobs_dir

implicit value of jobs_dir (if Unset)

The implicit value is only defined if location is not Unset. It is the ‘jobs’ subdirectory of the directory that location points to.

Provider1Definition.implicit_locale_dir

implicit value of locale_dir (if Unset)

The implicit value is only defined if location is not Unset. It is the ‘locale’ subdirectory of the directory that location points to.

Provider1Definition.implicit_units_dir

implicit value of units_dir (if Unset)

The implicit value is only defined if location is not Unset. It is the ‘units’ subdirectory of the directory that location points to.

Provider1Definition.implicit_whitelists_dir

implicit value of whitelists_dir (if Unset)

The implicit value is only defined if location is not Unset. It is the ‘whitelists’ subdirectory of the directory that location points to.

Provider1Definition.jobs_dir

Pathname of the directory with job definitions

Provider1Definition.locale_dir

Pathname of the directory with locale data

Provider1Definition.location

Base directory with provider data

Provider1Definition.name

Name of the provider

Provider1Definition.name_without_colon
Provider1Definition.namespace

Namespace of the provider

Provider1Definition.problem_list

list of ValidationError that were detected by the last call to read()

Provider1Definition.read(filename_list)

Load and merge settings from many files.

This method tries to open each file from the list of filenames, parse it as an INI file using PlainBoxConfigParser (a simple ConfigParser subclass that respects the case of key names). The list of files actually accessed is saved as available as Config.filename_list.

If any problem is detected during parsing (e.g. syntax errors) those are captured and added to the Config.problem_list.

After all files are loaded each Variable and Section defined in the Config class is assigned with the data from the merged configuration data.

Any variables that cannot be assigned and raise ValidationError are ignored but the list of problems is saved.

All unused configuration (extra variables that are not defined as either Variable or Section class) is silently ignored.

Note

This method resets _problem_list and _filename_list.

Provider1Definition.read_string(string)

Load settings from a string.

Parameters:string – The full text of INI-like configuration to parse and apply

This method parses the string as an INI file using PlainBoxConfigParser (a simple ConfigParser subclass that respects the case of key names).

If any problem is detected during parsing (e.g. syntax errors) those are captured and added to the Config.problem_list.

After parsing the string each Variable and Section defined in the Config class is assigned with the data from the configuration data.

Any variables that cannot be assigned and raise ValidationError are ignored but the list of problems is saved.

All unused configuration (extra variables that are not defined as either Variable or Section class) is silently ignored.

Note

This method resets _problem_list and _filename_list.

Provider1Definition.relocatable

Flag indicating if the provider is relocatable

Provider1Definition.units_dir

Pathname of the directory with unit definitions

Provider1Definition.validate_whole()[source]

Validate the provider definition object.

Raises:ValidationError – If the namespace is not defined and name is using a simplified format that doesn’t contain an embedded namespace part.
Provider1Definition.version

Version of the provider

Provider1Definition.whitelists_dir

Pathname of the directory with whitelists definitions

Provider1Definition.write(stream)

Write configuration data to a stream.

Parameters:stream – a file-like object that can be written to.

This method recreates the content of all the configuration variables in a manner that can be subsequently read back.

class plainbox.impl.secure.providers.v1.Provider1PlugIn(filename, definition_text, load_time, *, validate=None, validation_kwargs=None, check=None, context=None)[source]

Bases: plainbox.impl.secure.plugins.PlugIn

A specialized IPlugIn that loads Provider1 instances from their definition files

plugin_load_time

time, in fractional seconds, that was needed to load the plugin

plugin_name

plugin name, arbitrary string

plugin_object

plugin object, arbitrary object

plugin_wrap_time

time, in fractional seconds, that was needed to wrap the plugin

class plainbox.impl.secure.providers.v1.ProviderContentClassifier(provider: plainbox.impl.secure.providers.v1.Provider1)[source]

Bases: object

Support class for classifying content inside a provider.

The primary role of this class is to come up with the role of each file inside the provider. That includes all files reachable from any of the directories that constitute a provider definition. In addition, each file is associated with a base directory. This directory can be used to re-construct the same provider at a different location or in a different layout.

The secondary role is to provide a hint on what PlugIn to use to load such content (as units). In practice the majority of files are loaded with the UnitPlugIn class. Legacy .whitelist files are loaded with the WhiteListPlugIn class instead. All other files are handled by the ProviderContentPlugIn.

Note

This class is automatically instantiated by Provider1. The classify() method is exposed as Provider1.classify().

DOC_SET = frozenset({'README.md', 'README.txt', 'README.rst', 'README'})
EXECUTABLES

A set of executables that are expected to be built from source.

LEGAL_SET = frozenset({'COPYING.LESSER', 'COPYING', 'LICENSE'})
classify()[source]

Classify a file belonging to the provider

Parameters:filename – Full pathname of the file to classify
Returns:A tuple of information about the file. The first element is the FileRole symbol that describes the role of the file. The second element is the base path of the file. It can be subtracted from the actual filename to obtain a relative directory where the file needs to be located in case of provider re-location. The last, third element is the plug-in class that can be used to load units from that file.
Raises:ValueError – If the file cannot be classified. This can only happen if the file is not in any way related to the provider. All (including random junk) files can be classified correctly, as long as they are inside one of the well-known directories.
classify_fn_list

List of functions that aid in the classification process.

class plainbox.impl.secure.providers.v1.ProviderContentEnumerator(provider: plainbox.impl.secure.providers.v1.Provider1)[source]

Bases: object

Support class for enumerating provider content.

The only role of this class is to expose a plug in collection that can enumerate all of the files reachable from a provider. This collection is consumed by other parts of provider loading machinery.

Since it is a stock plug in collection it can be easily “mocked” to provide alternate content without involving modifications of the real file system.

Note

This class is automatically instantiated by Provider1. The content_collection() property is exposed as Provider1.content_collection().

content_collection

An plugin collection that enumerates all of the files in the provider.

This collections exposes all of the files in a provider. It can also be mocked for easier testing. It is the only part of the provider codebase that tries to discover data in a file system.

Note

By default the collection is not loaded. Make sure to call .load() to see the actual data. This is, again, a way to simplify testing and to de-couple it from file-system activity.

class plainbox.impl.secure.providers.v1.ProviderContentLoader(provider)[source]

Bases: object

Support class for enumerating provider content.

The only role of this class is to expose a plug in collection that can enumerate all of the files reachable from a provider. This collection is consumed by other parts of provider loading machinery.

Since it is a stock plug in collection it can be easily “mocked” to provide alternate content without involving modifications of the real file system.

Note

This class is automatically instantiated by Provider1. All four attributes of this class are directly exposed as properties on the provider object.

Attr provider:The provider back-reference
Attr is_loaded:Flag indicating if the content loader has loaded all of the content
Attr unit_list:A list of loaded whitelist objects
Attr problem_list:
 A list of problems experienced while loading any of the content
Attr path_map:A dictionary mapping from the path of each file to the list of units stored there.
Attr id_map:A dictionary mapping from the identifier of each unit to the list of units that have that identifier.
load(plugin_kwargs)[source]
class plainbox.impl.secure.providers.v1.ProviderContentPlugIn(filename, text, load_time, provider, *, validate=False, validation_kwargs=None, check=True, context=None)[source]

Bases: plainbox.impl.secure.plugins.PlugIn

PlugIn class for loading provider content.

Provider content comes in two shapes and sizes:
  • units (of any kind)
  • whitelists

The actual logic on how to load everything is encapsulated in wrap() though its return value is not so useful.

Attr unit_list:The list of loaded units
Attr whitelist_list:
 The list of loaded whitelists
discover_units()[source]

Discover all units that were loaded by this plug-in

Parameters:wrap_result – whatever was returned on the call to wrap().
Returns:an iterable of units.

Note

this method is always called after wrap().

discover_whitelists()[source]

Discover all whitelists that were loaded by this plug-in

Parameters:wrap_result – whatever was returned on the call to wrap().
Returns:an iterable of whitelists.

Note

this method is always called after wrap().

inspect()[source]

Interpret and wrap the content of the filename as whatever is appropriate. The return value of this class becomes the plugin_object()

Note

This method must not access neither unit_list nor whitelist_list. If needed, it can collect its own state in private instance attributes.

make_file_unit(filename, provider, role=None, base=None)[source]
plugin_load_time

time, in fractional seconds, that was needed to load the plugin

plugin_name

plugin name, arbitrary string

plugin_object

plugin object, arbitrary object

plugin_wrap_time

time, in fractional seconds, that was needed to wrap the plugin

class plainbox.impl.secure.providers.v1.ProviderNameValidator[source]

Bases: plainbox.impl.secure.config.PatternValidator

Validator for the provider name.

Two forms are allowed:

  • short form (requires a separate namespace definition)
  • verbose form (based on RFC3720 IQN-like strings)

The short form is supposed to look like Debian package name.

class plainbox.impl.secure.providers.v1.SecureProvider1PlugInCollection(**kwargs)[source]

Bases: plainbox.impl.secure.plugins.FsPlugInCollection

A collection of v1 provider plugins.

This FsPlugInCollection subclass carries proper, built-in defaults, that make loading providers easier.

This particular class loads providers from the system-wide managed locations. This defines the security boundary, as if someone can compromise those locations then they already own the corresponding system. In consequence this plug in collection does not respect PROVIDERPATH, it cannot be customized to load provider definitions from any other location. This feature is supported by the plainbox.impl.providers.v1.InsecureProvider1PlugInCollection

discovery_time

Time, in fractional seconds, that was required to discover all objects.

This time is separate from the load and wrap time of all each individual plug-in. Typically this is either a fixed cost or a predictable cost related to traversing the file system.

fake_plugins(plugins, problem_list=None)

Context manager for using fake list of plugins

Parameters:
  • plugins – list of PlugIn-alike objects
  • problem_list – list of problems (exceptions)

The provided list of plugins overrides any previously loaded plugins and prevent loading any other, real, plugins. After the context manager exits the previous state is restored.

get_all_items()

Get a list of all the pairs of plugin name and plugin

Returns:a list of tuples (plugin.plugin_name, plugin)
get_all_names()

Get a list of all the plug-in names

Returns:a list of plugin names
get_all_plugin_objects()

Get an list of plug-in objects

get_all_plugins()

Get a list of all the plug-ins

Returns:a list of plugin objects
get_by_name(name)

Get the specified plug-in (by name)

Parameters:name – name of the plugin to locate
Returns:PlugIn like object associated with the name
Raises:KeyError – if the specified name cannot be found
get_total_time() → float

Get the sum of load and wrap time of each plugin object

Returns:The total number of fractional seconds of wall-clock time spent by loading this collection. This value doesn’t include some small overhead of this class but is representative of the load times of pluggable code.
load()

Load all plug-ins.

This method loads all plug-ins from the search directories (as defined by the path attribute). Missing directories are silently ignored.

problem_list

List of problems encountered while loading plugins

wrap_and_add_plugin(plugin_name, plugin_obj, plugin_load_time)

Internal method of PlugInCollectionBase.

Parameters:
  • plugin_name – plugin name, some arbitrary string
  • plugin_obj – plugin object, some arbitrary object.
  • plugin_load_time – number of seconds it took to load this plugin

This method prepares a wrapper (PlugIn subclass instance) for the specified plugin name/object by attempting to instantiate the wrapper class. If a PlugInError exception is raised then it is added to the problem_list and the corresponding plugin is not added to the collection of plugins.

class plainbox.impl.secure.providers.v1.UnitPlugIn(filename, text, load_time, provider, *, validate=False, validation_kwargs=None, check=True, context=None)[source]

Bases: plainbox.impl.secure.providers.v1.ProviderContentPlugIn

A specialized plainbox.impl.secure.plugins.IPlugIn that loads a list of plainbox.impl.unit.Unit instances from a file.

discover_units()[source]
discover_whitelists()[source]
inspect()[source]

Load all units from their PXU representation.

Parameters:
  • filename – Name of the file with unit definitions
  • text – Full text of the file with unit definitions (lazy)
  • provider – A provider object to which those units belong to
  • validate – Enable unit validation. Incorrect unit definitions will not be loaded and will abort the process of loading of the remainder of the jobs. This is ON by default to prevent broken units from being used. This is a keyword-only argument.
  • validation_kwargs – Keyword arguments to pass to the Unit.validate(). Note, this is a single argument. This is a keyword-only argument.
  • check – Enable unit checking. Incorrect unit definitions will not be loaded and will abort the process of loading of the remainder of the jobs. This is OFF by default to prevent broken units from being used. This is a keyword-only argument.
  • context – If checking, use this validation context.
make_file_unit(filename, provider, role=None, base=None)
plugin_load_time

time, in fractional seconds, that was needed to load the plugin

plugin_name

plugin name, arbitrary string

plugin_object
plugin_wrap_time

time, in fractional seconds, that was needed to wrap the plugin

class plainbox.impl.secure.providers.v1.VersionValidator[source]

Bases: plainbox.impl.secure.config.PatternValidator

A validator for provider version.

Provider version must follow PEP440. See https://www.python.org/dev/peps/pep-0440/

class plainbox.impl.secure.providers.v1.WhiteListPlugIn(filename, text, load_time, provider=None, *, validate=False, validation_kwargs=None, check=True, context=None)[source]

Bases: plainbox.impl.secure.providers.v1.ProviderContentPlugIn

A specialized plainbox.impl.secure.plugins.IPlugIn that loads plainbox.impl.secure.qualifiers.WhiteList instances from a file.

discover_units()[source]
discover_whitelists()[source]
inspect()[source]
make_file_unit(filename, provider, role=None, base=None)
make_test_plan_unit(filename, text, provider)[source]
plugin_load_time

time, in fractional seconds, that was needed to load the plugin

plugin_name

plugin name, the name of the WhiteList

plugin_object

plugin object, arbitrary object

plugin_wrap_time

time, in fractional seconds, that was needed to wrap the plugin

plainbox.impl.secure.providers.v1.get_secure_PROVIDERPATH_list()[source]

Computes the secure value of PROVIDERPATH

This value is used by plainbox-trusted-launcher-1 executable to discover all secure providers.

Returns:A list of two strings: * /usr/local/share/plainbox-providers-1 * /usr/share/plainbox-providers-1
comments powered by Disqus