CONFspirator: Plot better configs!¶

http://img.shields.io/pypi/v/confspirator.svg

An offshoot of OpenStack’s oslo.config with a focus on nested configuration groups, and the ability to use yaml and toml instead of flat ini files.

CONFspirator doesn’t include any command-line integrations currently so you will need to add a command to your application to export a generated config using the built in functions.

It does have support for loading in config files, or a preloaded config dictionary against your config group tree.

The library’s focus is on in-code defaults and config field validation, while giving you a lot of power when dealing with nesting, dynamic config loading for plugins, and useful overlay logic.

It allows you to define sane defaults, document your config, validate the values when loading it in, and provides useful ways of working with that config during testing.

Getting started with CONFspirator¶

Installation¶

pip install confspirator

Usage¶

First lets put together a simple ConfigGroup, and register some config values:

# ./my_app/config/root.py
from confspirator import groups, fields

root_group = groups.ConfigGroup(
    "my_app", description="The root config group.")
root_group.register_child_config(
    fields.StrConfig(
        "top_level_config",
        help_text="Some top level config on the root group.",
        default="some_default",
    )
)

Then maybe let’s make a second group, but in another file to keep things clear:

# ./my_app/config/sub_section.py
from confspirator import groups, fields
from my_app.config import root

sub_group = groups.ConfigGroup(
    "sub_section", description="A sub group under the root group.")
sub_group.register_child_config(
    fields.BoolConfig(
        "bool_value",
        help_text="some boolean flag value",
        default=True,
    )
)

root.root_group.register_child_config(sub_group)

Now we want to load in our config against this group definition and check the values:

# ./my_app/config/__init__.py
import confspirator
from my_app.config import root

CONF = confspirator.load_file(
    root.root_group, "/etc/my_app/conf.yaml")

Assuming your config file looks like:

# String - Some top level config on the root group.
top_level_config: some_default
# A sub group under the root group.
sub_section:
  # Boolean - some boolean flag value
  bool_value: true

Then in your application code you can pull those values out and use them:

# ./my_app/do_thing.py
from my_app.config import CONF

print(CONF.top_level_config)
print(CONF.sub_section.bool_value)

Configuration fields in CONFspirator¶

CONFspirator supports multiple different fields for configuration, ranging from simple strings and ints, all the way to dictionaries and lists, with some special cases like ports, hostnames, and uri’s.

Options for all fields¶

These options are supported for all config fields in addition to the specific ones the field might have.

Options¶

name¶

The config field’s name. It is always the first parameter when defining a config field.

help_text¶

An explanation of how the option is used or what it is for.

default¶

The default value of the option.

required¶

If a value must be supplied for this option and cannot be empty or blank.

test_default¶

A default for when running in test mode.

sample_default¶

A default for sample config files.

unsafe_default (unused currently)¶

If the default value must be overridden.

Once implemented will warn or raise an error when a certain flag is given if the default value hasn’t been overridden.

secret (unused currently)¶

If the value should be obfuscated in log output.

Once implemented will be used to help provide information about what values to obfuscated in logs.

deprecated_location¶

Deprecated dot separated location. Acts like an alias to where the config used to be. e.g. ‘service.group1.old_name’

deprecated_for_removal¶

Indicates whether this opt is planned for removal in a future release.

deprecated_reason¶

Indicates why this opt is planned for removal in a future release. Silently ignored if deprecated_for_removal is False.

deprecated_since¶

indicates which release this opt was deprecated in. Accepts any string, though valid version strings are encouraged. Silently ignored if deprecated_for_removal is False

advanced (unused currently)¶

A bool True/False value if this option has advanced usage and is not normally used by the majority of users.

StrConfig¶

Simple string based config.

Options¶

name¶

The config’s name.

choices (optional)¶

Optional sequence of either valid values or tuples of valid values with descriptions.

quotes (optional)¶

If True and string is enclosed with single or double quotes, will strip those quotes.

regex (optional)¶

Optional regular expression (string or compiled regex) that the value must match on an unanchored search.

ignore_case (optional)¶

If True case differences (uppercase vs. lowercase) between ‘choices’ or ‘regex’ will be ignored.

max_length (optional)¶

If positive integer, the value must be less than or equal to this parameter.

Example usage¶

config_group.register_child_config(
    fields.StrConfig(
        "my_string_config",
        help_text="Some useful help text.",
        required=True,
        default="stuff",
    )
)

BoolConfig¶

Simple boolean based config.

Options¶

name¶

The config’s name.

Example usage¶

config_group.register_child_config(
    fields.BoolConfig(
        "my_boolean_config",
        help_text="Some useful help text.",
        required=True,
        default=False,
    )
)

IntConfig¶

Simple int based config.

Options¶

name¶

The config’s name.

min (optional)¶

Minimum value the integer can take.

max (optional)¶

Maximum value the integer can take.

Example usage¶

config_group.register_child_config(
    fields.IntConfig(
        "my_int_config",
        help_text="Some useful help text.",
        required=True,
        default=6,
        min=1,
        max=10,
    )
)

FloatConfig¶

Simple float based config.

Options¶

name¶

The config’s name.

min (optional)¶

Minimum value the float can take.

max (optional)¶

Maximum value the float can take.

Example usage¶

config_group.register_child_config(
    fields.FloatConfig(
        "my_float_config",
        help_text="Some useful help text.",
        required=True,
        default=6.4,
        min=1.2,
        max=10.9,
    )
)

ListConfig¶

A list config, with a configurable type for items.

Options¶

name¶

The config’s name.

item_type (optional)¶

Type of items in the list (see confspirator.types). If not set will default to a list of strings.

Example usage¶

config_group.register_child_config(
    fields.ListConfig(
        "my_list_config",
        help_text="Some useful help text.",
        required=True,
        default=["stuff", "things"],
    )
)

DictConfig¶

A dict config, with a configurable type for values.

Options¶

name¶

The config’s name.

value_type (optional)¶

Type of values in the dict (see confspirator.types). If not set will default to strings.

check_value_type (optional)¶

If value is already dict, should we check value type.

is_json (optional)¶

If True and value is string, will parse as json.

Example usage¶

config_group.register_child_config(
    fields.DictConfig(
        "my_dict_config",
        help_text="Some useful help text.",
        required=True,
        default={"stuff": "things"},
    )
)

IPConfig¶

IP address config.

Options¶

name¶

The config’s name.

version (optional)¶

One of either 4, 6, or None to specify either version.

Example usage¶

config_group.register_child_config(
    fields.IPConfig(
        "my_ip_config",
        help_text="Some useful help text.",
        required=True,
        default=0.0.0.0,
        version=4,
    )
)

PortConfig¶

Config for a TCP/IP port number. Ports can range from 0 to 65535.

Options¶

name¶

The config’s name.

min (optional)¶

Minimum value the port can take.

max (optional)¶

Maximum value the port can take.

choices (optional)¶

Sequence of valid values.

Example usage¶

config_group.register_child_config(
    fields.PortConfig(
        "my_port_config",
        help_text="Some useful help text.",
        required=True,
        default=222,
        min=2000,
        max=9999,
    )
)

HostNameConfig¶

Config for a hostname. Only accepts valid hostnames.

Options¶

name¶

The config’s name.

Example usage¶

config_group.register_child_config(
    fields.HostNameConfig(
        "my_hostname_config",
        help_text="Some useful help text.",
        required=True,
        default="prod.cluster.thing.net",
    )
)

HostAddressConfig¶

Option for either an IP or a hostname.

Options¶

name¶

The config’s name.

version (optional)¶

One of either 4, 6, or None to specify either version.

Example usage¶

config_group.register_child_config(
    fields.HostAddressConfig(
        "my_hostaddress_config",
        help_text="Some useful help text.",
        required=True,
        default="prod.cluster.thing.net",
    )
)

URIConfig¶

Option for either a URI.

Options¶

name¶

The config’s name.

max_length (optional)¶

If positive integer, the value must be less than or equal to this parameter.

schemes (optional)¶

List of valid URI schemes, e.g. ‘https’, ‘ftp’, ‘git’.

Example usage¶

config_group.register_child_config(
    fields.URIConfig(
        "my_url_config",
        help_text="Some useful help text.",
        required=True,
        default="https://example.com",
        schemes["https", "http"]
    )
)

Example config generation¶

CONFspirator supports generated config files from the built config tree. These will populate all the fields, and if a default value is supplied will put that in place.

To use this functionality you must supply your root config group, and call the function with a file location. CONFspirator does not supply any CLI support for this, so you will want to build a command into your application or project to import your root config group, and call the generation function.

Assuming the config group as shown in the getting started page:

# ./my_app/config/root.py
from confspirator import groups, fields

root_group = groups.ConfigGroup(
    "my_app", description="The root config group.")
root_group.register_child_config(
    fields.StrConfig(
        "top_level_config",
        help_text="Some top level config on the root group.",
        default="some_default",
    )
)

# ./my_app/config/sub_section.py
from confspirator import groups, fields
from my_app.config import root

sub_group = groups.ConfigGroup(
    "sub_section", description="A sub group under the root group.")
sub_group.register_child_config(
    fields.BoolConfig(
        "bool_value",
        help_text="some boolean flag value",
        default=True,
    )
)

root.root_group.register_child_config(sub_group)

You would call the generation logic as follows:

# ./my_app/commands.py

import confspirator
from my_app.config import root


def create_config():
    confspirator.create_example_config(
        root.root_group, "conf.yaml")

This would produce a yaml config example that looks like the following:

# String - Some top level config on the root group.
top_level_config: some_default
# A sub group under the root group.
sub_section:
  # Boolean - some boolean flag value
  bool_value: true

Alternatively if you wanted toml instead, you can simply change the file extension and the exporter will pick that up. Or if you want to use an extension other than yaml or toml you can explicitly set output_format to either yaml or toml, and the file extension will be ignored:

confspirator.create_example_config(
    root.root_group, "my_app.conf", output_format="toml")

In any case, if by extension or explicit output format toml is set, your generated example config will look as follows:

[my_app]
# String - Some top level config on the root group.
top_level_config = "some_default"

# A sub group under the root group.
[my_app.sub_section]
  # Boolean - some boolean flag value
  bool_value = true

For complicated nested configs yaml tends to be easier to deal with, but for people with a preference for ini style configs toml does provide a good option that still allows nesting.

CONFspirator in your unit tests¶

You often need ways to override config when running tests, so CONFspirator provides some powerful ways to do that even for the most complicated of nested configs.

Setting test defaults¶

The first smart thing to do is set the test_default value on a given config field for something you want to be a global test default value. This is useful for values you rarely need to change, and this means you can put your test defaults in the same places you define you config and put your actual defaults.

An example of this might be:

config_group.register_child_config(
    fields.StrConfig(
        "my_string_config",
        help_text="Some useful help text.",
        required=True,
        default="stuff",
        test_default="test_specific_stuff",
    )
)

To ensure CONFspirator uses the test_default you will need to put it into test_mode when running the load config functions:

CONF = confspirator.load_file(
    config_group, "/etc/my_app/conf.yaml", test_mode=True)

That does mean you will need some way to decide when loading config if your application is running in test_mode. This will vary depending on your framework, or unit testing tools, but as an example, in Django you could do it as follows:

test_mode = False
if "test" in sys.argv:
    test_mode = True

CONF = confspirator.load_file(
    config_group, "/etc/my_app/conf.yaml", test_mode=test_mode)

Overriding config for test cases¶

Often in unit or functional testing you need ways to override config for the duration of a test, a whole set of tests, or even in different phases of a test.

As such CONFspirator provides an all powerful modify_conf function that alows you to selectively alter your config entity for the needed scope.

Note

modify_conf assumes your test case classes inherit from unittest.TestCase. If they do not, then this will not work.

A simple example:

import confspirator

from my_app.config import CONF


@confspirator.modify_conf(
    CONF,
    {
        "my_app.top_level_config": [
            {"operation": "override", "value": "a new value"}
        ],
    }
)
class BasicTests(TestCase):

    def test_top_level_config(self):
        self.assertEqual(CONF.top_level_config, "a new value")

It can also be used to decorate a single test function:

import confspirator

from my_app.config import CONF


class BasicTests(TestCase):

    @confspirator.modify_conf(
        CONF,
        {
            "my_app.top_level_config": [
                {"operation": "override", "value": "a new value"}
            ],
        }
    )
    def test_top_level_config(self):
        self.assertEqual(CONF.top_level_config, "a new value")

Or even a section when using the with keyword:

import confspirator

from my_app.config import CONF


class BasicTests(TestCase):
    def test_top_level_config(self):
        with confspirator.modify_conf(
            CONF,
            {
                "my_app.top_level_config": [
                    {"operation": "override", "value": "a new value"}
                ],
            },
        ):
            self.assertEqual(CONF.top_level_config, "a new value")

parameters for modify_conf¶

modify_conf takes two argument which can be used positionally, or as keywords.

conf¶

This should be the loaded config entity and will be either an instance of GroupNamespace or in advanced cases LazyLoadedGroupNamespace.

operations¶

This is a dictionary of config values as dot separated paths to a list of operations.

It is possible to alter multiple config values at the same time, and run multiple operations on each. Operations will run in the order supplied and can be chained together (e.g. add a value to the start and end of a list).

Here is what a more complex example may look like:

operations={
    "my_app.api_settings.item_list_option": [
        {"operation": "remove", "value": "option1"},
        {"operation": "append", "value": "option15"},
    ],
    "my_app.api_settings.boolean_flag_option": [
        {"operation": "override", "value": False},
    ],
}

Available operations per config type¶

value:
- override
list:
- override
- preprend
- append
- remove
dict:
- override
- update
- delete
- overlay
GroupNamespace:
- override
- overlay

Overlay is essential a dict merge, where any keys present in the overlaying dictionay will be inserted or will override the ones in the target.

Advanced Usage¶

TODO: Examples on lazyloading for plugins and overlaying groups.