ml_collections.config_dict package

Classes for defining configurations of experiments and models.

ConfigDict class

class ml_collections.config_dict.ConfigDict(initial_dictionary: Optional[Mapping[str, Any]] = None, type_safe: bool = True, convert_dict: bool = True)

Base class for configuration objects used in DeepMind.

This is a container for configurations. It behaves similarly to Lua tables. Specifically:

• it has dot-based access as well as dict-style key access,
• it is type safe (once a value is set one cannot change its type).

Typical usage example:

from ml_collections import config_dict

cfg = config_dict.ConfigDict()
cfg.float_field = 12.6
cfg.integer_field = 123
cfg.another_integer_field = 234
cfg.nested = config_dict.ConfigDict()
cfg.nested.string_field = 'tom'

print(cfg)

Config dictionaries can also be used to pass named arguments to functions:

from ml_collections import config_dict

def print_point(x, y):
  print "({},{})".format(x, y)

point = config_dict.ConfigDict()
point.x = 1
point.y = 2
print_point(**point)

Note that, depending on your use case, it may be easier to use the create function in this package to construct a ConfigDict:

from ml_collections import config_dict point = config_dict.create(x=1, y=2)

Differently from standard dicts, ConfigDicts also have the nice property that iterating over them is deterministic, in a fashion similar to collections.OrderedDicts.

__init__(initial_dictionary: Optional[Mapping[str, Any]] = None, type_safe: bool = True, convert_dict: bool = True)

Creates an instance of ConfigDict.

Warning: In most cases, this faithfully reproduces the reference structure of initial_dictionary, even if initial_dictionary is self-referencing. However, unexpected behavior occurs if self-references are contained within list, tuple, or custom types. For example:

d = {}
d['a'] = d
d['b'] = [d]
cd = ConfigDict(d)
cd.a    # refers to cd, type ConfigDict. Expected behavior.
cd.b    # refers to d, type dict. Unexpected behavior.

Warning: FieldReference values may be changed. If initial_dictionary contains a FieldReference with a value of type dict or FrozenConfigDict, that value is converted to ConfigDict.

Parameters:

• initial_dictionary –

  May be one of the following:

  1) dict. In this case, all values of initial_dictionary that are dictionaries are also be converted to ConfigDict. However, dictionaries within values of non-dict type are untouched.

  2) ConfigDict. In this case, all attributes are uncopied, and only the top-level object (self) is re-addressed. This is the same behavior as Python dict, list, and tuple.

  3) FrozenConfigDict. In this case, initial_dictionary is converted to a ConfigDict version of the initial dictionary for the FrozenConfigDict (reversing any mutability changes FrozenConfigDict made).

• type_safe – If set to True, once an attribute value is assigned, its type cannot be overridden without .ignore_type() context manager (default: True).
• convert_dict – If set to True, all dict used as value in the ConfigDict will automatically be converted to ConfigDict (default: True).

convert_dict

Returns True if it is converting dicts to ConfigDict automatically.

copy_and_resolve_references(visit_map=None)

Returns a ConfigDict copy with FieldReferences replaced by values.

If the object is a FrozenConfigDict, the copy returned is also a FrozenConfigDict. However, note that FrozenConfigDict should already have FieldReferences resolved to values, so this method effectively produces a deep copy.

Note: As with __eq__() and __init__(), this may not behave as expected on a ConfigDict with self-references contained in lists, tuples, or custom types.

Parameters:visit_map – A mapping from ConfigDict object ids to their copy. Method is recursive in nature, and it will call “.copy_and_resolve_references(visit_map)” on each encountered object, unless it is already in visit_map. Returns:ConfigDict copy with previous FieldReferences replaced by values.

eq_as_configdict(other)

Type-invariant equals.

This is like __eq__, except it does not distinguish FrozenConfigDict from ConfigDict. For example:

cd = ConfigDict()
fcd = FrozenConfigDict()
fcd.eq_as_configdict(cd)  # Returns True

Parameters:other – Object to compare self to. Returns:True if self == other after conversion to ConfigDict. Return type:same

get(key: str, default=None)

Returns value if key is present, or a user defined value otherwise.

get_oneway_ref(key)

Returns a one-way FieldReference.

Example:

cfg = config_dict.ConfigDict(dict(a=1))
cfg.b = cfg.get_oneway_ref('a')

cfg.a = 2
print(cfg.b)  # 2

cfg.b = 3
print(cfg.a)  # 2 (would have been 3 if using get_ref())
print(cfg.b)  # 3

Parameters:key – Key for field we want to reference.

get_type(key)

Returns type of the field associated with a key.

ignore_type()

Context manager which temporarily turns off type safety recursively.

is_locked

Returns True if object is locked.

is_type_safe

Returns True if config dict is type safe.

items(preserve_field_references=False)

Returns list of dictionary key, value pairs, sorted by key.

Parameters:preserve_field_references – (bool) Whether to preserve FieldReferences if the ConfigDict has them. By default, False: any FieldReferences will be resolved in the result. Returns:The key, value pairs in the config, sorted by key.

iteritems(preserve_field_references=False)

Deterministically iterates over dictionary key, value pairs.

Parameters:preserve_field_references – (bool) Whether to preserve FieldReferences if the ConfigDict has them. By default, False: any FieldReferences will be resolved in the result. Yields:The key, value pairs in the config, sorted by key.

iterkeys()

Deterministically iterates over dictionary keys, in sorted order.

itervalues(preserve_field_references=False)

Deterministically iterates over values in a config, sorted by their keys.

Parameters:preserve_field_references – (bool) Whether to preserve FieldReferences if the ConfigDict has them. By default, False: any FieldReferences will be resolved in the result. Yields:The values in the config, sorted by their corresponding keys.

keys()

Returns the sorted list of all the keys defined in a config.

lock() → ml_collections.config_dict.config_dict.ConfigDict

Locks object, preventing user from adding new fields.

Returns:self

to_dict(visit_map=None, preserve_field_references=False)

Converts ConfigDict to regular dict recursively with valid references.

By default, the output dict will not contain FieldReferences, any present in the ConfigDict will be resolved. However, if preserve_field_references is True, the output dict will contain FieldReferences where the original ConfigDict has them. They will not be the same as the ConfigDict’s, and their ops will be applied and dropped.

Note: As with __eq__() and __init__(), this may not behave as expected on a ConfigDict with self-references contained in lists, tuples, or custom types.

Parameters:

• visit_map – A mapping from object ids to their dict representation. Method is recursive in nature, and it will call “.to_dict(visit_map)” on each encountered object, unless it is already in visit_map.
• preserve_field_references – (bool) Whether the output dict should have FieldReferences if the ConfigDict has them. By default, False: any FieldReferences will be resolved and the result will go to the dict.

Returns:

Dictionary with the same values and references structure as a calling

ConfigDict.

to_json(json_encoder_cls=None, **kwargs)

Returns a JSON representation of the object, fails if there is a cycle.

Parameters:

• json_encoder_cls – An optional JSON encoder class to customize JSON serialization.
• **kwargs – Keyword arguments for json.dumps. They cannot contain “cls” as this method specifies it on its own.

Returns:

JSON representation of the object.

Raises:

TypeError – If self contains set, frozenset, custom type fields or any other objects that are not JSON serializable.

to_json_best_effort(**kwargs)

Returns a best effort JSON representation of the object.

Tries to serialize objects not inherently supported by JSON encoder. This may result in the configdict being partially serialized, skipping the unserializable bits. Ensures that no errors are thrown. Fails if there is a cycle.

Parameters:**kwargs – Keyword arguments for json.dumps. They cannot contain “cls” as this method specifies it on its own. Returns:JSON representation of the object.

to_yaml(**kwargs)

Returns a YAML representation of the object.

ConfigDict serializes types of fields as well as the values of fields themselves. Deserializing the YAML representation hence requires using YAML’s UnsafeLoader:

` yaml.load(cfg.to_yaml(), Loader=yaml.UnsafeLoader) `

or equivalently:

` yaml.unsafe_load(cfg.to_yaml()) `

Please see the PyYAML documentation and https://msg.pyyaml.org/load for more details on the consequences of this.

Parameters:**kwargs – Keyword arguments for yaml.dump. Returns:YAML representation of the object.

unlock() → ml_collections.config_dict.config_dict.ConfigDict

Grants user the ability to add new fields to ConfigDict.

In most cases, the unlocked() context manager should be preferred to the direct use of the unlock method.

Returns:self

unlocked()

Context manager which temporarily unlocks a ConfigDict.

update(*other, **kwargs)

Update values based on matching keys in another dict-like object.

Mimics the built-in dict’s update method: iterates over a given mapping object and adds/overwrites keys with the given mapping’s values for those keys.

Differs from dict.update in that it operates recursively on existing keys that are already a ConfigDict (i.e. calls their update() on the corresponding value from other), and respects the ConfigDict’s type safety status.

If keyword arguments are specified, the ConfigDict is updated with those key/value pairs.

Parameters:

• *other – A (single) dict-like container, e.g. a dict or ConfigDict.
• **kwargs – Additional keyword arguments to update the ConfigDict.

Raises:

TypeError – if more than one value for other is specified.

update_from_flattened_dict(flattened_dict, strip_prefix='')

In-place updates values taken from a flattened dict.

This allows a potentially nested source ConfigDict of the following form:

cfg = ConfigDict({
    'a': 1,
    'b': {
        'c': {
            'd': 2
        }
    }
})

to be updated from a dict containing paths navigating to child items, of the following form:

updates = {
    'a': 2,
    'b.c.d': 3,
    'b.c.e': 4,
}

Note that update_from_flattened_dict will allow you add (not just update) leaf nodes - for example, ‘b.c.e’ above

This filters paths_dict to only contain paths starting with strip_prefix and strips the prefix when applying the update.

For example, consider we have the following values returned as flags:

flags = {
    'flag1': x,
    'flag2': y,
    'config': 'some_file.py',
    'config.a.b': 1,
    'config.a.c': 2
}

config = ConfigDict({
    'a': {
        'b': 0,
        'c': 0
    }
})

config.update_from_flattened_dict(flags, 'config.')

Then we will now have:

config = ConfigDict({
    'a': {
        'b': 1,
        'c': 2
    }
})

Parameters:

• flattened_dict – A mapping (key path) -> value.
• strip_prefix – A prefix to be stripped from path. If specified, only paths matching strip_prefix will be processed.

Raises:

KeyError – if any of the key paths can’t be found.

values(preserve_field_references=False)

Returns the list of all values in a config, sorted by their keys.

Parameters:preserve_field_references – (bool) Whether to preserve FieldReferences if the ConfigDict has them. By default, False: any FieldReferences will be resolved in the result. Returns:The values in the config, sorted by their corresponding keys.

FrozenConfigDict class

class ml_collections.config_dict.FrozenConfigDict(initial_dictionary=None, type_safe=True)

Immutable and hashable type of ConfigDict.

See ConfigDict() documentation above for details and usage.

FrozenConfigDict is fully immutable. It contains no lists or sets (at initialization, lists and sets are converted to tuples and frozensets). The only potential sources of mutability are attributes with custom types, which are not touched.

It is recommended to convert a ConfigDict to FrozenConfigDict after construction if possible.

__init__(initial_dictionary=None, type_safe=True)

Creates an instance of FrozenConfigDict.

Lists and sets are copied into tuples and frozensets. However, copying is kept to a minimum so tuples, frozensets, and other immutable types are not copied unless they contain mutable types.

Prohibited initial_dictionary structures: initial_dictionary may not contain any lists or tuples with dictionary, ConfigDict, or FieldReference elements, or else an error is raised at initialization. It also may not contain loops in the reference structure, i.e. the reference structure must be a Directed Acyclic Graph. This includes loops in list-element and tuple-element references. initial_dictionary’s reference structure need not be a tree.

Warning: Unexpected behavior may occur with types other than Python’s built-in types. See ConfigDict() documentation for details.

Warning: As with ConfigDict, FieldReference values may be changed. If initial_dictionary contains a FieldReference with a value of type dict or ConfigDict, that value will be converted to FrozenConfigDict.

Parameters:

• initial_dictionary –

  May be one of the following:

  1) dict. In this case all values of initial_dictionary that are dictionaries are also converted to FrozenConfigDict. If there are dictionaries contained in lists or tuples, an error is raised.

  2) ConfigDict. In this case all ConfigDict attributes are also converted to FrozenConfigDict.

  3) FrozenConfigDict. In this case all attributes are uncopied, and only the top-level object (self) is re-addressed.

• type_safe – See ConfigDict documentation. Note that this only matters if the FrozenConfigDict is converted to ConfigDict at some point.

FieldReference class

class ml_collections.config_dict.FieldReference(default, field_type=None, op=None, required=False)

Reference to a configuration element.

Typed configuration element that can take a None default value. Example:

from ml_collections import config_dict

cfg_field = config_dict.FieldReference(0)
cfg = config_dict.ConfigDict({
    'optional': configdict.FieldReference(None, field_type=str)
    'field': cfg_field,
    'nested': {'field': cfg_field}
})

with self.assertRaises(TypeError):
  cfg.optional = 10  # Raises an error because it's defined as an
                      # intfield.

cfg.field = 1  # Changes the value of both cfg.field and cfg.nested.field.
print(cfg)

This class also supports lazy computation. Example:

ref = config_dict.FieldReference(0)

# Using ref in a standard operation returns another FieldReference. The new
# reference ref_plus_ten will evaluate ref's value only when we call
# ref_plus_ten.get()
ref_plus_ten = ref + 10

ref.set(3)  # change ref's value
print(ref_plus_ten.get())  # Prints 13 because ref's value is 3

ref.set(-2)  # change ref's value again
print(ref_plus_ten.get())  # Prints 8 because ref's value is -2

__init__(default, field_type=None, op=None, required=False)

Creates an instance of FieldReference.

Parameters:

• default – Default value.
• field_type – Type for the values contained by the configuration element. If None the type will be inferred from the default value. This value is used as the second argument in calls to isinstance, so it has to follow that function’s requirements (class, type or a tuple containing classes, types or tuples).
• op – An optional operation that is applied to the underlying value when get() is called.
• required – If True, the get() method will raise an error if the reference does not contain a value. This argument has no effect when a default value is provided. Setting this to True will raise an error if op is not None.

Raises:

• TypeError – If field_type is not None and is different from the type of the default value.
• ValueError – If both the default value and field_type is None.

empty()

Returns True if the reference points to a None value.

get()

Gets the value of the FieldReference object.

This will dereference _pointer and apply all ops to its value.

Returns:The result of applying all ops to the dereferenced pointer. Raises:RequiredValueError – if required is True and the underlying value for the reference is False.

has_cycle(visited=None)

Finds cycles in the reference graph.

Parameters:visited – Set containing the ids of all visited nodes in the graph. The default value is the empty set. Returns:True if there is a cycle in the reference graph.

set(value, type_safe=True)

Overwrites the value pointed by a FieldReference.

Parameters:

• value – New value.
• type_safe – Check that old and new values are of the same type.

Raises:

• TypeError – If type_safe is true and old and new values are not of the same type.
• MutabilityError – If a cycle is found in the reference graph.

Additional Methods

create(**kwargs)	Creates a ConfigDict with the given named arguments as key-value pairs.
placeholder(field_type[, required])	Defines an entry in a ConfigDict that has no value yet.
required_placeholder(field_type)	Defines an entry in a ConfigDict with unknown but required value.
recursive_rename(conf, old_name, new_name)	Returns copy of conf with old_name recursively replaced by new_name.
CustomJSONEncoder(*[, skipkeys, …])	JSON encoder for ConfigDict and FieldReference.
JSONDecodeError
MutabilityError
RequiredValueError