Trees

All emdfile data classes inherit from the Node class, which adds core tree-building and metadata storage functionality. EMD trees must begin at an instance of the Root class. To define new classes which make use of the emdfile read and save methods, see the Node docstring. The Custom class enables composition of emdfile classes.

Node

class emdfile.classes.Node(name: str | None = 'node')

Base class for all EMD data object classes.

Nodes contain the machinery to create and modify filetree-like relations between themselves and other Node instances; to store arbitrary metadata; and to read/write themselves from/to HDF5 files.

EMD Trees

Nodes can be added to other nodes to create parent-child relations. EMD Trees must begin with a Root type node. So

>>> root = Root()
>>> node = Node('node1')
>>> root.tree(node)

creates the data tree

root
  |---node1

and

>>> ...
>>> _node = Node('node2')
>>> _node.tree(node)

extends the tree to

root
  |---node1
        |---node2

The node.tree method enables displaying the tree, fetching another node, cutting a branch of the tree off to create a new tree, or grafting to/from another tree or subtree. Usage includes

>>> node.tree()                # show the tree downstream of this node
>>> node.tree(show=True)       # show the full tree from the root node
>>> node.tree(show=False)      # show from current node
>>> node.tree('path/to/node')  # return the node at the chosen location
>>> node.tree('/path/to/node') # specifiy the location starting from root
>>> node.tree(node)            # add a child node; must be a Node instance
>>> node.tree(cut=True)        # remove & return a branch; include root metadata
>>> node.tree(cut=False)       # discard root metadata
>>> node.tree(cut='copy')      # copy root metadata
>>> node.tree(graft=node)      # remove & graft a branch; add new root metadata
>>> node.tree(graft=(node,True))    # as above
>>> node.tree(graft=(node,False))   # discard root metadata
>>> node.tree(graft=(node,'copy'))  # copy new root metadata
>>> node.tree(graft=(node,'overwrite'))  # add root metadata, overwrite conflicts
>>> node.tree(graft=(node,'copyover'))  # copy root metadata, overwrite conflicts

Showing the tree, retrieving or adding nodes, and cutting or grafting branches can equivalently be performed using the

>>> node.show_tree
>>> node.get_from_tree
>>> node.add_to_tree
>>> node.cut
>>> node.graft

methods. See their docstrings for more info, including discussion of root metadata merge conflict handling in cut and graft operations.

Roots

A node’s root can be returned with

>>> node.root

The default root is None, and such nodes are called unrooted. Unrooted nodes can’t perform tree operations other than be added to a root or rooted node. If written to file, a root will be added.

A root’s metadata is considered associated with all nodes in its tree, and any write operations emanating from this tree will include its root metadata. Cut and merge node.tree behaviors include several root metadata handling options - see those docstrings for details.

Metadata

Metadata can be added to a Node using

>>> node.metadata = Metadata(name='new_metadata', data={'item':1})

which then becomes accessible at

>>> node.metadata['new_metadata']

Any number of Metadata instances may be added in this way, and each Metadata instance can contain an arbitrary tree of items - see the Metadata docstring.

Inspect all the Metadata instances in a node with

>>> node.metadata

which returns a dictionary.

I/O

Nodes include their own read/write machinery which is used by the read and save functions. To generate new classes which will read and write seemlessly into EMD files using these functions, the new class should inherit from an appropirate emdfile class, then modify the class I/O methods as appropriate - see Downstream Integration below for details.

I/O: Downstream Integration

Each Node contains

>>> node.to_h5
>>> node.from_h5

methods, which write class instances to HDF5 files and generate new instances from appropriately formatted HDF5 groups, respectively. Each expects an h5py.Group argument.

.to_h5 creates a new child HDF5 group and populates it with top level tags and all Metadata instances in node.metadata. If the node has a data-containing subtype like Array, PointList, PointListArray, or Custom, these will add their own data - see the .to_h5 docstrings for individual classes for details.

If additional information needs to be stored in some new class inheriting from an emdfile class, the .to_h5 method should be modified by overwriting it in the class definition. The new method should include

>>> def to_h5(self, group):
>>>     '''describe the data being included '''
>>>     ...
>>>     grp = ParentClass.to_h5(self, group)
>>>     ...

where ParentClass is the parent class. Alternatively, the Python super().to_h5 syntax can be used, however in cases of multiple inheritance you should then be mindful of which class super() will refer to, which is determined by the method resolution order rules. This method call creates the HDF5 group you’re writing to and adds all the data normally included. Add any additional required code, then conclude the method definition with

>>> return grp

A simple way to include additional data is to add it as metadata before calling super().to_h5. For instance, for a class with some attribute x that needs to be stored write

>>> def to_h5(self, grou):
>>>     '''store attribute x'''
>>>     self.metadata = Metadata(
>>>         name = 'class_attributes',
>>>         data = {'x' : self.x}
>>>     )
>>>     grp = super().to_h5(self, group)
>>>     return grp

You’ll then need to make sure this data is read successfully by modifying another method, discussed next.

The .from_h5 method should not require modification in most instances - instead, its two helper methods should be overwritten. When run, node.from_h5 calls

>>> node._get_constructor_args
>>> node._populate_instance

The first method must return the arguments which will be passed to the class __init__ method. The second method is called post-instatiation to perform any additional setup or modification of the new instance.

In the example above, to ensure the x value is correctly populated after reading a class instance’s data from a file, you can use

>>> def _populate_instance(self, group):
>>>     md = self.metadata['class_attributes']
>>>     self.x = md['x']

In general a new class may have different constructor arguments than the emdfile node type it inherits from. In this case, modify ._get_constructor_args to return a dictionary of keywords and values which will be passed to the class __init__ method at read time.

The definition should begin

>>> @classmethod
>>> def _get_constructor_args(cls, group):

and end

>>> return args

where args is a dictionary. You can retrieve the constructor arguments from the parent class with

>>> parent_class_args = ParentClass._get_constructor_args(group)

The result is a dictionary of names and values corresponding to the parent class, i.e. if the superclass is Array the returned keys are ‘data’, ‘name’, ‘units’, ‘dims’, ‘dim_names’, ‘dim_units’, and ‘slicelabels’.

Downstream Integration: Hook

To integrate a Python module with emdfile, add

>>> _emd_hook = True

as a global variable in tje top-level namespace, e.g. in the top level __init__.py file. This ensures that any classes you define inheriting from emdfile will be discoverable by the emdfile.read method.

Extras: @newnode wrapper

Sometimes a node class method will return data which is itself an emdfile node. Decorating such a method with

>>> @newnode

modifies the method such that it (1) adds the new node to the tree of the generating node, and (2) adds metadata describing how the new node was made to itself, available at .metadata['_generating_md'].

add_to_tree(node)

Add node to the current tree as a child of this node. Note that if node has a root, this will error out - in this case, use either .graft or .force_add_to_tree instead.

cut(root_metadata=True)

Removes from the tree the branch beginning at this node, and returns it as a new tree. A new root is created at the base of the tree named ‘{old_root_name}_cut_{this_node_name}’.

Parameters:

root_metadata (True, False, or 'copy') – Specifies root metadata handling. If True, adds metadata from the original tree root to the new root. If False, adds no metadata. If ‘copy’, copies metadata from the old root to the new root.

Return type:

(Root) the new root node

force_add_to_tree(node)

Add node to the current tree as a child of this node, whether or not node has a root. If it has no root, performs a simple add. If has a root, performs a graft, excluding the root metadata from node. Note that this means the branch downstream of node will also be moved to the current tree.

classmethod from_h5(group)

Takes an h5py.Group which is open in read mode. Confirms that a Node of this name exists in this group, and loads and returns it with it’s metadata.

Parameters:

group (h5py Group)

Return type:

(Node)

get_from_tree(name)

Finds and returns the node from the current tree matching name, which must be a string with ‘/’ delimiters between ‘parent/child’ nodes. Search from the current node, or from the root node if name begins with ‘/’. So

>>> self.get_from_tree('x/y')

fetches node ‘y’ which is under node ‘x’ which is under the current node, and

>>> self.get_from_tree('/a/b')

fetches node ‘b’ which is under ‘a’ which is under the root node.

graft(node, merge_metadata=True)

Grafts a branch from one EMD tree onto another. The branch beginning at node is moved onto this tree underneath this node.

For the reverse - i.e. grafting from this tree to another tree - either use that tree’s .graft method, or use this tree’s ._graft.

Parameters:

• node (Node)

• merge_metadata (True, False, 'copy', or 'overwrite') – Specifies how root metadata should be treated. If True, adds the scion (incoming) root metadata to the stock (receiving) root, skipping entries that exist in both. If False, adds no metadata. If “overwrite”, entries existing in both scion and stock root metadata are overwritten. If “copy”, scion root metadata are copied to the stock root, skipping conflicts. If “copyover”, conflicting scion/stock roots are also copied and overwritten.

Return type:

(Root) this tree’s root node

static newnode(method)

Decorated methods must return a new node. After decoration the new node will be added to the parent node’s tree with a Metadata instance storing information about how the node was created, namely, the method’s name, the parent’s class and name, and all arguments passed to the method.

show_tree(root=False)

Display the object tree. If root is False, displays the branch of the tree downstream from this node, and if True, displays the full tree from the root node.

to_h5(group)

Creates a subgroup in group and writes this node into that group, including the group tags (emd_group_type, python_class), and the node’s metadata. No data beyond metadata and tags is written.

Parameters:

group (h5py Group)

Return type:

(h5py Group) the new node’s Group

tree(arg=None, **kwargs)

Usages -

>>> node.tree()                # show the tree downstream of this node
>>> node.tree(show=True)       # show the full tree from the root node
>>> node.tree(show=False)      # show from current node
>>> node.tree('path/to/node')  # return the node at the chosen location
>>> node.tree('/path/to/node') # specifiy the location starting from root
>>> node.tree(node)            # add a child node; must be a Node instance
>>> node.tree(cut=True)        # remove & return a branch; include root metadata
>>> node.tree(cut=False)       # discard root metadata
>>> node.tree(cut='copy')      # copy root metadata
>>> node.tree(graft=node)      # remove & graft a branch; add new root metadata
>>> node.tree(graft=(node,False))   # discard root metadata
>>> node.tree(graft=(node,'copy'))  # copy new root metadata
>>> node.tree(graft=(node,'overwrite'))  # add root metadata, overwrite conflicts
>>> node.tree(graft=(node,'copyover'))  # copy root metadata, overwrite conflicts

See the node.show, node.cut, node.graft, and node.add_to_tree docstrings for more info.

Root

class emdfile.classes.Root(name='root')

A Root is a Node with its .root property pointing to itself. EMD trees must begin with a Root, and tree building operations will raise an Exception if no root is present. Once a node is added to a tree, it can’t be removed or added to another tree unless a graft, cut, or force_add operation is used.

A Root instance’s metadata is considered associated with all nodes in its tree, and any write operation emanating from this tree will include its root metadata. Cut and merge node.tree include several handling options for root metadata - see those docstrings for details.

Custom

class emdfile.classes.Custom(name='custom')

The purpose of Custom nodes is to support creation of new classes by composition of emdfile classes. To do so, create a class inheriting from Custom, then add attributes which are emdfile classes, for instance

>>> class X(Custom):
>>>     def __init__(self, name, asize, bsize):
>>>         Custom.__init__(self, name=name)
>>>         self.a = Array(np.ones((asize)),'a')
>>>         self.b = Array(np.ones((bsize)),'b')

For some X instance x, on write x.a and x.b will be saved, along with any other attributes pointing to Node instances.

Downstream Integration

See the Node docstring. Node-like attributes will be written to file by the Custom.to_h5 method, however, to ensure those attributes are correctly read the new class must modify the ._get_constructor_args and ._populate_instance methods.

._get_emd_attr_data provides access to node-like attributes, which it returns as a dictionary. So for class X above, we could use

>>> @classmethod
>>> def _get_constructor_args(cls, group):
>>>     d = cls._get_emd_attr_data(group)
>>>     args = {
>>>         'a' : d['a'].shape,
>>>         'b' : d['b'].shape
>>>     }
>>> return args

to ensure the right arguments are passed to X.__init__. If for some X instance x the x.a and x.b attribute data changed after instantiation but before it was written to disk, these will need to be populated, which can be accomoplished e.g. with

>>> def _populate_instance(self, group):
>>>     d = cls._get_emd_attr_data(group)
>>>     self.a = d['a']
>>>     self.b = d['b']

to_h5(group)

Calls Node.to_h5 to greate the group’s node and write its metadata. Then writes every attribute yielding a Node instance using that node’s .to_h5 method.

Parameters:

group (h5py Group)

Return type:

(h5py Group) the new node’s Group