sio3pack.workflow.execution =========================== .. py:module:: sio3pack.workflow.execution Package Contents ---------------- .. py:class:: Channel(buffer_size, source_pipe, target_pipe, file_buffer_size = None, limit = None) A configuration of a channel. A channel is a connection between two pipes. :param int buffer_size: The maximum amount of data stored in the channel that has been written by the writer, but not yet read by the reader. This value must be positive. :param int source_pipe: The pipe this channel will be reading from. :param int target_pipe: The pipe this channel will be writing to. :param int file_buffer_size: Controls whether this channel is backed by a file on the disk. A larger buffer may then be allocated on the disk. :param int limit: Limits the maximum amount of data sent through the channel. .. py:method:: from_json(data) :classmethod: Create a new channel from a dictionary. :param dict data: The dictionary to create the channel from. .. py:method:: to_json() Convert the channel to a dictionary. .. py:class:: DescriptorManager(objects_manager, filesystem_manager) A class to manage file descriptors and their associated streams. :param ObjectsManager objects_manager: The objects manager. :param FilesystemManager filesystem_manager: The filesystem manager. .. py:method:: add(fd, stream) Add a stream to the descriptor manager. :param int fd: The file descriptor. :param Stream stream: The stream to add. .. py:method:: from_json(data) Load the descriptor manager from a JSON-serializable dictionary. :param dict data: The JSON-serializable dictionary to load from. .. py:method:: to_json() Convert the descriptor manager to a JSON-serializable dictionary. .. py:method:: items() Get the items in the descriptor manager. :return: A view of the descriptor manager's items. .. py:method:: all() Get all the streams in the descriptor manager. :return: A dictionary of all the streams in the descriptor manager. .. py:method:: get(fd) Get a stream by its file descriptor. :param int fd: The file descriptor. :return: The stream associated with the file descriptor. .. py:method:: size() Get the number of streams in the descriptor manager. :return: The number of streams in the descriptor manager. .. py:class:: EmptyFilesystem(id = None) Bases: :py:obj:`Filesystem` A base class to represent a filesystem. :param int id: The id of the filesystem in the task. .. py:method:: from_json(data, id, workflow) :classmethod: Create a new empty filesystem from a dictionary. :param data: The dictionary to create the empty filesystem from. :param id: The id of the empty filesystem. :param workflow: The workflow the empty filesystem belongs to. .. py:method:: to_json() Convert the empty filesystem to a dictionary. .. py:class:: Filesystem(id = None) A base class to represent a filesystem. :param int id: The id of the filesystem in the task. .. py:method:: from_json(data, id, workflow) :classmethod: Create a new filesystem from a dictionary. :param dict data: The dictionary to create the filesystem from. :param int id: The id of the filesystem. :param Workflow workflow: The workflow the filesystem belongs to. .. py:method:: to_json() Convert the filesystem to a dictionary. :return: The dictionary representation of the filesystem. .. py:method:: replace_templates(replacements) Replace strings in the task with the given replacements. :param replacements: The replacements to make. .. py:class:: FilesystemManager(task) A class to manage filesystems. :param Task task: The task the filesystem manager belongs to. :param list[Filesystem] filesystems: The list of filesystems. .. py:method:: from_json(data, workflow) Create filesystems from a list of dictionaries. :param list[dict] data: The list of dictionaries to create the filesystems from. :param Workflow workflow: The workflow the filesystems belong to. .. py:method:: to_json() Convert the filesystems to a list of dictionaries. .. py:method:: get_by_id(id) Get a filesystem by id. :param id: The id of the filesystem. .. py:method:: add(filesystem) Add a filesystem to the manager. :param filesystem: The filesystem to add. .. py:method:: replace_templates(replacements) Replace strings in the task with the given replacements. :param replacements: The replacements to make. .. py:method:: len() Get the number of filesystems. .. py:method:: has_by_id(id) Check if a filesystem with the given id exists. :param id: The id of the filesystem to check. :return: True if the filesystem exists, False otherwise. .. py:class:: ImageFilesystem(image, path = None, id = None) Bases: :py:obj:`Filesystem` A class to represent an image filesystem. An image can be for example a g++ compilator, a python interpreter, etc. :param int id: The id of the image filesystem. :param str image: The image to use. :param str path: The path to the image. If None, the path is "". .. py:method:: from_json(data, id, workflow) :classmethod: Create a new image filesystem from a dictionary. :param dict data: The dictionary to create the image filesystem from. :param id id: The id of the image filesystem. :param Workflow workflow: The workflow the image filesystem belongs to. .. py:method:: to_json() Convert the image filesystem to a dictionary. :return: The dictionary representation of the image filesystem. .. py:class:: ObjectFilesystem(object, id = None) Bases: :py:obj:`Filesystem` A base class to represent a filesystem. :param int id: The id of the filesystem in the task. .. py:method:: from_json(data, id, workflow) :classmethod: Create a new object filesystem from a dictionary. :param data: The dictionary to create the object filesystem from. :param id: The id of the object filesystem. :param workflow: The workflow the object filesystem belongs to. .. py:method:: to_json() Convert the object filesystem to a dictionary. .. py:method:: replace_templates(replacements) Replace strings in the task with the given replacements. :param replacements: The replacements to make. .. py:class:: MountNamespace(mountpoints = None, root = 0, id = None) A class to represent a mount namespace. It can mount :class:`Mountpoint` instances in the target filesystem. :param int id: The id of the mount namespace. :param list[Mountpoint] mountpoints: The mountpoints in the mount namespace. .. py:method:: from_json(data, id, filesystem_manager) :classmethod: Create a new mount namespace from a dictionary. :param data: The dictionary to create the mount namespace from. :param id: The id of the mount namespace. :param filesystem_manager: The filesystem manager to use. .. py:method:: to_json() Convert the mount namespace to a dictionary. .. py:method:: add_mountpoint(mountpoint) Add a mountpoint to the mount namespace. :param mountpoint: The mountpoint to add. .. py:class:: MountNamespaceManager(task, filesystem_manager) .. py:method:: from_json(data) Create a new mount namespace manager from a list of dictionaries. :param data: The list of dictionaries to create the mount namespace manager from. .. py:method:: add(mount_namespace) Add a mount namespace to the manager. :param MountNamespace mount_namespace: The mount namespace to add. .. py:method:: get_by_id(id) Get a mount namespace by its id. :param id: The id of the mount namespace. .. py:method:: to_json() Convert the mount namespace manager to a dictionary. .. py:method:: len() Get the number of mount namespaces. .. py:class:: Mountpoint(source, target, writable = False, capacity = None) A class to represent a mountpoint. :param "Filesystem" source: The source filesystem. :param str target: The target path in the filesystem. :param bool writable: Whether the mountpoint is writable or not. :param int capacity: The capacity of the mountpoint. If None, the capacity is unlimited. .. py:method:: from_json(data, filesystem_manager) :classmethod: Create a new mountpoint from a dictionary. :param dict data: The dictionary to create the mountpoint from. :param FilesystemManager filesystem_manager: The filesystem manager to use. .. py:method:: to_json() Convert the mountpoint to a dictionary. .. py:class:: Process(workflow, task, arguments = None, environment = None, image = '', mount_namespace = None, resource_group = None, pid_namespace = 0, working_directory = '/', start_after = None) A class to represent a process in a workflow. :param Workflow workflow: The workflow the process belongs to. :param Executiontask task: The task which the process belongs to. :param list[str] arguments: Executable arguments for the process. :param dict[str, str] environment: Environment variables for the process. :param str image: The image of the process, which can be a Docker image or similar. :param MountNamespace mount_namespace: The mount namespace to use for the process. :param ResourceGroup resource_group: The resource group of the process. :param str working_directory: The working directory of the process. :param int pid_namespace: The PID namespace of the process. :param list[int] start_after: The processes that must be finished before this process starts. .. py:method:: to_json() Convert the process to a JSON-serializable dictionary. .. py:method:: from_json(data, workflow, task) :classmethod: Create a new process from a dictionary. :param data: The dictionary to create the process from. :param workflow: The workflow the process belongs to. :param task: The task the process belongs to. .. py:method:: replace_templates(replacements) Replace strings in the process with the given replacements. :param dict[str, str] replacements: The replacements to make. .. py:class:: ResourceGroup(cpu_usage_limit = 100.0, instruction_limit = 1000000000.0, memory_limit = 2147483648, oom_terminate_all_tasks = False, pid_limit = 2, swap_limit = 0, time_limit = 1000000000.0, id = None) A resource group is a set of limits that can be applied to a task. It can limit CPU usage, instruction usage, memory usage, and more. :param id: The id of the resource group. :param cpu_usage_limit: The CPU usage limit. :param instruction_limit: The instruction usage limit. :param memory_limit: The memory limit. :param oom_terminate_all_tasks: Whether to terminate all tasks on OOM. :param pid_limit: The PID limit. :param swap_limit: The swap limit. :param time_limit: The time limit. .. py:method:: set_limits(cpu_usage_limit, instruction_limit, memory_limit, time_limit) Set the limits of the resource group. :param cpu_usage_limit: The CPU usage limit. :param instruction_limit: The instruction usage limit. :param memory_limit: The memory limit. :param time_limit: The time limit. .. py:method:: from_json(data, id) :classmethod: Create a new resource group from a dictionary. :param data: The dictionary to create the resource group from. :param id: The id of the resource group. .. py:method:: to_json() Convert the resource group to a dictionary. .. py:class:: ResourceGroupManager(task) A class to manage resource groups in a workflow. Allows creation, retrieval, and management of resource groups. .. py:method:: add(resource_group) Add a resource group to the resource group manager. :param ResourceGroup resource_group: The resource group to add. .. py:method:: get_by_id(id) Get a resource group by its id. :param id: The id of the resource group to get. .. py:method:: to_json() Convert the resource group manager to a dictionary. .. py:method:: from_json(data) Create a new resource group manager from a list of dictionaries. :param data: The list of dictionaries to create the resource group manager from. .. py:method:: all() Get all resource groups. .. py:class:: FileMode(*args, **kwds) Bases: :py:obj:`enum.Enum` Enum representing the mode of a file stream. .. py:class:: FileStream(filesystem, path, mode) Bases: :py:obj:`Stream` Class representing a file stream. A file will be opened and passed to the process as a file descriptor. :param Filesystem filesystem: The filesystem to use. :param str path: The path to the file. :param FileMode mode: The mode to open the file in. .. py:method:: from_json(filesystem_manager, data) :classmethod: Create a file stream from a JSON-serializable dictionary. :param FilesystemManager filesystem_manager: The filesystem manager. :param dict data: The JSON-serializable dictionary to create the file stream from. .. py:method:: to_json() Convert the file stream to a dictionary. :return: The dictionary representation of the file stream. .. py:class:: NullStream Bases: :py:obj:`Stream` Class representing a null stream. .. py:method:: from_json(data) :classmethod: Create a null stream from a JSON-serializable dictionary. :param dict data: The JSON-serializable dictionary to create the null stream from. .. py:method:: to_json() Convert the null stream to a dictionary. :return: The dictionary representation of the null stream. .. py:class:: ObjectReadStream(object) Bases: :py:obj:`ObjectStream` Class representing an object read stream. An object read stream is a stream that reads from an object via a file descriptor. :param Object object: The object to read from. .. py:class:: ObjectStream(type, object) Bases: :py:obj:`Stream` A base class for object streams. An object stream is a stream that reads or writes to an object via a file descriptor. :param StreamType type: The type of the stream. :param Object object: The object to use. .. py:method:: from_json(data, objects_manager) :classmethod: Create an object stream from a JSON-serializable dictionary. :param dict data: The JSON-serializable dictionary to create the object stream from. .. py:method:: to_json() Convert the object stream to a dictionary. .. py:method:: replace_templates(replacements) Replace strings in the object stream with the given replacements. .. py:class:: ObjectWriteStream(object) Bases: :py:obj:`ObjectStream` Class representing an object write stream. An object write stream is a stream that writes to an object via a file descriptor. :param Object object: The object to write to. .. py:class:: PipeReadStream(pipe_index) Bases: :py:obj:`PipeStream` Class representing a pipe read stream. A pipe read stream is a stream that reads from a pipe via a file descriptor. :param int pipe_index: The index of the pipe. .. py:class:: PipeStream(type, pipe_index) Bases: :py:obj:`Stream` A base class for pipe streams. A pipe stream is a stream that reads or writes to a pipe via a file descriptor. :param StreamType type: The type of the stream. :param int pipe_index: The index of the pipe. .. py:method:: from_json(data) :classmethod: Create a pipe stream from a JSON-serializable dictionary. :param dict data: The JSON-serializable dictionary to create the pipe stream from. .. py:method:: to_json() Convert the pipe stream to a dictionary. .. py:class:: PipeWriteStream(pipe_index) Bases: :py:obj:`PipeStream` Class representing a pipe write stream. A pipe write stream is a stream that writes to a pipe via a file descriptor. :param int pipe_index: The index of the pipe. .. py:class:: Stream(type) Base class for all streams. :param StreamType type: The type of the stream. .. py:method:: from_json(data, objects_manager, filesystem_manager) :classmethod: Create a stream from a JSON-serializable dictionary. :param dict data: The JSON-serializable dictionary to create the stream from. :param ObjectsManager objects_manager: The objects manager. :param FilesystemManager filesystem_manager: The filesystem manager. .. py:method:: to_json() :abstractmethod: Convert the stream to a JSON-serializable dictionary. .. py:method:: replace_templates(replacements) Replace strings in the stream with the given replacements. This method is a no-op for streams that do not support template replacement. .. py:class:: StreamType(*args, **kwds) Bases: :py:obj:`enum.Enum` Enum representing the type of stream.