Df Memory¶
- class dffml.df.memory.BaseMemoryDataFlowObject(config: Optional[Type[BaseConfig]])[source]¶
- class dffml.df.memory.MemoryDataFlowObjectContextConfig(uid: str)[source]¶
- no_enforce_immutable()¶
By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.
Examples
>>> from dffml import config >>> >>> @config ... class MyConfig: ... C: int >>> >>> config = MyConfig(C=2) >>> with config.no_enforce_immutable(): ... config.C = 1
- class dffml.df.memory.MemoryDefinitionSetContext(config: BaseConfig, parent: BaseInputNetworkContext, ctx: BaseInputSetContext)[source]¶
- async inputs(definition: Definition) AsyncIterator[Input] [source]¶
Asynchronous iterator of all inputs within a context, which are of a definition.
- class dffml.df.memory.MemoryInputNetwork(config: Optional[Type[BaseConfig]])[source]¶
Inputs backed by a set
- CONFIG¶
alias of
MemoryInputNetworkConfig
- CONTEXT¶
alias of
MemoryInputNetworkContext
- class dffml.df.memory.MemoryInputNetworkConfig[source]¶
- no_enforce_immutable()¶
By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.
Examples
>>> from dffml import config >>> >>> @config ... class MyConfig: ... C: int >>> >>> config = MyConfig(C=2) >>> with config.no_enforce_immutable(): ... config.C = 1
- class dffml.df.memory.MemoryInputNetworkContext(config: BaseConfig, parent: MemoryInputNetwork)[source]¶
- async add(input_set: BaseInputSet)[source]¶
Adds new input set to the network
- async added(watch_ctx: BaseInputSetContext) Tuple[bool, BaseInputSet] [source]¶
Returns when a new input set has entered the network within a context
- async cadd(ctx, *args: Input)[source]¶
Shorthand for creating a MemoryInputSet with an existing context.
>>> import asyncio >>> from dffml import * >>> >>> async def main(): ... async with MemoryOrchestrator() as orchestrator: ... async with orchestrator(DataFlow.auto()) as octx: ... await octx.ictx.sadd(StringInputSetContext("Hi")) >>> >>> asyncio.run(main())
- async ctx() Tuple[bool, BaseInputSetContext] [source]¶
Returns when a new input set context has entered the network
- async definition(ctx: BaseInputSetContext, definition: str) Definition [source]¶
Search for the definition within a context given its name as a string. Return the definition. Otherwise raise a DefinitionNotInContext error. If the context is not present, raise a ContextNotPresent error.
- definitions(ctx: BaseInputSetContext) BaseDefinitionSetContext [source]¶
Return a DefinitionSet context that can be used to access the inputs within the given context, by definition.
- async gather_inputs(rctx: BaseRedundancyCheckerContext, operation: Operation, dataflow: DataFlow, ctx: Optional[BaseInputSetContext] = None) AsyncIterator[BaseParameterSet] [source]¶
Generate all possible permutations of applicable inputs for an operation that, according to the redundancy checker, haven’t been run yet.
- async receive_from_parent_flow(inputs: List[Input])[source]¶
Takes input from parent dataflow and adds it to every active context
- async sadd(context_handle_string, *args: Input)[source]¶
Shorthand for creating a MemoryInputSet with a StringInputSetContext.
>>> import asyncio >>> from dffml import * >>> >>> async def main(): ... async with MemoryOrchestrator() as orchestrator: ... async with orchestrator(DataFlow.auto()) as octx: ... await octx.ictx.sadd("Hi") >>> >>> asyncio.run(main())
- class dffml.df.memory.MemoryInputNetworkContextEntry(ctx, definitions, by_origin)[source]¶
- property by_origin¶
Alias for field number 2
- property ctx¶
Alias for field number 0
- property definitions¶
Alias for field number 1
- class dffml.df.memory.MemoryInputSet(config: MemoryInputSetConfig)[source]¶
-
- async remove_unvalidated_inputs() MemoryInputSet [source]¶
Removes unvalidated inputs from internal list and returns the same.
- class dffml.df.memory.MemoryInputSetConfig(ctx: dffml.df.base.BaseInputSetContext, inputs: List[dffml.df.types.Input])[source]¶
- no_enforce_immutable()¶
By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.
Examples
>>> from dffml import config >>> >>> @config ... class MyConfig: ... C: int >>> >>> config = MyConfig(C=2) >>> with config.no_enforce_immutable(): ... config.C = 1
- class dffml.df.memory.MemoryKeyValueStore(config: Optional[Type[BaseConfig]])[source]¶
Key Value store backed by dict
- CONFIG¶
alias of
MemoryKeyValueStoreConfig
- CONTEXT¶
alias of
MemoryKeyValueStoreContext
- class dffml.df.memory.MemoryKeyValueStoreConfig[source]¶
- no_enforce_immutable()¶
By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.
Examples
>>> from dffml import config >>> >>> @config ... class MyConfig: ... C: int >>> >>> config = MyConfig(C=2) >>> with config.no_enforce_immutable(): ... config.C = 1
- class dffml.df.memory.MemoryKeyValueStoreContext(config: BaseConfig, parent: MemoryKeyValueStore)[source]¶
- class dffml.df.memory.MemoryLockNetwork(config: Optional[Type[BaseConfig]])[source]¶
- CONFIG¶
alias of
MemoryLockNetworkConfig
- CONTEXT¶
alias of
MemoryLockNetworkContext
- class dffml.df.memory.MemoryLockNetworkConfig[source]¶
- no_enforce_immutable()¶
By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.
Examples
>>> from dffml import config >>> >>> @config ... class MyConfig: ... C: int >>> >>> config = MyConfig(C=2) >>> with config.no_enforce_immutable(): ... config.C = 1
- class dffml.df.memory.MemoryLockNetworkContext(config: BaseConfig, parent: MemoryLockNetwork)[source]¶
- acquire(parameter_set: BaseParameterSet)[source]¶
Acquire the lock for each input in the input set which must be locked prior to running an operation using the input.
- class dffml.df.memory.MemoryOperationImplementationNetwork(config: Optional[Type[BaseConfig]])[source]¶
- CONFIG¶
- CONTEXT¶
- class dffml.df.memory.MemoryOperationImplementationNetworkConfig(operations: Dict[str, dffml.df.base.OperationImplementation] = <factory>)[source]¶
- no_enforce_immutable()¶
By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.
Examples
>>> from dffml import config >>> >>> @config ... class MyConfig: ... C: int >>> >>> config = MyConfig(C=2) >>> with config.no_enforce_immutable(): ... config.C = 1
- class dffml.df.memory.MemoryOperationImplementationNetworkContext(config: BaseConfig, parent: MemoryOperationImplementationNetwork)[source]¶
- async contains(operation: Operation) bool [source]¶
Checks if operation in is operations we have loaded in memory
- async dispatch(octx: BaseOrchestratorContext, operation: Operation, parameter_set: BaseParameterSet)[source]¶
Schedule the running of an operation
- async ensure_contains(operation: Operation)[source]¶
Raise errors if we don’t have and can’t instantiate an operation.
- async instantiable(operation: Operation, *, opimp: Optional[OperationImplementation] = None) bool [source]¶
Looks for class registered with ____ entrypoint using pkg_resources.
- async instantiate(operation: Operation, config: BaseConfig, *, opimp: Optional[OperationImplementation] = None) bool [source]¶
Instantiate class registered with ____ entrypoint using pkg_resources. Return true if instantiation was successful.
- async run(ctx: BaseInputSetContext, octx: BaseOrchestratorContext, operation: Operation, inputs: Dict[str, Any]) Union[bool, Dict[str, Any]] [source]¶
Run an operation in our network.
- async run_dispatch(octx: BaseOrchestratorContext, operation: Operation, parameter_set: BaseParameterSet, set_valid: bool = True)[source]¶
Run an operation in the background and add its outputs to the input network when complete
- class dffml.df.memory.MemoryOperationNetwork(config: Optional[Type[BaseConfig]])[source]¶
Operations backed by a set
- CONFIG¶
alias of
MemoryOperationNetworkConfig
- CONTEXT¶
alias of
MemoryOperationNetworkContext
- class dffml.df.memory.MemoryOperationNetworkConfig(operations: List[dffml.df.types.Operation] = <factory>)[source]¶
- no_enforce_immutable()¶
By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.
Examples
>>> from dffml import config >>> >>> @config ... class MyConfig: ... C: int >>> >>> config = MyConfig(C=2) >>> with config.no_enforce_immutable(): ... config.C = 1
- class dffml.df.memory.MemoryOperationNetworkContext(config: BaseConfig, parent: MemoryOperationNetwork)[source]¶
-
- async operations(dataflow: DataFlow, *, input_set: Optional[BaseInputSet] = None, stage: Stage = Stage.PROCESSING) AsyncIterator[Operation] [source]¶
Retrieve all operations in the network of a given stage filtering by operations who have inputs with definitions in the input set.
- class dffml.df.memory.MemoryOrchestrator(config: Optional[BaseConfig])[source]¶
- CONFIG¶
alias of
MemoryOrchestratorConfig
- CONTEXT¶
alias of
MemoryOrchestratorContext
- class dffml.df.memory.MemoryOrchestratorConfig(input_network: dffml.df.base.BaseInputNetwork = <factory>, operation_network: dffml.df.base.BaseOperationNetwork = <factory>, lock_network: dffml.df.base.BaseLockNetwork = <factory>, opimp_network: dffml.df.base.BaseOperationImplementationNetwork = <factory>, rchecker: dffml.df.base.BaseRedundancyChecker = <factory>, max_ctxs: int = None)[source]¶
- no_enforce_immutable()¶
By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.
Examples
>>> from dffml import config >>> >>> @config ... class MyConfig: ... C: int >>> >>> config = MyConfig(C=2) >>> with config.no_enforce_immutable(): ... config.C = 1
- class dffml.df.memory.MemoryOrchestratorContext(config: MemoryOrchestratorContextConfig, parent: BaseOrchestrator)[source]¶
-
- async initialize_dataflow(dataflow: DataFlow) None [source]¶
Initialize a DataFlow by preforming the following steps.
Add operations the operation network context
Instantiate operation implementations which are not instantiated within the operation implementation network context
Seed input network context with given inputs
- async operations_parameter_set_pairs(ctx: BaseInputSetContext, dataflow: DataFlow, *, new_input_set: Optional[BaseInputSet] = None, stage: Stage = Stage.PROCESSING) AsyncIterator[Tuple[Operation, BaseInputSet]] [source]¶
Use new_input_set to determine which operations in the network might be up for running. Cross check using existing inputs to generate per input set context novel input pairings. Yield novel input pairings along with their operations as they are generated.
- async run(*input_sets: Union[List[Input], BaseInputSet], strict: bool = True, ctx: Optional[BaseInputSetContext] = None, halt: Optional[Event] = None) AsyncIterator[Tuple[BaseContextHandle, Dict[str, Any]]] [source]¶
Run a DataFlow.
- class dffml.df.memory.MemoryOrchestratorContextConfig(uid: str, dataflow: dffml.df.types.DataFlow, reuse: Dict[str, dffml.df.base.BaseDataFlowObjectContext] = None, max_ctxs: int = None)[source]¶
- no_enforce_immutable()¶
By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.
Examples
>>> from dffml import config >>> >>> @config ... class MyConfig: ... C: int >>> >>> config = MyConfig(C=2) >>> with config.no_enforce_immutable(): ... config.C = 1
- class dffml.df.memory.MemoryParameterSet(config: MemoryParameterSetConfig)[source]¶
- class dffml.df.memory.MemoryParameterSetConfig(ctx: dffml.df.base.BaseInputSetContext, parameters: List[dffml.df.types.Parameter])[source]¶
- no_enforce_immutable()¶
By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.
Examples
>>> from dffml import config >>> >>> @config ... class MyConfig: ... C: int >>> >>> config = MyConfig(C=2) >>> with config.no_enforce_immutable(): ... config.C = 1
- class dffml.df.memory.MemoryRedundancyChecker(config)[source]¶
Redundancy Checker backed by Memory Key Value Store
- CONFIG¶
alias of
MemoryRedundancyCheckerConfig
- CONTEXT¶
alias of
MemoryRedundancyCheckerContext
- class dffml.df.memory.MemoryRedundancyCheckerConfig(kvstore: dffml.df.base.BaseKeyValueStore = <factory>)[source]¶
- no_enforce_immutable()¶
By default, all properties of a config object are immutable. If you would like to mutate immutable properties, you must explicitly call this method using it as a context manager.
Examples
>>> from dffml import config >>> >>> @config ... class MyConfig: ... C: int >>> >>> config = MyConfig(C=2) >>> with config.no_enforce_immutable(): ... config.C = 1
- class dffml.df.memory.MemoryRedundancyCheckerContext(config: BaseConfig, parent: MemoryRedundancyChecker)[source]¶
- async unique(operation: Operation, parameter_set: BaseParameterSet) str [source]¶
SHA384 hash of the parameter set context handle as a string, the operation.instance_name, and the sorted list of input uuids.