Source Source

Source subclasses are responsible for generating an integer value given an open source project’s source URL.

class dffml.source.source.BaseSource(config: Optional[Type[BaseConfig]])

Abstract base class for all sources. New sources must be derived from this class and implement the records method.

class dffml.source.source.BaseSourceContext(parent: BaseSource)

abstract async record(key: str)

Get a record from the source or add it if it doesn’t exist.

Examples

>>> import asyncio
>>> from dffml import *
>>>
>>> async def main():
...     async with MemorySource(records=[Record("example", data=dict(features=dict(dead="beef")))]) as source:
...         # Open, update, and close
...         async with source() as ctx:
...             example = await ctx.record("example")
...             # Let's also try calling `record` for a record that doesnt exist.
...             one = await ctx.record("one")
...             await ctx.update(one)
...             async for record in ctx.records():
...                 print(record.export())
>>>
>>> asyncio.run(main())
{'key': 'example', 'features': {'dead': 'beef'}, 'extra': {}}
{'key': 'one', 'extra': {}}

abstract async records() → AsyncIterator[Record]

Returns a list of records retrieved from self.src

Examples

>>> import asyncio
>>> from dffml import *
>>>
>>> async def main():
...     async with MemorySource(records=[Record("example", data=dict(features=dict(dead="beef")))]) as source:
...         async with source() as ctx:
...             async for record in ctx.records():
...                 print(record.export())
>>>
>>> asyncio.run(main())
{'key': 'example', 'features': {'dead': 'beef'}, 'extra': {}}

abstract async update(record: Record)

Updates a record for a source

Examples

>>> import asyncio
>>> from dffml import *
>>>
>>> async def main():
...     async with MemorySource(records=[]) as source:
...         # Open, update, and close
...         async with source() as ctx:
...             example = Record("one", data=dict(features=dict(feed="face")))
...             # ... Update one into our records ...
...             await ctx.update(example)
...             # Let's check out our records after calling `record` and `update`.
...             async for record in ctx.records():
...                 print(record.export())
>>>
>>> asyncio.run(main())
{'key': 'one', 'features': {'feed': 'face'}, 'extra': {}}

exception dffml.source.source.NoRecordsWithMatchingFeatures

Raised when SourcesContext.with_features() was called but no records were yielded.

class dffml.source.source.NoRecordsWithMatchingFeaturesHelper

Helper to print a diff of the features requested and the features available in unittest diff format

failureException

alias of NoRecordsWithMatchingFeatures

class dffml.source.source.Sources(*args)

CONTEXT

alias of SourcesContext

SINGLETON

alias of BaseSource

class dffml.source.source.SourcesContext(parent: AsyncContextManagerList)

async record(key: str)

Retrieve and or register record will all sources

async records(validation: Optional[Callable[[Record], bool]] = None) → AsyncIterator[Record]

Retrieves records from all sources

async update(record: Record)

Updates a record for a source

async with_features(features: List[str]) → AsyncIterator[Record]

Returns all records which have the requested features

class dffml.source.source.SubsetSources(*args: BaseSource, keys: Optional[List[str]] = None)

Restricts access to a subset of records during iteration based on their keys.

CONTEXT

alias of SubsetSourcesContext

class dffml.source.source.SubsetSourcesContext(parent: AsyncContextManagerList)

async records(validation: Optional[Callable[[Record], bool]] = None) → AsyncIterator[Record]

Retrieves records from all sources

class dffml.source.source.ValidationSources(validation: Callable[[Record], bool], *args: BaseSource)

Restricts access to a subset of records during iteration based on a validation function.

CONTEXT

alias of ValidationSourcesContext

class dffml.source.source.ValidationSourcesContext(parent: AsyncContextManagerList)

async records(validation: Optional[Callable[[Record], bool]] = None) → AsyncIterator[Record]

Retrieves records from all sources