Storage

The StorageTable is the abstract base for access to a records storage. Storage always means a place to put cached records from which they can be fetched again quickly.

Performance is achieved by associating each record with one or more scores, and having the StorageTable be able to quickly fetch records by a range of scores. The Interval defines such a range, and the StorageRecordsSpec specifies a set of records in storage via any number of them. Additionally, it has a recheck_predicate, which can be used to filter out records that aren’t wanted.

Each StorageTable provides a scratch space, which is a place to stage write operations that shouldn’t take effect immediately. This is used by the CachedTable during adjustments (i.e. when expiring old and loading new records). The CachedTable is meant to provide a consistent view of the records, and locking everything isn’t an option since adjustments may take a long time. The scratch space allows it to get all changes ready without affecting reads. Then, the prepared scratch space can be merged, which should be implemented to be very fast.

class tablecache.Interval

A number interval.

Represents an interval of the shape [ge,lt), i.e. with a closed lower and open upper bound.

__init__(ge, lt)

Parameters:

• ge (Score) –

• lt (Score) –

Return type:

None

covers(other)

Check whether this interval contains everything in other.

Parameters:

other (Self) –

Return type:

bool

static everything()

The interval from negative to positive infinity, covering everything.

Return type:

Self

intersects(other)

Check whether the intervals have any element in common.

Parameters:

other (Self) –

Return type:

bool

static only_containing(value)

The smallest interval containing the given value.

Return type:

Self

class tablecache.StorageRecordsSpec

A specification of records in storage.

Represents a (possibly empty) set of records in a storage table. These are all those which have an index score in the index with the given name which is contained in any of the given intervals.

Additionally, the record must satifsy the recheck predicate, i.e. it must return True when called with the record. The default recheck predicate accepts any record (i.e. only the index score is important). This predicate can be used to query the storage for a range of records that may contain some undesirable ones, and then filtering those out.

The score intervals must not overlap.

__init__(index_name, score_intervals, recheck_predicate=<function StorageRecordsSpec.always_use_record>)

Parameters:

• index_name (str) –

• score_intervals (list[Interval]) –

• recheck_predicate (RecheckPredicate) –

Return type:

None

class tablecache.StorageTable

Fast storage table.

Abstract interface for fast record storage. Offers methods to put records, get and delete records by primary key, as well as to get and delete multiple records that match score ranges. Each record is associated with one or more scores by which it can be queried. Implementations are expected to use a sorted data structure that enables fast access via those scores.

Also offers a scratch space, where records can marked to be added or deleted without affecting reads on the table until they are explicitly merged. This is meant to provide a consitent view of the data while (potentially slow) updates of the data are going on in the background. The implementation of the merge operation is expected to be relatively fast so that updates provide little disruption.

The behavior of the regular write operations (put_record() and delete_records()) is not necessarily well-defined when they occur concurrently (i.e. from separate tasks). When in doubt, locking should be used, or the scratch space, which is guaranteed to behave in the presence of multiple tasks.

abstract async clear()

Delete all data belonging to this table.

Return type:

None

abstract async delete_records(records_spec)

Delete multiple records.

Deletes exactly those records that would have been returned by get_records() when called with the same argument.

Asynchronously iterates over the records that are deleted as they exist in storage. Must be fully consumed to finish deletion.

Parameters:

records_spec (StorageRecordsSpec) – A specification of the records to delete.

Returns:

The records as they are deleted as an asynchronous iterator, in no particular order.

Return type:

AsyncIterable

abstract async get_records(records_spec)

Get multiple records.

Asynchronously iterates over all records that match the records spec. That’s all records that have a score in the specified index that is contained in one of the specified intervals, and additionally match the recheck predicate.

Records are guaranteed to be unique as long as the record spec’s intervals don’t overlap (as per their contract).

Parameters:

records_spec (StorageRecordsSpec) – A specification of the records to get.

Returns:

The requested records as an asynchronous iterator, in no particular order.

Return type:

AsyncIterable

abstract property name: str

A name for the table.

abstract async put_record(record)

Store a record.

If a record with the same primary key already exists, it is replaced.

Parameters:

record (Record) – The record to add.

Raise:

If the record is invalid in some way.

Return type:

None

abstract async scratch_discard_records(records_spec)

Mark a set of records to be deleted in scratch space.

Records marked for deletion have no effect on get operations until they are merged via scratch_merge().

This can be undone by adding the record again via scratch_put_record().

Asynchronously iterates over the records that are marked for discarding as they exist in storage. These records will continue to be available until scratch space is merged. Must be fully consumed to finish the operation.

Parameters:

records_spec (StorageRecordsSpec) – A specification of the records to mark for discarding.

Returns:

The records marked for discarding as an asynchronous iterator, in no particular order.

Return type:

AsyncIterable

abstract scratch_merge()

Merge scratch space.

Merge records added to scratch space via scratch_put_record() or marked for deletion via scratch_discard_records() so that these changes are reflected in get_record() and get_records().

This method is not async, as the switchover is meant to be fast. However, implementations may start background tasks to handle some cleanup during which further scratch operations are blocked.

Return type:

None

abstract async scratch_put_record(record)

Add a record to scratch space.

Records in scratch space have no effect on get operations until they are merged via scratch_merge().

Parameters:

record (Record) – The record to add to scratch space.

Return type:

None