Local storage

This submodule provides the LocalStorageTable, an implementation of StorageTable that stores its records in local Python data structures. It uses the sortedcontainers library for its indexes, which is installed as a dependency when the local extra is selected.

class tablecache.local.LocalStorageTable

Bases: StorageTable, Generic

A StorageTable that stores its data in native Python data structures.

This implementation of StorageTable uses sortedcontainers.SortedKeyList s to enable fast access to records via their scores. Using native data structures has the advantage that each index can store direct references to all records, so there is no additional redirection necessary when getting records via indexes.

Records inserted into the table are stored as-is, without any explicit validation. As long as it’s possible to calculate their scores and extract a primary key using the record scorer, they are accepted. It is up to the user to ensure that records are complete.

Read operations return the exact same record instances that were inserted. In case they are mutable, they must not be modified while they reside in storage. Make a copy. More specifically, if a record is modified in a way that changes its score for any index, that index becomes inconsistent and the record may not be returned in a read operation when it should be.

Regular write operations (put_record(), delete_records()) are blocked while scratch space is active (i.e. between the first call to scratch_put_record() or scratch_discard_records() and the subsequent call to scratch_merge()). They will resume once the merge completes. The merge is done in a background task, which shuffles some data around. While this task runs, scratch operations are blocked.

Read operations are generally prioritized over write operations. get_records() is never blocked entirely, although it may need to wait momentarily to take away a lock from an ongoing merge operation. Asynchronous write operations that can take a while (the scratch merge task in particular) regularly yield back to the event loop when it is safe, to allow read operations to jump in.

__init__(record_scorer, *, table_name=None)

Parameters:

• record_scorer (RecordScorer) – A RecordScorer used to calculate a record’s scores for all the indexes that need to be represented in storage. The score function must not raise exceptions, or the storage may be left in an undefined state.

• table_name (str) – Name of the table. Only informational. If not given, a random UUID string is generated.

Return type:

None

async clear()

Delete all data belonging to this table.

Return type:

None

async delete_records(records_spec)

Delete multiple records.

Asynchronously iterates over the records that are deleted as they exist in storage. Must be fully consumed to finish deletion.

This operation will block while scratch space is active and resume after the scratch merge finishes.

Internally, first finds all records matching records_spec, then deletes them. If another task adds a record after that first step, this record will not be deleted by this operation. Similarly, if another task deletes one of the records after that first step, this operation will attempt to delete it again. This won’t fail, but it will inflate the number of records that is returned.

Parameters:

records_spec (StorageRecordsSpec) – A specification of the records to delete.

Returns:

The records as they are deleted as an asynchronous iterator, in no particular order.

Return type:

AsyncIterable

async get_records(records_spec)

Get multiple records.

Asynchronously iterates over all records that match the records spec. That’s all records that have a score in the specified index that is contained in one of the specified intervals, and additionally match the recheck predicate.

Records are guaranteed to be unique as long as the record spec’s intervals don’t overlap (as per their contract).

Parameters:

records_spec (StorageRecordsSpec) – A specification of the records to get.

Returns:

The requested records as an asynchronous iterator, in no particular order.

Return type:

AsyncIterable

property name: str

A name for the table.

async put_record(record)

Store a record.

This operation will block while scratch space is active and resume after the scratch merge finishes.

Parameters:

record (Record) – The record to add.

Return type:

None

async scratch_discard_records(records_spec)

Mark a set of records to be deleted in scratch space.

Asynchronously iterates over the records that are marked for discarding as they exist in storage. This may include records that have already been marked for discarding. These records will continue to be available until scratch space is merged. Must be fully consumed to finish the operation.

This operation will block while a merge background task is running.

Parameters:

records_spec (StorageRecordsSpec) – A specification of the records to mark for discarding.

Returns:

The records marked for discarding as an asynchronous iterator, in no particular order.

Return type:

AsyncIterable

scratch_merge()

Merge scratch space.

This immediately causes read operations to reflect the state of the table that includes modifications in scratch space.

Spawns a background task that shuffles some data around. Until this completes, all write operations are locked.

Return type:

None

async scratch_put_record(record)

Add a record to scratch space.

This operation will block while a merge background task is running.

Parameters:

record (Record) – The record to add to scratch space.

Return type:

None