dlt.destinations.impl.lancedb.lancedb_adapter

ROLLBACK_TIMEOUT_SECONDS

How long to wait for a restore to become visible to the managed client.

lancedb_adapter

def lancedb_adapter(data: Any,
                    embed: TColumnNames = None,
                    merge_key: TColumnNames = None,
                    remove_orphans: Optional[bool] = None) -> DltResource

View source on GitHub

Prepares data for the LanceDB destination by specifying which columns should be embedded.

Arguments:

• data Any - The data to be transformed. It can be raw data or an instance of DltResource. If raw data, the function wraps it into a DltResource object.
• embed TColumnNames, optional - Specify columns to generate embeddings for. It can be a single column name as a string, or a list of column names.
• merge_key TColumnNames, optional - Specify columns to merge on. It can be a single column name as a string, or a list of column names.
• remove_orphans Optional[bool] - Whether a merge deletes the chunks a reloaded document no longer produces. Defaults to None, which turns it on for a single merge key.

Returns:

• DltResource - A resource with applied LanceDB-specific hints.

Raises:

• ValueError - If input for embed invalid or empty.

Examples:

    data = [{"name": "Marcel", "description": "Moonbase Engineer"}]
    lancedb_adapter(data, embed="description")

[DltResource with hints applied]

rollback_to_commit_tag

def rollback_to_commit_tag(
        dataset: "dlt.Dataset",
        tag: str,
        timeout: float = ROLLBACK_TIMEOUT_SECONDS) -> List[str]

View source on GitHub

Rolls every table of a LanceDB dataset back to the version named by tag.

A rollback appends a new version holding the tagged contents, so nothing is destroyed and the rollback itself can be undone. It waits for the managed client to see the restored version, because a load started inside that window fails.

Warnings:

LanceDB has no transaction spanning tables, so a failure part way through leaves the dataset partly rolled back. The names of the tables already restored are logged and returned, and running this again is safe.

Arguments:

• dataset dlt.Dataset - Dataset to roll back, from pipeline.dataset() or dlt.dataset(...).
• tag str - Commit tag naming the version to restore, as written by commit_tag.
• timeout float - Seconds to wait for the restore to become visible, per table.

Returns:

• List[str] - Names of the tables that were restored.

Raises:

• DestinationTerminalException - If no table of the dataset carries tag.

Example:

    import dlt
    from dlt.destinations.impl.lancedb.lancedb_adapter import rollback_to_commit_tag
    pipeline = dlt.pipeline("movies", destination="lancedb", dataset_name="analytics")
    rollback_to_commit_tag(pipeline.dataset(), "nightly")

['movies', '_dlt_loads']