Provenance: Calibration Data Lineage

The Provenance system tracks how calibration parameters are generated and modified, based on W3C PROV-DM (Provenance Data Model) concepts. It enables tracing parameter value history, impact analysis across parameters, root cause identification, and execution comparison.

W3C PROV-DM Concepts

QDash provenance is based on three core concepts (Entity, Activity, Agent) and their relations. See the diagram below for the full architecture and lineage graph example:

Provenance Lineage

Mapping to QDash

PROV-DM Concept	QDash Implementation	Description
Entity	`ParameterVersion`	A specific version of a calibration parameter
Activity	`Activity`	Execution of a calibration task
Agent	User/System	User or system that executed the task

Relation Types

Relation	Meaning	Example
`wasGeneratedBy`	Entity was generated by Activity	`qubit_frequency:Q0:v3` was generated by `CheckFrequency` task
`wasDerivedFrom`	Entity was derived from Entity	`t1:Q0:v2` was derived from `qubit_frequency:Q0:v3`
`used`	Activity used Entity	`CheckT1` task used `qubit_frequency:Q0:v3`

Architecture

Components

The Provenance architecture (API Layer → ProvenanceService → Repositories → MongoDB collections) is shown in the diagram above.

Data Models

ParameterVersion (Entity)

python

class ParameterVersion:
    entity_id: str          # "qubit_frequency:Q0:exec123:task456"
    parameter_name: str     # "qubit_frequency"
    qid: str               # "Q0" or "Q0-Q1"
    value: float | int | str
    unit: str              # "Hz", "s", etc.
    version: int           # Auto-incrementing version number
    valid_from: datetime   # When this version became active
    valid_until: datetime | None  # None if current version
    execution_id: str      # Execution that created this version
    task_id: str           # Task that created this version
    task_name: str         # Human-readable task name
    project_id: str
    chip_id: str

Entity ID Format

{parameter_name}:{qid}:{execution_id}:{task_id}

Example: qubit_frequency:Q0:exec_20240115_001:check_freq_task

ProvenanceRelation

python

class ProvenanceRelation:
    relation_type: str     # "wasGeneratedBy", "wasDerivedFrom", "used"
    source_id: str         # Source entity/activity ID
    source_type: str       # "entity" or "activity"
    target_id: str         # Target entity/activity ID
    target_type: str       # "entity" or "activity"
    project_id: str

API Endpoints

Entity Operations

Endpoint	Method	Description
`/provenance/entities/{entity_id}`	GET	Get a specific parameter version
`/provenance/history`	GET	Get parameter version history

Graph Traversal

Endpoint	Method	Description
`/provenance/lineage/{entity_id}`	GET	Trace ancestors (inputs)
`/provenance/impact/{entity_id}`	GET	Trace descendants (outputs)

Comparison & Analytics

Endpoint	Method	Description
`/provenance/compare`	GET	Parameter diff between two executions
`/provenance/stats`	GET	Provenance statistics
`/provenance/executions`	GET	Recent execution IDs
`/provenance/changes`	GET	Recent parameter changes with delta

Usage Examples

1. Get Parameter History

python

# GET /provenance/history?parameter_name=qubit_frequency&qid=Q0&limit=10

response = {
    "parameter_name": "qubit_frequency",
    "qid": "Q0",
    "versions": [
        {
            "entity_id": "qubit_frequency:Q0:exec003:task789",
            "value": 5.123e9,
            "version": 3,
            "valid_from": "2024-01-15T10:30:00Z",
            "task_name": "CheckFrequency"
        },
        {
            "entity_id": "qubit_frequency:Q0:exec002:task456",
            "value": 5.121e9,
            "version": 2,
            "valid_from": "2024-01-14T15:00:00Z",
            "task_name": "CheckFrequency"
        }
    ],
    "total_versions": 3
}

2. Explore Lineage

python

# GET /provenance/lineage/t1:Q0:exec003:task_t1?max_depth=3

response = {
    "origin": {
        "node_type": "entity",
        "node_id": "t1:Q0:exec003:task_t1",
        "entity": {
            "parameter_name": "t1",
            "value": 50e-6,
            "task_name": "CheckT1"
        }
    },
    "nodes": [
        {"node_type": "activity", "node_id": "activity:check_t1:exec003"},
        {"node_type": "entity", "node_id": "qubit_frequency:Q0:exec003:task_freq"}
    ],
    "edges": [
        {
            "relation_type": "wasGeneratedBy",
            "source_id": "t1:Q0:exec003:task_t1",
            "target_id": "activity:check_t1:exec003"
        },
        {
            "relation_type": "used",
            "source_id": "activity:check_t1:exec003",
            "target_id": "qubit_frequency:Q0:exec003:task_freq"
        }
    ]
}

3. Compare Executions

python

# GET /provenance/compare?execution_id_before=exec001&execution_id_after=exec002

response = {
    "execution_id_before": "exec001",
    "execution_id_after": "exec002",
    "added_parameters": [
        {"parameter_name": "t2_echo", "qid": "Q0", "value_after": 80e-6}
    ],
    "removed_parameters": [],
    "changed_parameters": [
        {
            "parameter_name": "qubit_frequency",
            "qid": "Q0",
            "value_before": 5.121e9,
            "value_after": 5.123e9,
            "delta": 2e6,
            "delta_percent": 0.039
        }
    ],
    "unchanged_count": 15
}

4. Monitor Recent Changes

python

# GET /provenance/changes?limit=5&within_hours=24

response = {
    "changes": [
        {
            "entity_id": "qubit_frequency:Q0:exec003:task789",
            "parameter_name": "qubit_frequency",
            "qid": "Q0",
            "value": 5.123e9,
            "previous_value": 5.121e9,
            "delta": 2e6,
            "delta_percent": 0.039,
            "version": 3,
            "task_name": "CheckFrequency",
            "valid_from": "2024-01-15T10:30:00Z"
        }
    ],
    "total_count": 5
}

Lineage Graph Visualization

The lineage graph example showing cross-execution parameter tracing (Execution 002 → 003) with wasGeneratedBy, used, and wasDerivedFrom relations is included in the Provenance Lineage diagram above.

Database Collections

parameter_versions

javascript

{
    "_id": ObjectId("..."),
    "entity_id": "qubit_frequency:Q0:exec003:task789",
    "parameter_name": "qubit_frequency",
    "qid": "Q0",
    "value": 5.123e9,
    "value_type": "float",
    "unit": "Hz",
    "error": 1e3,
    "version": 3,
    "valid_from": ISODate("2024-01-15T10:30:00Z"),
    "valid_until": null,
    "execution_id": "exec003",
    "task_id": "task789",
    "task_name": "CheckFrequency",
    "project_id": "project_alpha",
    "chip_id": "64Qv3"
}

Indexes:

{ project_id: 1, parameter_name: 1, qid: 1, version: -1 }
{ project_id: 1, execution_id: 1 }
{ entity_id: 1 } (unique)
{ valid_from: -1 }

provenance_relations

javascript

{
    "_id": ObjectId("..."),
    "relation_type": "wasGeneratedBy",
    "source_id": "qubit_frequency:Q0:exec003:task789",
    "source_type": "entity",
    "target_id": "activity:check_freq:exec003",
    "target_type": "activity",
    "project_id": "project_alpha"
}

Indexes:

{ project_id: 1, source_id: 1 }
{ project_id: 1, target_id: 1 }
{ relation_type: 1 }

activities

javascript

{
    "_id": ObjectId("..."),
    "activity_id": "activity:check_freq:exec003",
    "execution_id": "exec003",
    "task_id": "task789",
    "task_name": "CheckFrequency",
    "task_type": "calibration",
    "qid": "Q0",
    "started_at": ISODate("2024-01-15T10:29:00Z"),
    "ended_at": ISODate("2024-01-15T10:30:00Z"),
    "status": "completed",
    "project_id": "project_alpha",
    "chip_id": "64Qv3"
}

Implementation Files

Component	File Path
Router	`src/qdash/api/routers/provenance.py`
Service	`src/qdash/api/services/provenance_service.py`
Schemas	`src/qdash/api/schemas/provenance.py`
Repository	`src/qdash/repository/provenance.py`
UI Components	`ui/src/components/features/provenance/`
API Client	`ui/src/client/provenance/provenance.ts`

Frontend Components

Component	Purpose
`ProvenancePageContent`	Main page container
`ExecutionComparisonPanel`	Execution comparison UI
`LineageExplorerPanel`	Lineage exploration UI
`ParameterHistoryPanel`	Parameter history display
`ProvenanceGraph`	Graph visualization

Implementation Notes

Entity ID Format

Always generate Entity IDs using the same format:

python

def generate_entity_id(
    parameter_name: str,
    qid: str,
    execution_id: str,
    task_id: str
) -> str:
    return f"{parameter_name}:{qid}:{execution_id}:{task_id}"

Version Management

For the same (parameter_name, qid) combination, version auto-increments
valid_until is set when the next version is created
The currently active version has valid_until = null

Recording Relations

When a calibration task completes, record the following:

wasGeneratedBy: Generated parameter → Activity
used: Activity → Input parameters used
wasDerivedFrom: New version → Previous version (for same parameter)

Provenance: Calibration Data Lineage ​

W3C PROV-DM Concepts ​

Mapping to QDash ​

Relation Types ​

Architecture ​

Components ​

Data Models ​

ParameterVersion (Entity) ​

Entity ID Format ​

ProvenanceRelation ​

API Endpoints ​

Entity Operations ​

Graph Traversal ​

Comparison & Analytics ​

Usage Examples ​

1. Get Parameter History ​

2. Explore Lineage ​

3. Compare Executions ​

4. Monitor Recent Changes ​

Lineage Graph Visualization ​

Database Collections ​

parameter_versions ​

provenance_relations ​

activities ​

Implementation Files ​

Frontend Components ​

Implementation Notes ​

Entity ID Format ​

Version Management ​

Recording Relations ​

Provenance: Calibration Data Lineage

W3C PROV-DM Concepts

Mapping to QDash

Relation Types

Architecture

Components

Data Models

ParameterVersion (Entity)

Entity ID Format

ProvenanceRelation

API Endpoints

Entity Operations

Graph Traversal

Comparison & Analytics

Usage Examples

1. Get Parameter History

2. Explore Lineage

3. Compare Executions

4. Monitor Recent Changes

Lineage Graph Visualization

Database Collections

parameter_versions

provenance_relations

activities

Implementation Files

Frontend Components

Implementation Notes

Entity ID Format

Version Management

Recording Relations