Efficient storage and retrieval of genomic variant data using TileDB. Scalable VCF/BCF ingestion, incremental sample addition, compressed storage, parallel queries, and export capabilities for population genomics.
Installs just this skill. Get the whole plugin for auto-invocation.
⚡ How it fires
How this skill gets triggered: by you, by Claude, or both.
Fires itselfClaude auto-loads it when your prompt matches the work.
You can call itInvoke it directly when you want it.
Slash command/tiledbvcf
👁️ Context preview
The summary Claude sees to decide when to auto-load this skill.
Efficient storage and retrieval of genomic variant data using TileDB. Scalable VCF/BCF ingestion, incremental sample addition, compressed storage, parallel queries, and export capabilities for population genomics.
📊 Stats
Stars31,545
Forks3,146
LanguagePython
LicenseMIT
📦 Ships with scientific-agent-skills
</> SKILL.md
tiledbvcf.SKILL.md
---name: tiledbvcf
description: Efficient storage and retrieval of genomic variant data using TileDB. Scalable VCF/BCF ingestion, incremental sample addition, compressed storage, parallel queries, and export capabilities for population genomics.
license: MIT license
metadata: {"version": "1.0", "skill-author": "Jeremy Leipzig"}
---# TileDB-VCF
## Overview
TileDB-VCF is a high-performance C++ library with Python and CLI interfaces for efficient storage and retrieval of genomic variant-call data. Built on TileDB's sparse array technology, it enables scalable ingestion of VCF/BCF files, incremental sample addition without expensive merging operations, and efficient parallel queries of variant data stored locally or in the cloud.
## When to Use This Skill
This skill should be used when:
- Learning TileDB-VCF concepts and workflows
- Prototyping genomics analyses and pipelines
- Working with small-to-medium datasets (< 1000 samples)
- Need incremental addition of new samples to existing datasets
- Require efficient querying of specific genomic regions across many samples
- Working with cloud-stored variant data (S3, Azure, GCS)
- Need to export subsets of large VCF datasets
- Building variant databases for cohort studies
- Educational projects and method development
- Performance is critical for variant data operations
Create TileDB-VCF datasets and incrementally ingest variant data from multiple VCF/BCF files. This is appropriate for building population genomics databases and cohort studies.
**Requirements:**
- **Single-sample VCFs only**: Multi-sample VCFs are not supported
- **Index files required**: VCF/BCF files must have indexes (.csi or .tbi)
**Common operations:**
- Create new datasets with optimized array schemas
- Ingest single or multiple VCF/BCF files in parallel
- Add new samples incrementally without re-processing existing data
- Configure memory usage and compression settings
- Handle various VCF formats and INFO/FORMAT fields
- Resume interrupted ingestion processes
- Validate data integrity during ingestion
### 2. Efficient Querying and Filtering
Query variant data with high performance across genomic regions, samples, and variant attributes. This is appropriate for association studies, variant discovery, and population analysis.
**Common operations:**
- Query specific genomic regions (single or multiple)
- Filter by sample names or sample groups
- Extract specific variant attributes (position, alleles, genotypes, quality)
- Access INFO and FORMAT fields efficiently
- Combine spatial and attribute-based filtering
- Stream large query results
- Perform aggregations across samples or regions
### 3. Data Export and Interoperability
Export data in various formats for downstream analysis or integration with other genomics tools. This is appropriate for sharing datasets, creating analysis subsets, or feeding other pipelines.
**Common operations:**
- Export to standard VCF/BCF formats
- Generate TSV files with selected fields
- Create sample/region-specific subsets
- Maintain data provenance and metadata
- Lossless data export preserving all annotations
- Compressed output formats
- Streaming exports for large datasets
### 4. Population Genomics Workflows
TileDB-VCF excels at large-scale population genomics analyses requiring efficient access to variant data across many samples and genomic regions.
**Common workflows:**
- Genome-wide association studies (GWAS) data preparation
- Rare variant burden testing
- Population stratification analysis
- Allele frequency calculations across populations
- Quality control across large cohorts
- Variant annotation and filtering
- Cross-population comparative analysis
## Key Concepts
### Array Schema and Data Model
**TileDB-VCF Data Model:**
- Variants stored as sparse arrays with genomic coordinates as dimensions
- Samples stored as attributes allowing efficient sample-specific queries
- INFO and FORMAT fields preserved with original data types
- Automatic compression and chunking for optimal storage
**Schema Configuration:**
```python
# Custom schema with specific tile extents
config = tiledbvcf.ReadConfig(
memory_budget=2048, # MB
region_partition=(0, 3095677412), # Full genome
sample_partition=(0, 10000) # Up to 10k samples
)
```
### Coordinate Systems and Regions
**Critical:** TileDB-VCF uses **1-based genomic coordinates** following VCF standard:
- Positions are 1-based (first base is position 1)
- Ranges are inclusive on both ends
- Region "chr1:1000-2000" includes positions 1000-2000 (1001 bases total)
**Region specification formats:**
```python
# Single region
regions = ["chr1:1000000-2000000"]
# Multiple regions
regions = ["chr1:1000000-2000000", "chr2:500000-1500000"]
- TileDB Academy (All Documentation): https://cloud.tiledb.com/academy/
**Getting Started:**
- Free account signup: https://cloud.tiledb.com
- Contact: sales@tiledb.com for enterprise needs
## Scaling to TileDB-Cloud
When your genomics workloads outgrow single-node processing, TileDB-Cloud provides enterprise-scale capabilities for production genomics pipelines.
**Note**: This section covers TileDB-Cloud capabilities based on available documentation. For complete API details and current functionality, consult the official TileDB-Cloud documentation and API reference.
### Setting Up TileDB-Cloud
**1. Create Account and Get API Token**
```bash
# Sign up at https://cloud.tiledb.com
# Generate API token in your account settings
```
**2. Install TileDB-Cloud Python Client**
```bash
# Base installation
pip install tiledb-cloud
# With genomics-specific functionality
pip install tiledb-cloud[life-sciences]
```
**3. Configure Authentication**
```bash
# Set environment variable with your API token
export TILEDB_REST_TOKEN="your_api_token"
```
```python
import tiledb.cloud
# Authentication is automatic via TILEDB_REST_TOKEN
# No explicit login required in code
```
### Migrating from Open Source to TileDB-Cloud
**Large-Scale Ingestion**
```python
# TileDB-Cloud: Distributed VCF ingestion
import tiledb.cloud.vcf
# Use specialized VCF ingestion module
# Note: Exact API requires TileDB-Cloud documentation
# This represents the available functionality structure
tiledb.cloud.vcf.ingestion.ingest_vcf_dataset(
source="s3://my-bucket/vcf-files/",
output="tiledb://my-namespace/large-dataset",
namespace="my-namespace",
acn="my-s3-credentials",
ingest_resources={"cpu": "16", "memory": "64Gi"}
)
```
**Distributed Query Processing**
```python
# TileDB-Cloud: VCF querying across distributed storage
