Qdrant Manager

Agent: AIEngineer

Manage Qdrant Cloud collections for the Physical AI textbook RAG system. This skill handles vectorization of textbook chapters, metadata tagging, and collection management for efficient retrieval-augmented generation.

Quick Setup

# Set environment variables
export QDRANT_URL="https://xxx.qdrant.io"
export QDRANT_API_KEY="your-api-key"
export OPENAI_API_KEY="sk-..."

# Initialize collection
.claude/skills/qdrant-manager/scripts/setup.sh --init

# Vectorize all chapters
.claude/skills/qdrant-manager/scripts/setup.sh --vectorize docs/

# Query the collection
.claude/skills/qdrant-manager/scripts/setup.sh --query "What is ROS 2?"

Command Options

| Option | Description | Default | |--------|-------------|---------| | --init | Initialize/create the collection | - | | --vectorize PATH | Vectorize markdown files in PATH | - | | --query TEXT | Query the collection | - | | --delete | Delete the collection | - | | --info | Show collection info | - | | --list | List all collections | - | | --collection NAME | Collection name | textbook_chapters | | --model MODEL | Embedding model | text-embedding-3-small | | --batch-size N | Batch size for vectorization | 10 | | --top-k N | Number of results for query | 5 | | -h, --help | Show help message | - |

What It Does

1. Collection Management

Creates and manages Qdrant collections with optimized settings for textbook content:

# Collection schema
{
    "vectors": {
        "size": 1536,  # text-embedding-3-small
        "distance": "Cosine"
    },
    "on_disk_payload": True  # Optimize for large documents
}

2. Chapter Vectorization

Processes markdown files with intelligent chunking:

Chunk Strategy: Split by headers (##, ###) for semantic coherence
Overlap: 100 tokens overlap between chunks
Max Chunk Size: 500 tokens per chunk
Metadata Extraction: Title, chapter, section, language, keywords

3. Metadata Schema

Each vector point includes rich metadata for filtered retrieval:

{
    "id": "uuid-v4",
    "vector": [0.1, 0.2, ...],
    "payload": {
        "content": "The actual text content...",
        "source": "docs/chapter-1/index.md",
        "chapter": 1,
        "section": "Introduction",
        "title": "Introduction to Physical AI & ROS 2",
        "language": "en",
        "locale": "en-US",
        "keywords": ["ros2", "physical-ai", "robotics"],
        "word_count": 245,
        "created_at": "2026-01-02T00:00:00Z",
        "chunk_index": 0,
        "total_chunks": 5
    }
}

4. RAG Query Support

Semantic search with metadata filtering:

# Example: Query with language filter
results = client.search(
    collection_name="textbook_chapters",
    query_vector=embed("What is ROS 2?"),
    query_filter={
        "must": [
            {"key": "language", "match": {"value": "en"}}
        ]
    },
    limit=5
)

Bundled Resources

1. Python Dependencies

File: requirements.txt

qdrant-client>=1.7.0
openai>=1.0.0
tiktoken>=0.5.0
python-dotenv>=1.0.0
rich>=13.0.0

2. Collection Configuration

File: assets/collection_config.json

{
    "vectors": {
        "size": 1536,
        "distance": "Cosine"
    },
    "optimizers_config": {
        "indexing_threshold": 20000
    },
    "on_disk_payload": true,
    "replication_factor": 1
}

3. Chunking Configuration

File: assets/chunking_config.json

{
    "chunk_size": 500,
    "chunk_overlap": 100,
    "separators": ["## ", "### ", "\n\n", "\n"],
    "length_function": "tiktoken",
    "model": "text-embedding-3-small"
}

4. Vectorization Script

File: scripts/vectorize.py

from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance
from openai import OpenAI
import tiktoken
import uuid
import json
from pathlib import Path

class TextbookVectorizer:
    def __init__(self, qdrant_url, qdrant_api_key, openai_api_key):
        self.qdrant = QdrantClient(url=qdrant_url, api_key=qdrant_api_key)
        self.openai = OpenAI(api_key=openai_api_key)
        self.encoder = tiktoken.encoding_for_model("text-embedding-3-small")

    def create_collection(self, name: str, vector_size: int = 1536):
        self.qdrant.create_collection(
            collection_name=name,
            vectors_config=VectorParams(size=vector_size, distance=Distance.COSINE)
        )

    def chunk_markdown(self, content: str, source: str) -> list:
        # Split by headers for semantic coherence
        chunks = []
        current_chunk = ""
        current_section = "Introduction"

        for line in content.split("\n"):
            if line.startswith("## "):
                if current_chunk:
                    chunks.append({"content": current_chunk, "section": current_section})
                current_section = line[3:].strip()
                current_chunk = line + "\n"
            else:
                current_chunk += line + "\n"
                if len(self.encoder.encode(current_chunk)) > 500:
                    chunks.append({"content": current_chunk, "section": current_section})
                    current_chunk = ""

        if current_chunk:
            chunks.append({"content": current_chunk, "section": current_section})

        return chunks

    def embed(self, text: str) -> list:
        response = self.openai.embeddings.create(
            input=text,
            model="text-embedding-3-small"
        )
        return response.data[0].embedding

    def vectorize_file(self, filepath: Path, collection: str):
        content = filepath.read_text()
        chunks = self.chunk_markdown(content, str(filepath))

        points = []
        for i, chunk in enumerate(chunks):
            vector = self.embed(chunk["content"])
            points.append(PointStruct(
                id=str(uuid.uuid4()),
                vector=vector,
                payload={
                    "content": chunk["content"],
                    "source": str(filepath),
                    "section": chunk["section"],
                    "chunk_index": i,
                    "total_chunks": len(chunks)
                }
            ))

        self.qdrant.upsert(collection_name=collection, points=points)
        return len(points)

5. Input Variables

| Variable | Required | Default | Description | |----------|----------|---------|-------------| | qdrant_url | Yes | - | Qdrant Cloud cluster URL | | qdrant_api_key | Yes | - | Qdrant API key | | collection_name | No | textbook_chapters | Collection name | | embedding_model | No | text-embedding-3-small | OpenAI embedding model | | vector_size | No | 1536 | Vector dimension |

Usage Instructions

Step 1: Set Environment Variables

# Add to .env file
QDRANT_URL=https://your-cluster.qdrant.io
QDRANT_API_KEY=your-api-key
OPENAI_API_KEY=sk-...

Step 2: Initialize Collection

.claude/skills/qdrant-manager/scripts/setup.sh --init --collection textbook_chapters

Step 3: Vectorize Chapters

# Vectorize English chapters
.claude/skills/qdrant-manager/scripts/setup.sh --vectorize docs/

# Vectorize Urdu translations
.claude/skills/qdrant-manager/scripts/setup.sh --vectorize i18n/ur/docusaurus-plugin-content-docs/current/

Step 4: Query for RAG

# Simple query
.claude/skills/qdrant-manager/scripts/setup.sh --query "How do I install ROS 2?"

# Query with top-k
.claude/skills/qdrant-manager/scripts/setup.sh --query "Gazebo simulation" --top-k 10

Step 5: Verify Collection

# Show collection info
.claude/skills/qdrant-manager/scripts/setup.sh --info

# List all collections
.claude/skills/qdrant-manager/scripts/setup.sh --list

Verification Checklist

[ ] QDRANT_URL environment variable set
[ ] QDRANT_API_KEY environment variable set
[ ] OPENAI_API_KEY environment variable set
[ ] Collection created successfully
[ ] Chapters vectorized without errors
[ ] Query returns relevant results
[ ] Metadata filtering works correctly

Integration with RAG Chatbot

This skill provides the vector storage for the RAG chatbot (Feature 002). The chatbot queries this collection to retrieve relevant context for answering user questions.

# In RAG chatbot
from qdrant_client import QdrantClient

client = QdrantClient(url=QDRANT_URL, api_key=QDRANT_API_KEY)

def get_context(query: str, language: str = "en") -> list:
    query_vector = embed(query)
    results = client.search(
        collection_name="textbook_chapters",
        query_vector=query_vector,
        query_filter={"must": [{"key": "language", "match": {"value": language}}]},
        limit=5
    )
    return [hit.payload["content"] for hit in results]

Troubleshooting

| Issue | Solution | |-------|----------| | Connection refused | Check QDRANT_URL is correct and cluster is running | | Authentication failed | Verify QDRANT_API_KEY is valid | | Embedding rate limit | Reduce --batch-size or add delays | | Collection not found | Run --init first to create collection | | Out of memory | Use smaller --batch-size | | Slow vectorization | Use --batch-size 20 for faster processing |

Requirements

Python 3.9+
Qdrant Cloud account (or self-hosted Qdrant)
OpenAI API key (for embeddings)
pip packages: qdrant-client, openai, tiktoken

Cost Considerations

| Resource | Cost | |----------|------| | Qdrant Cloud Free Tier | 1GB storage, 1M vectors | | OpenAI text-embedding-3-small | $0.02 / 1M tokens | | Estimated for 3 chapters | ~$0.01 (one-time) |

Qdrant Documentation
OpenAI Embeddings
Skill: vercel-fastapi-link - Backend API with RAG endpoints
Feature: 002-rag-chatbot - RAG chatbot specification

Changelog

v1.0.0 (2026-01-02)

Initial Release

Collection creation and management
Markdown chunking with header-based splitting
OpenAI embedding integration (text-embedding-3-small)
Metadata extraction (chapter, section, language, keywords)
Query support with filtering
Batch vectorization with progress tracking