This is a submission for the AssemblyAI Voice Agents Challenge

What I Built

I created a real-time voice-controlled spell casting system that transforms spoken Harry Potter spells into instant keyboard commands. This project addresses the Real-Time Performance category by achieving ultra-low latency voice recognition for gaming applications where every millisecond matters.

The system recognizes over 30 different spells (like "Lumos", "Wingardium Leviosa", "Stupefy") and instantly triggers corresponding game actions through keyboard shortcuts. It features advanced fuzzy matching to handle pronunciation variations and partial transcript processing for immediate response - perfect for immersive gaming experiences.

Demo

🎥 YouTube Demo Video - Watch the spell casting in action!

Key features demonstrated:

• ⚡ Sub-300ms response time from speech to action
• 🎯 Accurate recognition of complex spell names
• 🔄 Handles pronunciation variations and partial words
• 🛡️ Smart spam prevention for rapid casting
• 🎮 Seamless integration with game controls

GitHub Repository

Technical Implementation & AssemblyAI Integration

Core Architecture

The system leverages AssemblyAI's Universal-Streaming technology with aggressive optimization for minimal latency:

# Optimized streaming parameters for ultra-low latency
client.connect(
    StreamingParameters(
        sample_rate=16000,
        format_turns=True,
        # Aggressive turn detection for faster response
        end_of_turn_confidence_threshold=0.5,  # Lower threshold for faster detection
        min_end_of_turn_silence_when_confident=100,  # Reduced from 160ms
        max_turn_silence=1500,  # Reduced from 2400ms
    )
)

Real-Time Processing Innovation

The key innovation is dual-layer processing that handles both partial and complete transcripts:

def on_turn(self: Type[StreamingClient], event: TurnEvent):
    transcript = event.transcript
    is_partial = not event.end_of_turn

    if is_partial:
        # Process partial transcript for immediate response (min 4 characters)
        if len(transcript) >= 4:
            print(f"👂 Partial: {transcript}")
            if process_transcript(transcript, confidence_threshold=0.8, is_partial=True):
                print("✨ Spell cast from partial transcript!")
    else:
        # Process complete transcript with lower threshold
        print(f"🗣️ Complete: {transcript}")
        process_transcript(transcript, confidence_threshold=0.6, is_partial=False)

Intelligent Spell Matching

I implemented an optimized fuzzy matching system that prioritizes speed:

def optimized_fuzzy_match(text, spell_list, threshold=0.6):
    text = text.lower().strip()

    # First, try exact matches in pre-computed variations
    if text in SPELL_VARIATIONS:
        return SPELL_VARIATIONS[text]

    # Quick substring check for common patterns
    for spell in spell_list:
        if spell in text or text in spell:
            if len(text) >= len(spell) * 0.7:  # At least 70% of spell length
                return spell

    # Fallback to SequenceMatcher only when needed
    # ... fuzzy matching logic

Performance Optimizations

1. Pre-computed Spell Variations: Common spell variations are cached for instant lookup
2. Spam Prevention: Prevents accidental rapid-fire casting with time-based cooldowns
3. Early Exit Logic: Avoids expensive fuzzy matching when exact matches are found
4. Partial Processing: Acts on partial transcripts for sub-300ms response times

AssemblyAI Features Utilized

• Universal-Streaming: Core real-time transcription with 300ms latency
• Turn Detection: Intelligent endpointing for natural speech flow
• High Accuracy: Handles complex fantasy terminology and pronunciation variations
• Partial Transcripts: Enables immediate response without waiting for complete utterances

Results

The system consistently achieves sub-300ms latency from speech input to game action, making spell casting feel truly magical and responsive. The combination of AssemblyAI's ultra-fast streaming with optimized processing creates an immersive gaming experience where voice commands feel as natural as pressing keys.

Perfect for Harry Potter games, VR experiences, or any application requiring instant voice command recognition! 🧙‍♂️✨