A voice-activated Discord bot that listens in voice channels, responds to its wake word, and talks back using AI-generated speech. Built for friend group servers — it joins your call, listens for its name, understands what you said, thinks of a response, and speaks it out loud. Supports up to 10 simultaneous speakers.
Discord Voice Channel
│
▼
Opus Audio (per user)
│
▼
AudioSink ──► Resample to 16kHz mono PCM
│
▼
Energy VAD ──► Detect speech boundaries
│
▼
Deepgram Nova-3 (streaming STT)
│
▼
TranscriptBuffer ──► Multi-speaker context
│
▼
Wake Word Detection ("Hey Smartbot")
│
▼
Gemini 2.5 Flash ──► AI response + tool use
│
▼
ElevenLabs TTS ──► Spoken audio
│
▼
Discord Voice (interruptible playback)
The bot captures each user's audio independently, runs voice activity detection to find speech boundaries, streams audio to Deepgram for real-time transcription, buffers multi-speaker context, and triggers the AI brain when someone says the wake word. Responses stream sentence-by-sentence through TTS for low-latency voice output.
- Wake word activation — say the bot's name to get its attention
- Multi-speaker awareness — tracks who said what across up to 10 users
- Conversation memory — maintains context within a session and remembers facts about users across sessions
- Personality modes — toggle Hype Man or Debate Mode to change behavior
- Interruptible — say the wake word while it's talking to cut it off
| Tool | What it does |
|---|---|
| Web Search | Current info via Tavily |
| Weather | Conditions and forecast for any city |
| Game Lookup | Builds, guides, tier lists for LoL, CS2, Rust, Monster Hunter |
| Sports Scores | Live scores, results, and standings |
| Music Control | Queue songs through Flavibot or other music bots |
| Calculator | Math expressions and unit conversions |
| Timers | Set voice-announced reminders |
| Soundboard | Play sounds from the server's Discord soundboard |
| Quotes | Save and recall memorable moments |
| Playlists | Create, manage, and play saved playlists |
| Birthdays | Track birthdays and get upcoming reminders |
| Horoscopes | Zodiac readings based on saved birthdays |
| Morning Briefing | Weather + top headlines summary |
| Leaderboard | Server stats — most talkative, games played, trivia wins |
- Trivia — multiple categories, multi-round with scoring
- Would You Rather — vote on dilemmas
- Most Likely To — vote on who fits the prompt
- Hot Takes — rate controversial statements 1-10
- Rock Paper Scissors — play against the bot or challenge a friend
- Polls — custom voice-based polls
- Story Mode — collaborative storytelling with genre selection
- Docker and Docker Compose
- API keys for: Discord, Google Gemini, ElevenLabs, Deepgram, Tavily, OpenWeatherMap
git clone https://github.com/samkeller-dev/smartbot.git
cd smartbot
cp .env.example .envEdit .env and fill in your API keys and server IDs.
Create a bot at discord.com/developers with these intents enabled:
- Message Content
- Server Members
- Voice States
Invite the bot to your server with permissions for:
- Connect, Speak, Use Voice Activity (voice)
- Send Messages, Read Message History (text)
docker compose up --buildThe bot will connect to Discord, sync slash commands to your configured guilds, and join voice when invited via the /smartbot slash command.
All configuration is done through environment variables. See .env.example for the full list.
| Variable | Required | Description |
|---|---|---|
DISCORD_BOT_TOKEN |
Yes | Bot token from Discord Developer Portal |
DISCORD_GUILD_IDS |
Yes | Comma-separated guild IDs for slash command sync |
DISCORD_TEXT_CHANNELS |
No | guild_id:channel_id pairs for text output |
GOOGLE_API_KEY |
Yes | Google AI API key for Gemini |
GEMINI_MODEL |
No | Model name (default: gemini-2.5-flash) |
ELEVENLABS_API_KEY |
Yes | ElevenLabs API key |
ELEVENLABS_VOICE_ID |
Yes | Voice ID from ElevenLabs voice library |
DEEPGRAM_API_KEY |
Yes | Deepgram API key (primary STT) |
TAVILY_API_KEY |
Yes | Tavily API key for web search |
OPENWEATHERMAP_API_KEY |
Yes | OpenWeatherMap API key |
WAKE_WORD |
No | Wake word to activate the bot (default: smartbot) |
STT_PROVIDER |
No | deepgram (default) or whisper (local fallback) |
VAD_THRESHOLD |
No | Voice activity sensitivity, 0.0-1.0 (default: 0.5) |
DEFAULT_CITY |
No | Default city for weather lookups (default: Atlanta) |
bot/
├── main.py # Entry point, bot setup, cog loading
├── config.py # Environment variable loader
├── patches.py # Runtime patches for dependencies
├── ai/
│ ├── brain.py # Gemini conversation loop with tool dispatch
│ ├── prompts.py # System prompt builder
│ └── tools.py # Tool definitions (Gemini function calling)
├── audio/
│ ├── sink.py # Discord audio capture + resampling
│ ├── vad.py # Energy-based voice activity detection
│ ├── transcriber.py # STT dispatcher (Deepgram / Whisper)
│ ├── deepgram_stream.py # Deepgram WebSocket streaming client
│ ├── transcript_buffer.py # Multi-speaker transcript aggregation
│ ├── wake_detector.py # Wake word detection in transcripts
│ ├── player.py # Interruptible audio playback
│ └── gaming_vocab.py # Custom vocabulary for game terms
├── cogs/
│ ├── voice_cog.py # Voice channel lifecycle + response pipeline
│ ├── fun_cog.py # Fun slash commands
│ ├── admin_cog.py # Admin slash commands
│ └── soundboard_cog.py # Soundboard integration
├── db/
│ ├── database.py # SQLite connection management
│ └── models.py # User profiles, quotes, facts
├── games/
│ ├── manager.py # Game session state management
│ ├── trivia.py # Trivia game
│ ├── would_you_rather.py # Would You Rather game
│ ├── most_likely.py # Most Likely To game
│ ├── hot_takes.py # Hot Takes rating game
│ ├── rps.py # Rock Paper Scissors
│ ├── polls.py # Voice polls
│ ├── story_mode.py # Collaborative storytelling
│ └── wyr_questions.py # Would You Rather question bank
├── tools/
│ ├── web_search.py # Tavily web search
│ ├── weather.py # OpenWeatherMap integration
│ ├── discord_bot_caller.py # Send commands to other bots
│ ├── quote_logger.py # Save/recall quotes
│ ├── timer_manager.py # Voice-announced timers
│ ├── user_memory.py # User facts and preferences
│ ├── calculator.py # Math and unit conversions
│ ├── gaming.py # Game info lookups
│ ├── sports.py # Sports scores and standings
│ ├── soundboard.py # Discord soundboard control
│ ├── birthday_tracker.py # Birthday storage and reminders
│ ├── greetings.py # Custom join greetings
│ ├── horoscope.py # Zodiac/horoscope lookups
│ ├── stats_tracker.py # Usage stats and leaderboards
│ ├── briefing.py # Morning briefing aggregator
│ ├── jokes.py # Dad jokes
│ ├── random_fun.py # Dice, coins, Magic 8-Ball
│ ├── modes.py # Personality mode toggles
│ └── playlist.py # Playlist management
├── tts/
│ ├── elevenlabs_client.py # ElevenLabs TTS synthesis
│ └── cartesia_client.py # Cartesia TTS (alternative)
└── utils/
├── audio_utils.py # PCM resampling utilities
├── logging_config.py # Log formatter setup
└── text_channel.py # Text channel message helpers
- Create the handler in
bot/tools/your_tool.py - Add the tool definition to
TOOL_DEFINITIONSinbot/ai/tools.py - Register the import in
bot/ai/brain.py_ensure_tools_loaded() - Reference it in the system prompt in
bot/ai/prompts.py
docker compose up --build # Build and start everything
docker compose logs -f bot # Stream bot logs
docker compose restart bot # Restart bot container
docker compose down # Stop everything- Runtime: Python 3.12, async-first
- Discord: discord.py 2.7.1 + discord-ext-voice-recv
- LLM: Google Gemini 2.5 Flash (streaming, function calling)
- STT: Deepgram Nova-3 (streaming WebSocket) with faster-whisper fallback
- TTS: ElevenLabs (eleven_flash_v2_5)
- VAD: Energy-based (RMS thresholding, pure numpy)
- Search: Tavily
- State: Redis (session mutex, timers, transcript buffer)
- Storage: SQLite via aiosqlite (user profiles, quotes, stats)
- Deployment: Docker Compose (bot + Redis)
Private project. All rights reserved.