ODAI (ODAI AI Assistant) is a comprehensive AI assistant platform built on FastAPI that orchestrates multiple specialized agents to handle diverse user requests. The system uses OpenAI's Agent framework with a hub-and-spoke architecture where a central orchestrator routes requests to specialized agents.
- Main Application (
api.py
): FastAPI application with WebSocket support
- Orchestrator (
connectors/orchestrator.py
): Central agent that routes requests
- Voice Orchestrator (
connectors/voice_orchestrator.py
): Specialized for voice interactions
- Individual Agents: Specialized tools for specific services and APIs
- Services Layer: Authentication, chat management, location services
- Firebase Integration: User management, chat history, token tracking
User Request → WebSocket → Orchestrator → Specialized Agent(s) → Response → User
The orchestrator uses the H.A.N.D.O.F.F. decision framework:
- Has capability: Does the agent solve this task?
- Access: Does it have the right data/API permissions?
- Novelty/Need: Is a tool call necessary vs. known info?
- Delay/Cost: Prefer fewer/cheaper calls if quality unaffected
- Output quality: Will it return the needed format/info?
- Failure fallback: Choose alternates if first likely fails
- Fusion: Orchestrate multiple agents and merge results
- GMail Agent: Email management (send, receive, search, reply)
- Google Calendar Agent: Event scheduling and calendar management
- Google Docs Agent: Document creation, search, and collaboration
- Slack Agent: Team communication integration
- Twilio Assistant: Voice and SMS capabilities
- Google Search Agent: Web search functionality
- Google News Agent: News headlines and stories
- Fetch Website Agent: Website content extraction
- Google Shopping Agent: Product search and price comparison
- Plaid Agent: Bank account integration (balances, transactions)
- Plaid Connector Agent: Account connection setup
- FinnHub Agent: Stock market data and financial information
- CoinMarketCap Agent: Cryptocurrency prices and market data
- Exchange Rate Agent: Currency conversion
- FlightAware Agent: Real-time flight status and tracking
- Flights Agent: Flight booking and information
- AMADEUS Agent: Flight and hotel search
- Amtrak Agent: Train status and schedules
- Caltrain Agent: Regional train information
- Yelp Agent: Restaurant and business search with reviews
- TripAdvisor Agent: Travel recommendations and reviews
- Ticketmaster Agent: Event tickets and venue information
- MovieGlu Agent: Movie showtimes and theater information
- AccuWeather Agent: Detailed weather forecasts
- WeatherAPI Agent: Current weather and forecasts
- Location Service: IP-based location detection
- Amazon Agent: Product search and details
- Google Shopping Agent: Cross-retailer product comparison
- EasyPost Agent: Package tracking across carriers
- Open External URL Agent: URL opening in browser
- Google Connections Agent: Google account OAuth setup
Capabilities:
- Fetch and search inbox messages
- Send emails with attachments
- Reply to existing threads
- Search emails by sender or content
- HTML and plain text message processing
Key Functions:
fetch_google_email_inbox()
: Retrieve inbox messagessearch_google_mail()
: Search emails by querysend_google_email()
: Send new emailsreply_to_google_email()
: Reply to existing threads
Authentication: Google OAuth2 with Gmail scope
Capabilities:
- Connect to bank and credit card accounts
- Retrieve account balances
- Fetch transaction history
- Support multiple financial institutions
Key Functions:
get_accounts_at_plaid()
: Get account balances and detailsget_transactions_at_plaid()
: Retrieve transaction history
Environment: Sandbox (development) and Production environments
Capabilities:
- Perform web searches using SerpAPI
- Return organic search results with metadata
- Extract rich snippets and sitelinks
Key Functions:
search_google()
: Execute search queries and return results
Integration: SerpAPI for Google search results
WeatherAPI Agent (
connectors/weatherapi.py
):
- Current weather conditions
- Multi-day forecasts (1-14 days)
- Hourly predictions
- Supports various location formats
AccuWeather Agent (
connectors/accuweather.py
):
- Real-time weather by coordinates
- Detailed hourly and daily forecasts
- Weather alerts and conditions
FlightAware Agent (
connectors/flightaware.py
):
- Real-time flight status
- Departure/arrival information
- Gate and terminal details
Flight Agent (
connectors/flights.py
):
- IATA flight code lookups
- Airport information
- Booking capabilities (inactive in current version)
The central ODAI orchestrator uses gpt-4o and implements the O.D.A.R. loop:
- Observe: Parse user input and context
- Decide: Choose appropriate agent(s)
- Act: Execute tool calls (parallel when safe)
- Respond: Synthesize results for user
Handoff Agents (35+ available):
ORCHESTRATOR_AGENT = Agent(
name="ODAI",
model="gpt-4o",
handoffs=[
YELP_AGENT, COINMARKETCAP_AGENT, GMAIL_AGENT,
PLAID_AGENT, GOOGLE_CALENDAR_AGENT, GOOGLE_DOCS_AGENT,
# ... all 35+ agents
]
)
Tool Call Tracking:
The system tracks 100+ tool calls with user-friendly progress messages:
TOOL_CALLS = {
"search_businesses_at_yelp": "Searching Yelp...",
"get_stock_price_at_finnhub": "Getting Stock Price...",
"send_google_email": "Sending Email...",
# ... 100+ tool mappings
}
Specialized for real-time voice interactions using
RealtimeAgent
:
- Optimized for conversational speech
- Reduced tool set for faster responses
- Voice-specific prompting and response formatting
- Integration with Twilio for phone calls
- Local Development: Uses
.env
files
- Production: Google Secret Manager integration
- API Keys: 25+ service integrations managed
- Environment Detection: Automatic local vs. production switching
class Settings(BaseSettings):
openai_api_key: str
production: bool
# 25+ API keys for various services
plaid_client_id: str
google_client_id: str
serpapi_api_key: str
# ... additional service keys
- Models (
firebase/models/
): User, Chat, Tokens, Usage tracking
- Authentication: Token-based user management
- Chat History: Persistent conversation storage
- Analytics: Segment integration for usage tracking
WSS /chats/{chat_id}?token={auth_token}
Message Format:
{
"message": "User prompt text",
"thread_id": "unique_thread_identifier"
}
GET /
: Health check and static file servingPOST /waitlist
: Email collectionPOST /google_access_request
: OAuth initiationGET /update_integrations
: Refresh agent configurations- Development-only endpoints for token reset
Each agent follows a consistent pattern:
@function_tool(is_enabled=enable_check_function)
def agent_function(wrapper: RunContextWrapper[ChatContext], ...params) -> dict:
"""Tool description and usage instructions."""
# API call logic
return ToolResponse(
response_type="agent_specific_type",
agent_name="Service Name",
friendly_name="Human Readable Name",
display_response=True,
response=processed_data
).to_dict()
AGENT = Agent(
name="Agent Name",
instructions=PROMPT_PREFIX + specific_instructions,
handoffs=[related_agents],
tools=[agent_function, other_tools]
)
- Full feature set available
- Detailed responses with formatting
- Support for complex multi-step operations
- Can handle long-form content
- Optimized for speech synthesis
- Concise, conversational responses
- Limited tool set for faster execution
- Special formatting for voice output
- Integration with phone system via Twilio
backend/
├── api.py # Main FastAPI application
├── connectors/ # All agent implementations
│ ├── orchestrator.py # Central orchestrator
│ ├── voice_orchestrator.py # Voice-specific orchestrator
│ ├── gmail.py # Email agent
│ ├── plaid_agent.py # Financial services
│ └── [35+ other agents]
├── services/ # Core business logic
├── websocket/ # Real-time communication
├── firebase/ # Database models
├── routers/ # API route handlers
└── tests/ # Comprehensive test suite
Agents are automatically registered through:
- Import in orchestrator files
- Addition to handoffs list
- Integration configuration in
integrations.yaml
- Tool call mapping in
TOOL_CALLS
dictionary
- Unit Tests: Individual agent functionality
- Integration Tests: Full agent workflows
- E2E Tests: Complete user journeys
- Firebase Tests: Database operations
- WebSocket Tests: Real-time communication
Each agent has a standardized configuration:
- id: AgentID
name: "Human Readable Name"
description: "Detailed capability description"
logo: "https://logo-url"
prompts:
- "Example usage prompt 1"
- "Example usage prompt 2"
This configuration drives:
- Frontend integration display
- User onboarding prompts
- Agent capability documentation
- Marketing and user education materials
- Firebase Authentication: User identity management
- OAuth2 Integration: Google, Plaid, and other service auth
- API Key Management: Secure secret storage
- Token Validation: All requests authenticated
- Rate Limiting: Built into individual agents
- Environment Isolation: Separate dev/prod configurations
- User data encrypted in transit and at rest
- Service tokens stored securely in Firebase
- Agent responses filtered for sensitive information
- Audit logging for all user interactions
- Segment Integration: User behavior analytics
- Token Usage: OpenAI API consumption tracking
- Agent Performance: Success rates and latencies
- Error Monitoring: Sentry integration for error tracking
- Agent usage frequency
- Tool call success rates
- User engagement patterns
- Error rates and types
- Performance benchmarks
ODAI represents a sophisticated multi-agent AI system capable of handling diverse user needs through specialized, interconnected agents. The architecture prioritizes modularity, scalability, and user experience while maintaining security and performance standards. The system continues to evolve with new agent integrations and enhanced capabilities.
For development questions or agent integration requests, refer to the individual agent files and test suites for implementation details.