The application runs in the background and continuously listens for a custom wake word. Once activated, users can interact naturally using voice or text to perform everyday tasks such as:

• Open desktop applications
• Search the web
• Browse websites
• Manage files and folders
• Execute terminal commands
• Automate browser workflows
• Capture screenshots
• Analyze on-screen content
• Control the operating system
• Remember user preferences and previous conversations

Unlike traditional chatbot applications, Nova AI is built around an AI Agent architecture where the language model is responsible only for reasoning and planning while the local application executes actions securely.

Motivation

The goal of this project is to explore how Large Language Models can become intelligent operating system assistants instead of simple chat interfaces.

Rather than asking an AI for information only, users can delegate complete workflows such as:

"Open my frontend project, start the development server, open Chrome, and search for the latest React documentation."

The AI decomposes the request into executable tasks and performs them through specialized local tools.

Core Architecture

The application follows a layered architecture consisting of:

User
    │
    ▼
Wake Word Detection
    │
    ▼
Speech-To-Text
    │
    ▼
Gemini AI
(Task Planning)
    │
    ▼
Agent Planner
    │
    ▼
Tool Executor
    │
    ▼
Windows / Browser / File System

Gemini is never responsible for directly controlling the computer.

Instead, Gemini generates a structured execution plan that is validated before being executed by the local tool system.

This separation significantly improves safety, reliability, and maintainability.

Major Features

Voice Assistant

• Wake word activation
• Continuous background listening
• Speech-to-text
• Text-to-speech responses
• Natural conversation

AI Agent

• Natural language understanding
• Task decomposition
• Multi-step planning
• Tool selection
• Conversation context
• Memory management

Computer Automation

• Open applications
• Switch between windows
• Keyboard automation
• Mouse automation
• Execute terminal commands
• Screenshot capture
• File management

Browser Automation

• Search the web
• Navigate websites
• Fill forms
• Extract webpage content
• Summarize information
• Browser task automation

Vision Intelligence

The assistant can understand the user's desktop by analyzing screenshots using Gemini Vision.

Examples include:

• Find a button on screen
• Read visible text
• Understand application layouts
• Assist with software usage

Memory System

Nova AI maintains several levels of memory:

• Conversation history
• User preferences
• Frequently used projects
• Personalized shortcuts
• Long-term memories
• Tool execution history

This enables more personalized and context-aware interactions over time.

Security Model

A major design goal of the project is preventing unrestricted AI access to the operating system.

Instead of allowing arbitrary execution, every action passes through a permission layer.

Gemini
    │
Generate Plan
    │
Permission Manager
    │
Tool Validation
    │
Execution

Potentially dangerous actions require explicit user confirmation before execution.

Technology Stack

Desktop

• Electron
• React
• TypeScript
• Node.js

UI

• Tailwind CSS
• shadcn/ui
• Framer Motion

Artificial Intelligence

• Gemini API
• Gemini Function Calling
• Gemini Vision

Voice

• Wake Word Detection
• Whisper (Speech-to-Text)
• Text-to-Speech

Automation

• RobotJS
• Playwright
• Windows APIs

Data

• SQLite
• Vector Database
• Local Storage

Learning Objectives

This project focuses on learning and implementing:

• AI Agent architecture
• Tool-based LLM execution
• Desktop software engineering
• Electron application architecture
• Operating system integration
• Browser automation
• Voice interfaces
• Computer vision
• Secure AI execution
• Context and memory management
• Scalable software architecture

Current Status

Planning & System Architecture

The project is currently in the architecture and design phase, including system design, technology selection, security planning, and workflow definition before implementation begins.