Case Study
The Future of Physical AI: Building a High-EQ Companion Robot to Support Elderly Care
Solution
Robotics | AI
Industry
Healthcare
Core Technology
PyTorch, TensorFlow, ROS, OpenCV, NVIDIA Jetson, LangGraph
Overview
Across the world, populations are aging faster than healthcare systems can adapt. This shift has led to significant strain on families, increased caregiver burnout, and a growing prevalence of isolation among older adults navigating the complexities of aging.
Xavor’s physical AI team set out to create a solution that could not only provide practical assistance to caregivers and older adults but also offer companionship. That vision became Navi, a high-EQ companion robot that combines next-gen conversational AI with cognitive robotics to provide not just hands-on support but also ensure emotional well-being.
Execution framework
From the outset, development had to be managed as a single connected effort between:
- Support real-time vehicle trackingEmbedded systems, industrial design, and mobility
- AI/MLConversation systems and computer vision
- Product ArchitectureBackend and DevOps
This integrated approach was essential in turning Navi into a cohesive care solution that could provide both emotional support and practical assistance. However, aligning these functions within a single real-world robotic system posed a unique set of technical and execution challenges.
Challenges
Building a companion robot for elderly care means solving problems that span both software and hardware constraints. AI model development, data scarcity, and complex engineering challenges shape the technical landscape, while creating an experience that feels human introduces an entirely different dimension of complexity. As a first-of-its-kind initiative, there was no existing playbook to follow — every major design decision required original thinking, with no established reference architectures, best practices, or off-the-shelf solutions to draw from.
As development progressed, the team encountered several challenges that needed careful balancing across engineering, design, and human-centered interaction:
1. Edge Computing on a Constrained Platform
Running advanced AI models on NVIDIA Jetson Orin’s ARM64 architecture pushed the limits of available hardware. Standard libraries and tools are often unsupported or require custom adaptation, making development far more complex than conventional environments. Computer vision, conversational AI, voice interaction, and autonomous navigation all had to operate seamlessly on a single edge device, requiring meticulous system design and careful prioritization of resources.
2. Real-Time Performance with Zero Room for Error
Activity recognition and fall detection are needed to operate in real-time with millisecond-level responsiveness. In elderly care, even the slightest delay in detecting a critical event can have serious consequences, leaving no room for compromise in system design or execution.
3. Creating an Emotionally Intelligent Companion
Building a conversational agent that goes beyond scripted responses to understand emotional context, maintain continuity across interactions, and provide genuine companionship demanded a re-evaluation of how AI engages with vulnerable users.
4. Persistent Long-Term Memory
Enabling Navi to remember past interactions, track patient context, and maintain emotional continuity over time turned out to be one of the most technically demanding aspects. Memory preservation had to balance long-term retention with real-time responsiveness to ensure every interaction was meaningful.
5. Data Scarcity and Multi-Class Model Training
Publicly available datasets for patient activity recognition in clinical settings simply do not exist due to privacy and regulatory restrictions. The team needed to design simulated clinical environments and record a proprietary dataset from scratch to train the computer vision models. The activity recognition system had to reliably distinguish a wide range of behaviors, from everyday actions like eating, sitting, and walking to critical events such as falls or signs of agitation. Achieving this depended on meticulous model design, extensive testing, and careful handling of edge cases.
6. Bridging the Gap Between Simulation and Reality
Even with purpose-built living labs designed to mirror real care environments, early user testing surfaced navigation edge cases that controlled settings had not exposed, such as unpredictable movement patterns and environmental variables. Resolving these issues meant testing in real care environments with actual users, followed by updating the system accordingly.
The Solution
Xavor’s physical AI team engineered Navi as a fully integrated edge-AI companion, uniting computer vision, conversational intelligence, and autonomous navigation on a single robotic platform powered by NVIDIA Jetson Orin. Each system is created to deliver performance, responsiveness, and reliability in a constrained hardware environment.
Emotionally Intelligent Conversational Agent
Navi’s conversational engine integrates cloud-based speech recognition, intelligent agent workflows, and a graph-database-backed long-term memory system. This architecture allows Navi to remember past interactions, interpret emotional context, and engage elderly users with warmth, continuity, and a sense of genuine companionship. Wake word detection and speech recognition leverage Azure Speech Services, while natural-sounding voice responses are powered by ElevenLabs. This combination ensures Navi communicates with clarity, warmth, and approachability, creating an interaction experience that is both natural and human-centric.
- Activity Recognition & Fall DetectionThe team developed a multi-class activity recognition system capable of detecting a wide spectrum of daily activities, including eating, drinking, sitting, standing, and walking, alongside transitional movements, agitation states, and falls. The model was built on an action recognition framework and extended with custom-trained classes using proprietary data captured in simulated clinical settings.
- Gait AnalysisA custom gait analysis pipeline was developed from the ground up to provide real-time measurements of clinically relevant mobility metrics, including step length, step width, and cadence. This enables early detection of physical decline or fall risk, empowering proactive intervention.
- Autonomous Navigation & DockingLeveraging robotics middleware, LiDAR-based mapping, and a 3D camera setup, Navi can learn the layout of its environment during initial setup and navigate autonomously while avoiding obstacles. The system includes automated docking, allowing Navi to return to its charging station independently when battery levels run low.
Outcomes
Navi has set a new benchmark in eldercare by transforming safety, caregiver support, and patient well-being. It combines edge-AI, real-time monitoring, and emotionally intelligent interaction to deliver an impact that goes beyond technology, redefining what a companion robot could achieve.
- Real-Time Monitoring & AlertsNavi's safety architecture operates across multiple layers, from embedded hardware responses to continuous sensor-driven monitoring. Physical safety events — such as contact detection or edge recognition — are handled at the hardware controller level, ensuring near instantaneous response, independent of software load. A multi-sensor perception system provides continuous 360° environmental awareness, enabling real-time detection and response to dynamic obstacles, including people. Battery status is monitored continuously, with automated alerts and self-managed recovery ensuring the system remains operational without caregiver intervention. Navi delivers millisecond-level activity recognition, instantly detecting and alerting to critical events such as falls and sedentary behavior. This gives caregivers peace of mind and ensures rapid response when every second counts.
- Reduced Caregiver BurdenNavi reduces manual observation time by ~50%. That means one caregiver can now monitor more than two residents simultaneously through the mobile app as opposed to the previous 1:1 manual observation model. By continuously monitoring patients and automatically generating activity logs, Navi frees caregivers from the strain of constant manual observation, allowing them to focus on higher-value, hands-on care.
- High-EQ CompanionshipNavi maintains a dedicated long-term memory profile for each resident, preserving past interactions, personal preferences, and emotional context so every conversation builds on shared history instead of resetting. As a result, caregivers and residents describe Navi as less of a device and more of a presence.
- Autonomous & Self-Sufficient OperationNavi is built for fully unattended operation within care environments. On first deployment, the user helps Navi to map its surroundings so that it can navigate autonomously against that persistent map for all subsequent operations — adapting in real time to changing environments and moving obstacles. When idle or running low on power, Navi independently returns to its charging station and resumes operation once recharged, completing the full loop without staff involvement. Manual override remains available through the companion mobile app for supervised control when needed.
- Continuous Real-World ValidationFour units of Navi are currently deployed in two controlled customer environments, with structured feedback loops based on 500+ operational hours actively informing system improvements. This phased approach ensures that every iteration is grounded in actual usage, not just simulation, building toward a solution that is proven, not just promising.
tech stack
- Edge Computing:Dual NVIDIA Jetson Orin processors in a distributed on-device architecture — one dedicated to AI inference and perception, the other to real-time robotics control and navigation. All critical processing runs on the device with no dependency on cloud connectivity. Moreover, Navi supports online models as well.
- Robotics and Navigation:ROS-based architecture. Multi-layer LiDAR sensing combined with depth camera perception and IMU-based orientation tracking. Autonomous SLAM-based mapping, dynamic path planning, obstacle avoidance, and self-docking — all running on the dedicated robotics compute node.
- Conversational AI:LangGraph, Azure Speech Services, ElevenLabs, NVIDIA RIVA, Memo0, Neo4j Aura, Custom Hybrid Graph powered by Neo4j Aura.
- Computer Vision:Py-Torch, TensorFlow, OpenCV — running fully on-device on the AI compute node. A dedicated perception camera feeds person detection and activity recognition pipelines, separate from the navigation imaging system to avoid resource contention.
Privacy & Data Handling
Because Navi operates in care environments where continuous monitoring and conversation history are central to its function, privacy and data handling were treated as core engineering requirements rather than afterthoughts. The platform was designed with the following principles in mind:
- On-Device Processing:Vision and perception models run entirely on the device, with all activity recognition, fall detection, and gait analysis processed at the edge. For the conversational pipeline, speech recognition and voice synthesis run in the cloud, while response generation is configurable — caregivers can select between a fully on-device LLM for offline, privacy-sensitive operation or a cloud-hosted model.
- Data Retention:Data access and ownership remain with the research organization. Data retention is fully configurable based on the research organization's requirements. By default, video, audio, and conversation logs are stored locally on-device and automatically expire after seven days. Organizations can also connect their own cloud storage for permanent retention, ensuring data ownership and lifecycle policies remain entirely under their control.
- Consent & Access ControlsResearchers take consent from the participants/volunteers. Consent is obtained through strict IRB-approved protocols, with participants or their legally authorized representatives providing written authorization before data collection begins. Role-based permissions ensure caregivers see only their assigned residents, while all access and configuration events are captured in immutable audit logs aligned with HIPAA accountability requirements.
conclusion
Developing a high-EQ companion robot pushed the boundaries of AI and robotics. Xavor combined real-time activity recognition, intelligent conversation, and autonomous navigation to create a system that operates effectively in real-world care environments. The result is a strong foundation for solutions that can support safer, more responsive, and more dignified care.
Exploring physical AI solutions for your organization?
We specialize in building intelligent, edge-powered systems that bring AI into the physical world.