The evolution of the modern companion robot is marked by an increasing demand for multifunctionality, aiming to provide rich, interactive, and adaptive experiences. Typically, functions such as natural voice interaction, immersive visual feedback, and autonomous movement are developed on disparate software platforms—Android for its robust application ecosystem and Linux for low-level hardware control and real-time processing. This segregation presents a significant integration challenge: harmonizing these distinct system environments into a cohesive, cost-effective, and efficient unitary companion robot. This project addresses this core challenge by proposing and implementing a novel integration methodology centered on the Raspberry Pi 3 platform. We demonstrate a system where a multi-OS approach, facilitated by strategic inter-process communication, successfully unifies conversational AI, holographic projection, and specified-following capabilities into a single, functional companion robot.
1. Architectural Overview and Hardware Platform Rationale
The foundational premise of our companion robot design is the strategic use of not one, but two Raspberry Pi 3 Model B units. This decision stems from the need to isolate different runtime environments while maintaining tight functional coupling. The Raspberry Pi 3 was selected as the core hardware platform due to its unique combination of accessibility, connectivity, and sufficient computational power for our target applications.
The key hardware specifications that make the Raspberry Pi 3 ideal for this multifunctional companion robot are summarized below:
| Feature | Specification | Relevance to Companion Robot |
|---|---|---|
| Processor | Broadcom BCM2837 64-bit Quad-Core CPU | Provides adequate power for concurrent Android UI tasks and Linux-based computer vision. |
| Memory | 1GB LPDDR2 RAM | Sufficient for running lightweight Android and Linux distributions simultaneously across two units. |
| Connectivity | Built-in 802.11n WiFi & Bluetooth 4.1 | Enables cloud-based speech processing, local network communication between Pis, and peripheral connectivity. |
| I/O & Expansion | 4x USB 2.0, 40-pin GPIO header, HDMI, CSI | Allows connection of microphone, speakers, camera module, motor drivers, and display for holography. |
| Storage | MicroSD card slot | Enables separate bootable OS images for Android and Linux, defining the role of each unit. |
The system architecture bifurcates responsibilities between two physically separate but communicatively linked Raspberry Pi units, which we designate as Pi-A and Pi-B. This segregation is crucial for the performance and integration of the companion robot.
The functional allocation is governed by the following high-level design rule:
$$ \text{System Role}(\text{Pi}) = \begin{cases} \text{Android OS} & \text{if primary function involves high-level UI, cloud API, or media} \\ \text{Linux OS} & \text{if primary function involves real-time sensing, control, or low-level hardware access} \end{cases} $$
Based on this rule, the final hardware and software mapping for our companion robot is established:
| Unit | Primary OS | Core Functions | Key Peripherals |
|---|---|---|---|
| Raspberry Pi-A (Pi-A) | Android 7.0 (LineageOS) | 1. Voice Interaction (IFLYTEK Cloud) 2. Holographic Projection Playback |
USB Microphone, HDMI Display, USB Speaker, WiFi Dongle |
| Raspberry Pi-B (Pi-B) | Raspbian Linux | 1. Specified-Following via Face Tracking | Risym CSI Camera Module, GPIO-connected Motor Driver, DC Motors/Wheels |
2. Integration of High-Level Functions on Android (Pi-A)
The Pi-A unit serves as the user-facing “personality” core of the companion robot. Its Android environment hosts the two functions that require a rich graphical user interface (GUI), internet connectivity for cloud services, and stable media playback capabilities.
2.1. Intelligent Voice Interaction System
The voice interaction module transforms the companion robot from a passive device into an interactive partner. The process is a multi-stage pipeline involving local capture, cloud processing, and local synthesis.
The software architecture on Pi-A was implemented using Android Studio, with a custom APK managing the entire workflow. The core interaction loop can be modeled as a sequential process \( P_v \):
$$ P_v: S_{\text{audio}} \xrightarrow{\text{ASR}} T_{\text{query}} \xrightarrow{\text{NLU}} I_{\text{semantic}} \xrightarrow{\text{Dialog Mgmt}} R_{\text{cloud}} \xrightarrow{\text{TTS}} S_{\text{response}} $$
where:
- \( S_{\text{audio}} \): Raw audio signal from microphone.
- \( \text{ASR} \): Automatic Speech Recognition (IFLYTEK Cloud).
- \( T_{\text{query}} \): Textual transcription of user query.
- \( \text{NLU} \): Natural Language Understanding (IFLYTEK Cloud).
- \( I_{\text{semantic}} \): Structured intent and semantic slots.
- \( \text{Dialog Mgmt} \): Cloud-based dialog manager generating textual response \( R_{\text{cloud}} \).
- \( \text{TTS} \): Text-To-Speech synthesis (IFLYTEK Cloud).
- \( S_{\text{response}} \): Audio response played through speaker.
A critical integration feature is the local keyword detection filter \( F_k \). Before sending \( T_{\text{query}} \) to the cloud, the application scans it for specific command keywords (e.g., “follow me”, “stop”). If a keyword \( k \) is detected, the process forks:
$$ \text{If } k \in T_{\text{query}} \text{ then } C_{\text{command}}(k) \xrightarrow{\text{Socket}} \text{Pi-B} $$
This mechanism allows the high-level Android-based companion robot interface to issue direct commands to the low-level control system on Pi-B, enabling true cross-platform functionality.
2.2. Holographic Projection Module
The holographic projection adds a compelling visual dimension to the companion robot, creating the illusion of a 3D character. This is achieved not with true holography but with a “Pepper’s Ghost” illusion using a semi-transparent holographic film.
The optical setup on Pi-A is geometrically constrained. The HDMI display (light source) is positioned horizontally above the projection film. The film is mounted at a 45° angle relative to the user’s line of sight. For a clear image, the relationship between the display height \( h_d \), the film height \( h_f \), and the user’s viewing distance \( d_v \) must be considered. A simplified geometric model ensures the reflected virtual image \( I_v \) appears at the desired location for the user interacting with the companion robot.
The software implementation leverages Android’s `VideoView` component. The custom APK loads and seamlessly loops pre-rendered animations (created using tools like MikuMikuDance) in the background of the main interface. The video playback logic is encapsulated in the function \( \text{PlayHologram}(V) \), where \( V \) is the video resource, ensuring continuous and stable projection that is core to the companion robot’s visual presence.
3. Integration of Real-Time Control on Linux (Pi-B)
The Pi-B unit acts as the “autonomous body” of the companion robot. Its Linux environment, specifically Raspbian, is chosen for its low overhead, direct hardware access via libraries like `WiringPi`, and excellent support for real-time computer vision with OpenCV.
3.1. Specified-Following via Visual Face Tracking
This function enables the companion robot to actively orient itself towards the user, a fundamental behavior for engagement. It is a classic feedback control problem where the camera image is the sensor and the motorized base is the actuator.
The control pipeline on Pi-B can be described as follows:
- Image Acquisition & Preprocessing: The CSI camera captures frames \( I_t \) at time \( t \). Frames are converted to grayscale and normalized.
- Face Detection: We utilize OpenCV’s Haar Cascade classifiers, a machine learning-based approach. The detection function is:
$$ \text{Faces}_t = \text{CascadeClassifier.detectMultiScale}(I_t, \text{scaleFactor}=1.1, \text{minNeighbors}=5) $$
We combine frontal (\( C_{\text{front}} \)) and profile (\( C_{\text{profile}} \)) classifiers for robustness: \( \text{CascadeClassifier} = C_{\text{front}} \cup C_{\text{profile}} \). - Error Calculation: For the largest detected face region with center pixel coordinates \( (x_f, y_f) \) and frame center \( (x_c, y_c) \), the horizontal error \( e_t \) is:
$$ e_t = x_f – x_c $$
A dead zone \( \delta \) is introduced to prevent chatter: \( \text{if } |e_t| \leq \delta \text{ then } e_t = 0 \). - Proportional Control: A simple but effective P-controller generates a motor command signal \( u_t \):
$$ u_t = K_p \cdot e_t $$
where \( K_p \) is the proportional gain determining the turning speed of the companion robot. - Motor Actuation: The sign of \( u_t \) determines direction (clockwise/counter-clockwise), and its magnitude is mapped to a Pulse-Width Modulation (PWM) duty cycle for the motor driver. The `WiringPi` library handles the GPIO output.
The system forms a closed-loop: \( I_t \rightarrow e_t \rightarrow u_t \rightarrow \text{Motor Movement} \rightarrow \text{New Camera Position} \rightarrow I_{t+1} \). This loop allows the companion robot to continuously minimize \( e_t \), thereby keeping the user centered in its field of view.
4. System-Wide Integration via Socket Communication
The true integration of the multifunctional companion robot is achieved by establishing a reliable communication channel between the Android (Pi-A) and Linux (Pi-B) subsystems. We implement a TCP Socket connection over the local WiFi network, creating a client-server model.
Let \( A \) represent the Android application process on Pi-A and \( L \) represent the Linux control process on Pi-B. The communication protocol is defined as follows:
$$ \text{Server } L: \text{Listen}(IP_{\text{pi-b}}, Port_s) $$
$$ \text{Client } A: \text{Connect}(IP_{\text{pi-b}}, Port_s) $$
Once connected, the companion robot’s high-level brain (A) can send simple string commands \( C_{str} \) to the low-level body (L). The command set is minimal:
$$ C_{str} \in \{ \text{“FOLLOW_ON”}, \text{“FOLLOW_OFF”}, \text{“MOVE_FORWARD”}, \text{“STOP”} \} $$
Upon receiving a command, process \( L \) parses it and changes the state of its internal control loop. For example, receiving “FOLLOW_OFF” sets an internal flag \( \text{tracking\_active} = \text{false} \), causing the motor command \( u_t \) to be zero regardless of the visual error \( e_t \). This decouples the systems when needed while maintaining the capability for unified action, making the companion robot’s behavior cohesive.
The overall integration architecture of the complete companion robot system is summarized in the following table:
| Layer | Component (Pi-A) | Component (Pi-B) | Integration Mechanism |
|---|---|---|---|
| Hardware | RPi 3, Display, Mic, Speaker | RPi 3, Camera, Motors, Driver | Separate physical units, shared power supply. |
| Operating System | Android 7.0 | Raspbian Linux | Dual-boot SD cards; isolated runtime environments. |
| Core Function | Speech I/O, Hologram UI | Face Tracking, Motor Control | Functional specialization per OS strengths. |
| Communication | Socket Client (WiFi) | Socket Server (WiFi) | TCP/IP over local network; string-based protocol. |
| User Command Flow | Voice → Keyword → Command String | Command String → Control Flag → Action | Seamless pipeline from natural speech to physical movement. |
5. Implementation Results and Performance Analysis
The proposed integration method was validated by building a functional prototype of the companion robot. Each subsystem and their integration were tested rigorously.
Pi-A (Android System) Results: The Android application successfully provided a stable interface. Voice interaction latency was primarily governed by cloud processing (IFLYTEK), with typical end-to-end response times between 2-3 seconds, which is acceptable for a conversational companion robot. The holographic projection playback was smooth and continuous, with the VideoView component reliably looping the animation without resource conflict with the voice processing threads. The local keyword filter accurately detected pre-defined commands and triggered Socket messages.
Pi-B (Linux System) Results: The face tracking system operated at approximately 8-12 frames per second (FPS) on the Raspberry Pi 3, sufficient for smooth following behavior of the companion robot. The P-controller with a dead zone effectively eliminated motor jitter when the user was nearly centered. The tracking performance could be summarized by the following measurable outcomes:
| Metric | Condition | Result | Implication for Companion Robot |
|---|---|---|---|
| Tracking Accuracy | Well-lit environment, frontal face | >95% detection rate | Reliable primary orientation behavior. |
| Tracking Range | Horizontal field of view | ~±30 degrees from center | Defines the effective engagement zone. |
| System Latency | Error detection to motor response | < 200ms | Responsive and natural following motion. |
| Power Consumption | Both Pis, motors, display active | ~10W (5V/2A) | Feasible for portable battery operation. |
Cross-System Integration Results: The Socket communication link proved highly reliable on the local network. Commands sent from the Android app were received and executed by the Linux system with negligible latency (<50ms). This enabled seamless high-level control of the companion robot’s movement through natural voice, fulfilling the core integration objective. For instance, the spoken phrase “Hey robot, follow me” would result in the keyword “follow” being detected, the string “FOLLOW_ON” being sent via Socket, and the Pi-B immediately activating its face-tracking control loop, causing the entire assembly to turn and follow the user.
6. Discussion and Conclusion
This project has demonstrated a practical and effective methodology for integrating diverse functionalities into a unified companion robot platform. The choice of using two Raspberry Pi 3 units, each running an operating system tailored to its functional cluster (Android for high-level interaction, Linux for real-time control), proved successful in circumventing the typical conflicts of driver and library compatibility. This separation of concerns aligns with good system engineering principles.
The integration glue—a simple TCP Socket communication protocol—was lightweight yet powerful enough to orchestrate complex behaviors like voice-initiated following. The cost-effectiveness of this approach is significant, as the entire companion robot brain is built upon inexpensive, widely available single-board computers.
However, limitations exist. The dual-computer architecture increases physical complexity and power requirements. Future iterations of this companion robot could explore more tightly integrated solutions using a single more powerful board (like a Raspberry Pi 4 or NVIDIA Jetson Nano) capable of running containerized environments (e.g., Docker) or a Type-1 hypervisor to manage both Android and Linux kernels on the same hardware, reducing footprint and potential communication latency further.
Furthermore, the functionalities themselves can be deepened. The vision system could move from simple face tracking to user identification and gesture recognition, allowing the companion robot to differentiate between users and respond to non-verbal cues. The holographic output could evolve from pre-rendered animations to a real-time rendered avatar whose expressions and lip movements are synchronized with the TTS output, vastly enhancing the sense of presence.
In conclusion, the presented work validates that a multifunctional companion robot integrating cloud-based AI, local media, and autonomous mobility is feasible using mainstream maker-level hardware and thoughtful software architecture. The key lies not in forcing all functions into a single monolithic software stack, but in strategically partitioning them according to the demands of their underlying operating systems and using robust inter-process communication to synthesize their capabilities into a single, coherent companion robot experience. This paradigm offers a scalable blueprint for future development in accessible, multi-modal robotic companions.