Cactus: An Open-Source Low-Latency AI Engine for Mobile Devices and Wearables

Summary

Cactus is an open-source, low-latency AI engine designed specifically for mobile devices and wearables. It features OpenAI-compatible APIs across major languages, supporting chat, vision, speech-to-text, RAG, tool calling, and cloud handoff. The engine is built on a zero-copy computation graph (PyTorch for mobile) with custom models optimized for RAM and quantization, and includes ARM SIMD kernels optimized for Apple, Snapdragon, Exynos, and other mobile processors. It offers custom attention mechanisms, KV-cache quantization, and chunked prefill for efficient on-device AI inference.