Sustained 150 FPS YOLO Inference on a Budget Intel CPU (No GPU) via Nim and OpenVINO Execution Provider

Projects
1

Hello community,

I want to share the architecture and performance numbers of my standalone Windows x64 evaluation build designed for low-overhead, real-time machine vision loops on ordinary desktop CPUs.

The core objective was to completely bypass heavy runtime interpreters (no Python at runtime), eliminate cloud/network dependencies, and achieve a flat memory profile during continuous manufacturing shift rotations.

The Architecture & Optimization Knobs:

  • Runtime Core: Written natively in Nim (compiling directly to C/C++, zero virtual machine overhead, flat memory profile).
  • Ingestion Pipeline: Video capture via Microsoft Media Foundation (NV12) → fast frame conversion using libyuv in a dedicated preview thread → inference thread.
  • Dual-Slot Execution: To prevent constant tensor resizing on every frame, the pipeline uses a two-tensor cascade: a 416 input for full-frame global localization and a 256 input for localized Region of Interest (ROI) tracking.
  • Hardware Routing: Driven by ONNX Runtime paired with the Intel OpenVINO Execution Provider, locked to physical CPU topology via ort_intra_threads: 6.

Sustained Performance (Tested on standard Intel Core i5-11400 CPU, strictly NO GPU):

  • Inference Speed: The 256x256 ROI tracking loop sits at ~6–7 ms per frame, enabling stable execution in the 130–170 FPS class.
  • Memory Footprint: Over a continuous 10-hour stress test (millions of inferences), the heap allocation remained completely flat at ~3 MiB (private working set ~182 MiB) with zero memory leaks detected.

Honest Trade-offs (No corporate polish):

  • Model Swapping: When changing models, the old OpenVINO session is abandoned rather than cleanly released. It causes a minor, temporary leak only at the moment of the switch, which doesn’t affect production lines where models don’t hot-swap mid-stream.
  • Teardown: The application invokes ExitProcess for final cleanup because Windows combined with Nim/NiGui finalizers can conflict on the way out.

The standalone binary package ships with stock, untouched YOLOX-nano (COCO-80) weights for instant validation.

I am looking for [infer] console timing numbers on other CPU architectures, as well as feedback from anyone running this build with a global-shutter USB camera to evaluate bounding box stability under fast motion.

Full README, configuration layout, and binaries are available here: