PCloud

What I Built

PCloud is a personal hosting and automated benchmarking pipeline for open-source LLMs. It manages the server lifecycle, runs evaluation harnesses, and monitors hardware constraints on a dedicated home server.

Why It Matters

Deploying local models requires verifying their accuracy and safety. General benchmarks often hide performance drops on specific workloads (like code generation or instruction following). PCloud automates evaluation runs on local hardware to verify models before utilizing them in active coding workflows.

Pipeline Architecture

• Remote Orchestration: An orchestrator script (evaluate_model.py) runs on my workstation, connecting over SSH to a remote headless machine (ko8e) to boot llama-server instances.
• Hardware constraints: Applies a 220W power limit via nvidia-smi to prevent thermal throttling on the RTX 2080 Ti. It loads GGUF models with Flash Attention and 8-bit quantized KV caching to fit parameters within VRAM limits.
• Benchmark Suite: Runs models against a 5-task suite (ifeval, mbpp, humaneval, arc_challenge, mmlu_global_facts), outputting results to local JSON metadata and compiling leaderboard.md.
• Hybrid API routing: To prevent parsing errors on long generations, evaluations split traffic: multiple-choice scoring runs via /v1/completions, while code generation and instruction-following tasks route via /v1/chat/completions.