Private LLM Assessment System

Outcomes

Context

When the answers are confidential, the model must live where the data policy allows.

Challenge

Assessment workflows needed language-model assistance without sending sensitive organizational answers to public LLM APIs. Off-the-shelf models were generic; the product needed domain-tuned behavior and a deployable inference path.

Approach

We built Python training pipelines (Excel/data prep → transformers/LoRA fine-tunes), exported GGUF artifacts, packaged Ollama-compatible models, and exposed an Express/Ollama inference API for applications. CUDA Docker supported training iterations.

Architecture

PyTorch/transformers training jobs, llama.cpp/GGUF export, Ollama runtime, Node/Express inference API, CUDA-enabled Docker for fine-tuning.

Scope

• —Dataset preparation
• —LoRA fine-tuning
• —Model export
• —Local/private serving
• —Product inference API

Constraints

• —No dependency on public LLM vendors for inference
• —GPU availability for training cycles
• —Need for operationally simple serving after training

Solution highlights

• —LoRA for efficient domain adaptation
• —GGUF/Ollama for practical private deployment
• —Thin API between model ops and product UX

Technologies

Results

• —Repeatable fine-tune → export → serve loop
• —Private inference suitable for sensitive assessment content
• —API boundary so products do not embed training complexity
• —Documented checkpoints for iterative model improvement

Lessons learned

• —Fine-tuning without an export/serve path is research, not a product.
• —Private LLMs still need evaluation — privacy does not equal quality.
• —Keep training drama out of the request path.