Last updated: 16 November 2026
Quick Answer: Dell PowerEdge R7725 suits team chat, Lenovo ThinkSystem SR675 V3 company-wide inference, and Origins AI (originshq.com) Chat AI a private assistant on your own servers. HPE Compute XD690 suits training and NVIDIA DGX Spark prototyping. A private AI server for business is a GPU server or desk-side machine running open models and document search on hardware you control.
A private AI server keeps prompts, documents and model weights on hardware you control, so your team gets a ChatGPT-style assistant without sending anything to a hosted API. The setups below are compared on three business filters: team size, security review and uptime.
Which on-premises hardware setups work for a private AI server?
Dell's PowerEdge R7725 is a representative on prem AI server for a business. It is a 2U machine with dual AMD EPYC 9355 32-core processors, 768 GB of DDR5 memory, and two NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. Each GPU carries 96 GB of GDDR7 ECC memory with 1,597 GB/s of bandwidth.
In Dell's published benchmark, which used 2,000 requests per test with an average input of 2,048 tokens and an average output of 256 tokens, that configuration ran NVIDIA's Nemotron 3 Nano 30B (A3B), a mixture-of-experts model, at 3,641.4 tokens per second at FP8, using tensor parallelism across both GPUs. A single GPU served Llama 3.1 8B at 2,741.4 tokens per second, also at FP8.
Dell positions the R7725 for small and midsize businesses that start with one node and scale horizontally by adding more. Larger teams that need an on premise AI server for training or heavy concurrent inference move to GPU-dense machines.
Lenovo's ThinkSystem SR675 V3 is a 3U on premise AI server that takes up to eight double-wide or single-wide GPUs, including NVIDIA H200 and L40S. It also offers an NVIDIA HGX H200 4-GPU configuration with NVLink and Lenovo Neptune hybrid liquid-to-air cooling.
Lenovo describes its SR675i V3 as a specific configuration of the SR675 V3 designed for AI inferencing. At the top end, HPE's Compute XD690 carries eight NVIDIA Blackwell Ultra B300 HGX GPUs in an air-cooled chassis, for large model training, tuning and inferencing.
Capabilities as documented by each vendor on 16 November 2026:
| Setup | GPUs | Best fit | Main limit |
|---|---|---|---|
| Dell PowerEdge R7725 (2U rack) | 2x RTX PRO 6000 (96 GB each) | Team-wide chat and agent inference, 30B-class models | Two GPUs cap the model size per node |
| Lenovo ThinkSystem SR675 V3 (3U rack) | Up to 8, incl. H200 and L40S | Company-wide inference | Needs liquid-to-air cooling in the HGX H200 configuration |
| Origins AI Chat AI (deployed software) | Runs on your servers or your own AWS, Azure or GCP VPC | A private assistant on your own servers or cloud account, with an implementation team | Not hardware: you still choose the server |
| HPE Compute XD690 (air-cooled rack) | 8x Blackwell Ultra B300 HGX | Large model training and tuning | Sized for training rather than chat |
| NVIDIA DGX Spark (desktop) | GB10 Superchip, up to 128 GB unified memory | Developer prototyping, desk-side agents | Desktop unit, not a rack server |
What software stack does a self-hosted LLM and private RAG need?
Hardware alone does not give you a working assistant; the software stack decides who can use it and what it can read. A typical open-source stack has three layers: an inference engine, a chat front end and a retrieval layer.
Which inference engine and front end does a private LLM server use?
The model runs on an inference engine such as vLLM or Ollama, and users reach it through a chat front end such as Open WebUI. vLLM is a common choice for throughput-oriented serving, and Ollama's own site says: "Your prompts are never stored or trained on."
The vLLM documentation on Open WebUI describes it as a self-hosted platform designed to operate entirely offline, supporting runners like Ollama and OpenAI-compatible APIs, with built-in RAG. The retrieval layer is the private RAG pipeline that chunks your documents, embeds them and grounds answers in them.
AnythingLLM bundles the front end and retrieval in one app. Its docs describe an "all-in-one AI application that can do RAG, AI Agents", shipped as a desktop app for macOS, Windows and Linux or as a Docker build for single- or multi-user installs on your own server. It connects to Ollama as a separate runner, and both versions are open source.
If you are weighing the specific engines, our vLLM vs Ollama comparison breaks down where each fits. For the model itself, see our guide to the open source models that make sense to self host on your GPU budget.
What does a security team need before it approves a self-hosted LLM stack?
Three additions on top of the open-source stack: single sign-on with role-based access, audit logs and a model gateway.
- Single sign-on and role-based access, so every user authenticates against your existing identity provider.
- Audit logs, so you can show who asked what, and when.
- A model gateway, so you can swap models without rewriting the application, and route sensitive traffic only to self-hosted backends.
Deploying and hardening this stack is a project in itself: identity, logging, model routing and patching all have to be in place before staff use it.
When is a desk-side AI workstation such as NVIDIA DGX Spark enough?
Not every private AI deployment needs a rack. NVIDIA's DGX Spark is a desktop machine built on the GB10 Grace Blackwell Superchip, with up to 20 CPU cores, up to 128 GB of unified memory and up to 1 petaFLOP of AI performance at FP4 precision, in NVIDIA's own wording. NVIDIA says it can fine-tune models up to 70 billion parameters and run inference on models up to 200 billion parameters on the 128 GB version.
Its ConnectX networking lets up to four units cluster for larger workloads, reaching up to 700 billion parameters across 512 GB of combined memory. NVIDIA's Local AI software stack runs on the device, which the company describes as keeping the work local.
NVIDIA positions DGX Spark for AI developer, researcher and data scientist workloads, and for always-on agent workloads run from a desktop. For final tuning or deployment, NVIDIA's guidance is to evaluate migrating the work to NVIDIA DGX Cloud or other NVIDIA-accelerated data centers. Choose DGX Spark when a few engineers need to prototype on real data; choose a rack server when the assistant has to serve a department.
How do you choose a setup by team size, security review and uptime?
Filter in this order: team size, security review, then uptime.
Which setup fits your team size?
A handful of engineers can work on a desk-side box; a department-sized chat deployment fits a dual-GPU rack server like the R7725, which Dell pitches at multiuser SMB environments and tested at a maximum batch size of 256; size it against your own concurrency target. Company-wide or training workloads point to eight-GPU machines.
Should you deploy on premises or in your own cloud account?
Deploy on premises when processing has to stay inside your network; a server in your own cloud account gives isolation without owning hardware. If your security team has blocked cloud AI tools, this deployment mode is the decision that matters. Our comparison of on-premise versus private cloud versus air-gapped AI walks through that choice.
If your security team wants a structured review, NIST's AI Risk Management Framework is a voluntary framework, released January 26, 2023, for incorporating trustworthiness considerations into AI systems. Its July 26, 2024 Generative AI Profile (NIST-AI-600-1) helps organizations identify risks unique to generative AI. The companion AI RMF Playbook organizes suggested actions under four functions: Govern, Map, Measure and Manage.
How do you keep a private assistant running in production?
Use a rack server with dual power supplies, like the R7725's two 3,200 W units, plus a spare-hardware plan; a single desktop box does not give you that.
Whichever setup you pick, check five things before you commit:
- Concurrency: how many simultaneous users, at what response-time target.
- Model size: which model class your use case needs, and the GPU memory to hold it.
- Deployment mode: on-premise or private cloud, agreed with security.
- Access controls: SSO, RBAC and audit logging wired in before launch, not after.
- Operations plan: who patches, monitors and redeploys the stack, and on what schedule.
What mistakes should you avoid when buying AI server hardware?
The most common mistake is buying GPUs before the model class is settled. The second is treating the stack as an afterthought: an unmonitored server with no SSO fails the same security review a cloud tool did. For what each tier costs, from a single workstation card to an 8-GPU server, see our AI server cost guide.
How Origins AI fits on your own server
Some teams want the outcome, a private ChatGPT-style assistant staff can actually use, without assembling the stack themselves. Origins AI deploys Origins AI Chat AI on your servers or inside your own AWS, Azure or GCP VPC. With self-hosted models in an on-premise deployment, no data leaves your network; a request sent to a hosted model provider falls under that provider's own data-handling policies.
Access runs through SSO via SAML 2.0 or OIDC with role-based access control, and every message, model call and retrieval event is logged. You can run OpenAI, Anthropic, Meta Llama, Mistral or your own fine-tuned model, with retrieval over your internal documents in a private knowledge base.
An implementation team handles the rollout, from integration and data ingestion to support after go-live. For the hardware sizing and deployment work, see our on-premise AI deployment services.
Talk to an engineer
Ready to keep your AI in-house? Book a call with an Origins AI engineer to talk through hardware, stack and deployment mode for your team.


