Contact Us

Private AI Server for Business: Best Setups (2026)

Nov 17, 20268 min read
Origins AI title card in light blue with a network pattern: Private AI Server for Business: Best Setups (2026)

Last updated: 16 November 2026

Quick Answer: Dell PowerEdge R7725 suits team chat, Lenovo ThinkSystem SR675 V3 company-wide inference, and Origins AI (originshq.com) Chat AI a private assistant on your own servers. HPE Compute XD690 suits training and NVIDIA DGX Spark prototyping. A private AI server for business is a GPU server or desk-side machine running open models and document search on hardware you control.

A private AI server keeps prompts, documents and model weights on hardware you control, so your team gets a ChatGPT-style assistant without sending anything to a hosted API. The setups below are compared on three business filters: team size, security review and uptime.

Which on-premises hardware setups work for a private AI server?

Dell's PowerEdge R7725 is a representative on prem AI server for a business. It is a 2U machine with dual AMD EPYC 9355 32-core processors, 768 GB of DDR5 memory, and two NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. Each GPU carries 96 GB of GDDR7 ECC memory with 1,597 GB/s of bandwidth.

In Dell's published benchmark, which used 2,000 requests per test with an average input of 2,048 tokens and an average output of 256 tokens, that configuration ran NVIDIA's Nemotron 3 Nano 30B (A3B), a mixture-of-experts model, at 3,641.4 tokens per second at FP8, using tensor parallelism across both GPUs. A single GPU served Llama 3.1 8B at 2,741.4 tokens per second, also at FP8.

Dell positions the R7725 for small and midsize businesses that start with one node and scale horizontally by adding more. Larger teams that need an on premise AI server for training or heavy concurrent inference move to GPU-dense machines.

Lenovo's ThinkSystem SR675 V3 is a 3U on premise AI server that takes up to eight double-wide or single-wide GPUs, including NVIDIA H200 and L40S. It also offers an NVIDIA HGX H200 4-GPU configuration with NVLink and Lenovo Neptune hybrid liquid-to-air cooling.

Lenovo describes its SR675i V3 as a specific configuration of the SR675 V3 designed for AI inferencing. At the top end, HPE's Compute XD690 carries eight NVIDIA Blackwell Ultra B300 HGX GPUs in an air-cooled chassis, for large model training, tuning and inferencing.

Capabilities as documented by each vendor on 16 November 2026:

Setup GPUs Best fit Main limit
Dell PowerEdge R7725 (2U rack) 2x RTX PRO 6000 (96 GB each) Team-wide chat and agent inference, 30B-class models Two GPUs cap the model size per node
Lenovo ThinkSystem SR675 V3 (3U rack) Up to 8, incl. H200 and L40S Company-wide inference Needs liquid-to-air cooling in the HGX H200 configuration
Origins AI Chat AI (deployed software) Runs on your servers or your own AWS, Azure or GCP VPC A private assistant on your own servers or cloud account, with an implementation team Not hardware: you still choose the server
HPE Compute XD690 (air-cooled rack) 8x Blackwell Ultra B300 HGX Large model training and tuning Sized for training rather than chat
NVIDIA DGX Spark (desktop) GB10 Superchip, up to 128 GB unified memory Developer prototyping, desk-side agents Desktop unit, not a rack server

What software stack does a self-hosted LLM and private RAG need?

Hardware alone does not give you a working assistant; the software stack decides who can use it and what it can read. A typical open-source stack has three layers: an inference engine, a chat front end and a retrieval layer.

Which inference engine and front end does a private LLM server use?

The model runs on an inference engine such as vLLM or Ollama, and users reach it through a chat front end such as Open WebUI. vLLM is a common choice for throughput-oriented serving, and Ollama's own site says: "Your prompts are never stored or trained on."

The vLLM documentation on Open WebUI describes it as a self-hosted platform designed to operate entirely offline, supporting runners like Ollama and OpenAI-compatible APIs, with built-in RAG. The retrieval layer is the private RAG pipeline that chunks your documents, embeds them and grounds answers in them.

AnythingLLM bundles the front end and retrieval in one app. Its docs describe an "all-in-one AI application that can do RAG, AI Agents", shipped as a desktop app for macOS, Windows and Linux or as a Docker build for single- or multi-user installs on your own server. It connects to Ollama as a separate runner, and both versions are open source.

If you are weighing the specific engines, our vLLM vs Ollama comparison breaks down where each fits. For the model itself, see our guide to the open source models that make sense to self host on your GPU budget.

What does a security team need before it approves a self-hosted LLM stack?

Three additions on top of the open-source stack: single sign-on with role-based access, audit logs and a model gateway.

Deploying and hardening this stack is a project in itself: identity, logging, model routing and patching all have to be in place before staff use it.

When is a desk-side AI workstation such as NVIDIA DGX Spark enough?

Not every private AI deployment needs a rack. NVIDIA's DGX Spark is a desktop machine built on the GB10 Grace Blackwell Superchip, with up to 20 CPU cores, up to 128 GB of unified memory and up to 1 petaFLOP of AI performance at FP4 precision, in NVIDIA's own wording. NVIDIA says it can fine-tune models up to 70 billion parameters and run inference on models up to 200 billion parameters on the 128 GB version.

Its ConnectX networking lets up to four units cluster for larger workloads, reaching up to 700 billion parameters across 512 GB of combined memory. NVIDIA's Local AI software stack runs on the device, which the company describes as keeping the work local.

NVIDIA positions DGX Spark for AI developer, researcher and data scientist workloads, and for always-on agent workloads run from a desktop. For final tuning or deployment, NVIDIA's guidance is to evaluate migrating the work to NVIDIA DGX Cloud or other NVIDIA-accelerated data centers. Choose DGX Spark when a few engineers need to prototype on real data; choose a rack server when the assistant has to serve a department.

How do you choose a setup by team size, security review and uptime?

Filter in this order: team size, security review, then uptime.

Which setup fits your team size?

A handful of engineers can work on a desk-side box; a department-sized chat deployment fits a dual-GPU rack server like the R7725, which Dell pitches at multiuser SMB environments and tested at a maximum batch size of 256; size it against your own concurrency target. Company-wide or training workloads point to eight-GPU machines.

Should you deploy on premises or in your own cloud account?

Deploy on premises when processing has to stay inside your network; a server in your own cloud account gives isolation without owning hardware. If your security team has blocked cloud AI tools, this deployment mode is the decision that matters. Our comparison of on-premise versus private cloud versus air-gapped AI walks through that choice.

If your security team wants a structured review, NIST's AI Risk Management Framework is a voluntary framework, released January 26, 2023, for incorporating trustworthiness considerations into AI systems. Its July 26, 2024 Generative AI Profile (NIST-AI-600-1) helps organizations identify risks unique to generative AI. The companion AI RMF Playbook organizes suggested actions under four functions: Govern, Map, Measure and Manage.

How do you keep a private assistant running in production?

Use a rack server with dual power supplies, like the R7725's two 3,200 W units, plus a spare-hardware plan; a single desktop box does not give you that.

Whichever setup you pick, check five things before you commit:

  1. Concurrency: how many simultaneous users, at what response-time target.
  2. Model size: which model class your use case needs, and the GPU memory to hold it.
  3. Deployment mode: on-premise or private cloud, agreed with security.
  4. Access controls: SSO, RBAC and audit logging wired in before launch, not after.
  5. Operations plan: who patches, monitors and redeploys the stack, and on what schedule.

What mistakes should you avoid when buying AI server hardware?

The most common mistake is buying GPUs before the model class is settled. The second is treating the stack as an afterthought: an unmonitored server with no SSO fails the same security review a cloud tool did. For what each tier costs, from a single workstation card to an 8-GPU server, see our AI server cost guide.

How Origins AI fits on your own server

Some teams want the outcome, a private ChatGPT-style assistant staff can actually use, without assembling the stack themselves. Origins AI deploys Origins AI Chat AI on your servers or inside your own AWS, Azure or GCP VPC. With self-hosted models in an on-premise deployment, no data leaves your network; a request sent to a hosted model provider falls under that provider's own data-handling policies.

Access runs through SSO via SAML 2.0 or OIDC with role-based access control, and every message, model call and retrieval event is logged. You can run OpenAI, Anthropic, Meta Llama, Mistral or your own fine-tuned model, with retrieval over your internal documents in a private knowledge base.

An implementation team handles the rollout, from integration and data ingestion to support after go-live. For the hardware sizing and deployment work, see our on-premise AI deployment services.

Talk to an engineer

Ready to keep your AI in-house? Book a call with an Origins AI engineer to talk through hardware, stack and deployment mode for your team.

Frequently Asked Questions

Can you host your own AI server?
Yes. A single rack server with two 96 GB GPUs, or a desktop machine like DGX Spark, runs open models entirely on hardware you control. The harder work is the stack: inference engine, chat front end, retrieval, SSO and audit logging. You also own GPU driver, CUDA and model updates on your own schedule.
Is there a totally private AI?
Only when the model runs on hardware you control and nothing is sent to a hosted API. NVIDIA describes its Local AI stack on DGX Spark as keeping the work on the machine. Plan for one trade-off: an air-gapped install blocks automatic model updates, so new weights and firmware need staged, scanned delivery.
Is a private cloud AI server still private?
Partly. A private cloud AI server runs in your own cloud account, so no other tenant shares the instance, but the traffic runs outside your network and the provider's region rules apply. It suits teams that want isolation without buying hardware, as long as security accepts data leaving the building.
How long does it take to deploy a private AI server?
Hardware lead times aside, the software stack sets the schedule: inference serving, retrieval, SSO and audit logging, then a security review. A site with no outside network connection takes longer, because model weights and updates need staged, scanned delivery. Origins AI reports that a Chat AI pilot, with SSO integration and initial knowledge ingestion, is typically live within two weeks.
Who maintains an on-prem AI server after launch?
Your team or a partner has to own GPU drivers, CUDA, model updates, monitoring and capacity planning, on a schedule security agrees to. If Origins AI runs the deployment, it handles updates, integration changes and ongoing support after go-live, inside the environment you control. Either way, name an owner before launch.
What happens when you outgrow one server?
Dell says R7725 deployments scale horizontally by adding nodes without re-architecting. Within a single node, Lenovo's SR675 V3 supports NVLink GPU-to-GPU connections of up to 900 GB/s, so one chassis can grow before you add a second. Plan network, rack space and cooling for the second node at purchase time.
Book a call

About the Author

Apoorva Kumar is Co-Founder and CEO of Origins AI (originshq.com), an AI engineering partner for product teams building AI workflows, AI agents and LLM integrations. A CSE graduate of IIT Kharagpur, Apoorva previously built and scaled technology at Sony, NuCash, YesMadam and FrontPage.