Private AI on GPU servers you control
Private AI means running AI models on infrastructure you control, usually a GPU server on your premises or in a South African colocation data centre, so sensitive data stays under your governance. Arke Technologies designs, supplies through our Server Hub division, deploys and manages private AI for organisations across South Africa.
What is a GPU server?
A GPU server is a server built to hold one or more graphics processing units (GPUs) together with the power supplies, cooling and PCIe lanes to keep them running at full load. GPUs do the parallel arithmetic that AI models depend on, and a large language model (LLM) runs quickly only when it fits in GPU memory. Memory, not raw speed, is therefore the first sizing question.
As a rule of thumb, an 8-billion-parameter model compressed to 4-bit needs about 8 GB of memory, while a 70-billion-parameter model compressed the same way needs 40 to 48 GB, plus headroom for longer conversations (serverhub.co.za, September 2026). Most enterprise AI purchases land on a two-socket 2U server with up to four GPUs rather than a training cluster (serverhub.co.za, September 2026), and Server Hub's guide to which NVIDIA GPU suits which workload (opens serverhub.co.za) explains the trade-offs between memory types and form factors. Private AI is one part of our AI consulting and implementation service.
Data sovereignty in South Africa: what the law requires
Data sovereignty is the principle that data is governed by the laws of the country where it is collected or held, including who can legally compel access to it. Data residency describes only where the data physically sits. For AI the difference matters: a prompt sent to an overseas model service transfers whatever personal information it contains.
South Africa has no single law requiring all data to stay in the country, but several data sovereignty laws and rules point that way:
- POPIA section 72 permits transfers of personal information outside South Africa only where the recipient is bound by comparable protection, the person consents, or another listed condition applies.
- POPIA section 19 requires appropriate security safeguards wherever the data is processed.
- Sector rules in financial services, health and government often set stricter expectations for where records are held and who may reach them.
- Contracts and tenders may ask suppliers to confirm in-country processing.
Private AI meets these in the simplest way: the model runs where the data already lives. See our POPIA compliance service and our guide to data centres in South Africa.
Choosing the hardware: rackmount GPU servers and workstations
Most private AI runs on a rackmount GPU server, and the choice of platform should follow your GPUs, rack and support model rather than the badge. The main tiers, from Server Hub's guide to GPU-server spec sheets (serverhub.co.za, September 2026):
- 2U. An accelerator server holds up to about four double-width GPUs on air cooling. Dell PowerEdge and HPE ProLiant (the server line formerly sold under the HP name) both build these, and it is the mainstream choice for enterprise inference.
- 4U. Takes up to about ten PCIe GPUs for inference at scale; a 4U Supermicro GPU server is a common example.
- 8U. Carries NVIDIA's 8-GPU SXM baseboard for training models too large for one card.
A 1U GPU server saves rack space but holds fewer cards and leaves the least room for airflow. At the other end, a 16-GPU server is a training-class specification with power and cooling to match; we scope one only when a model genuinely needs it, because few inference workloads do.
We prefer NVIDIA-Certified configurations, in which NVIDIA and the server maker have tested the exact server, GPU and firmware together (NVIDIA-Certified servers explained (opens serverhub.co.za)). A tower case suits a single AI workstation under a desk, and anything a team shares belongs in a rack. For a data-science team that wants a private model on the desk, an AI workstation or an NVIDIA DGX Spark can do the job without a server room (AI workstations for local LLMs (opens serverhub.co.za)).
What a GPU server costs, and why a cheap GPU server rarely is
Last reviewed: September 2026.
An NVIDIA GPU server price is set mostly by the GPUs: their class, count and memory type. High-bandwidth memory on training-class GPUs costs far more per gigabyte than the GDDR memory on cards built for inference and graphics (serverhub.co.za, September 2026). The rest of the bill comes from:
the CPU platform and generation, which sets PCIe bandwidth to the GPUs;- power supplies and cooling, since an eight-GPU training node is a 10 kW-class machine (serverhub.co.za, September 2026);
- UPS and backup power sized for load-shedding;
- networking, storage, and rack space or colocation fees;
- software, support and the engineering to deploy and run it.
A cheap GPU server from eBay or an overseas auction looks attractive until shipping, import duty, VAT, missing parts and the lack of a local warranty are added. Server Hub prices every unit in rand with VAT included, bench-tests it before dispatch, and gives a local warranty of 12 months on new and certified refurbished units and 90 days on pre-owned, always in addition to your Consumer Protection Act rights. Cheap GPU server hosting in the cloud has the opposite trap: a low entry price and an hourly bill that grows with use. For current prices, request a hardware quote (opens serverhub.co.za) or read the DGX Spark price guide (opens serverhub.co.za).
Governance, procurement and continuity
- Compliance. POPIA controls, ISO 27001-aligned security, King IV Principle 12 reporting for boards, and PFMA or MFMA documentation for public bodies.
- Reporting. A monthly report on usage, capacity, incidents and changes; escalation paths written into the SLA.
- Procurement. Level 1 B-BBEE contributor (sworn affidavit available on request); company documents, CSD supplier number and SARS tax compliance status on request; buy by proposal, purchase order or tender response.
- Continuity. You own the hardware, your data, your model configurations and the documentation. Every configuration is documented, and on exit we hand over credentials and runbooks so another team can take over.
- One accountable team. Hardware supply, deployment, security and support sit under one contract with Arke, so a fault never falls between a hardware vendor and a software team.