Skip to content
My Docs

AI Hardware recommendations

The optimal hardware for machine learning and AI depends heavily on the specific tasks, with different configurations suited for AI development, machine learning training, and large language models (LLMs).

For AI development and machine learning training, focus on a balanced system with a powerful CPU, ample RAM, fast storage, and a high-performance GPU.

  • Processor (CPU): Intel Xeon W and AMD Threadripper Pro platforms are favored for their reliability, PCIe lane support, and memory performance. These platforms offer the stability and bandwidth needed for demanding AI workloads. A minimum of 16 cores is a good starting point, with at least 4 cores allocated per GPU to ensure smooth operation.
  • Video Card (GPU): NVIDIA GPUs are the industry standard for their performance and extensive support. While almost any NVIDIA GPU can be used, newer, high-end models generally deliver better performance. Consumer cards like the NVIDIA GeForce RTX 4080 and 4090 offer strong performance, but “Professional” GPUs like the RTX 5000 Ada and 6000 Ada are better suited for multi-GPU setups due to their cooling and size. The RTX 6000 Ada, with its 48GB VRAM, is particularly well-suited for handling data with “large feature size”, such as high-resolution images.
  • Memory (RAM): A general rule of thumb is to have at least twice as much CPU memory as the total GPU memory in the system. For example, a system with two GeForce RTX 4090 GPUs (48GB total VRAM) should ideally have 128GB of system RAM. Adequate RAM is crucial to prevent bottlenecks when handling large datasets, which are common in machine learning tasks.
  • Storage (Drives): Fast NVMe SSDs are recommended for optimal performance, especially for staging job runs, as data streaming speeds can become a bottleneck. SATA SSDs are suitable for larger datasets that exceed NVMe capacity, while platter drives are generally reserved for archival storage. Consider RAID arrays for additional storage space and network-attached storage for backup and data sharing.

Large language models (LLMs) demand even more specialized hardware configurations due to their massive computational requirements.

  • CPU: Server-grade platforms like Intel Xeon or AMD EPYC are essential for LLM servers due to their PCIe lanes, high memory bandwidth and capacity, and ECC memory support. While the exact CPU model may not be the primary factor, platforms with these features are crucial for handling the intense workloads associated with LLMs.
  • GPU: “Professional” or “Compute” level GPUs, like NVIDIA’s RTX 6000 Ada, L40S, and H100 or AMD’s MII Instinct GPUs, are recommended for LLMs. These GPUs provide the large amounts of VRAM needed to handle the massive parameters of LLMs. A 70B parameter model near its native precision, for instance, may require approximately 200GB of VRAM.
  • Multiple GPUs: LLM servers often employ multiple GPUs to handle the demanding computational load, with 4 to 8 GPUs being a common configuration. Distributing the workload across multiple GPUs can significantly improve performance.
  • System RAM: NVIDIA recommends at least twice the amount of CPU system memory as the total GPU VRAM for LLM servers. This configuration helps to ensure efficient buffering and data transfer between the CPU and GPUs, which is crucial for smooth operation.
  • Storage: High-capacity NVMe SSDs are the preferred choice for LLM servers due to their speed and ability to handle large datasets. LLM parameters and datasets often occupy large amounts of storage, and fast storage is essential for efficient data access and processing.

For day-to-day use where AI features enhance productivity, AI PCs are a good option. They prioritize energy efficiency while providing enough power for tasks like real-time transcription and intelligent camera enhancements. Think of them as focusing on AI consumption rather than development. AI-ready mobile workstations bridge the gap between AI PCs and dedicated AI development setups. These workstations are designed for professionals who need the power to develop and work with complex AI models, including AI model training, data analysis, and intensive AI content creation.

No matter the specific AI task, software and network optimization also play a critical role.

  • Leveraging a scalable operating system like Linux, using AI-optimized libraries and frameworks such as TensorFlow or PyTorch, and employing middleware like Kubernetes for container orchestration can significantly improve efficiency and performance.
  • Efficient data management, including high-performance storage systems, tiered storage architectures, data preprocessing, and optimized data input/output operations, is also crucial for minimizing latency and maximizing performance.
  • Finally, a robust network infrastructure with high-speed networks, advanced networking technologies like InfiniBand, and redundancy and failover capabilities is critical, especially for distributed AI environments.