Understanding Model Precision – The Foundation of AI Infrastructure Design

One thing I’ve noticed whilst studying AI infrastructure is that many engineers immediately jump to talking about GPUs.

“How many H100s do I need?”

“Will four GPUs be enough?”

“Should I use NVLink?”

They’re all good questions.

The problem is they’re being asked too early.

Before we can size GPUs…

Before we can decide whether tensor parallelism is required…

Before we can even begin designing the AI network…

We first need to understand model precision.

In my opinion, this is one of the most overlooked topics in AI infrastructure.

Most discussions immediately jump to GPU servers, networking or storage, yet none of those decisions can be made correctly until we know how the model itself is represented in memory.

Everything starts here.

---

AI Infrastructure Always Starts with the Business Requirement

One thing I quickly realised whilst studying AI infrastructure is that infrastructure isn’t the starting point.

The customer doesn’t ask for H100 GPUs.

The customer doesn’t ask for RoCE.

The customer doesn’t ask for InfiniBand.

Instead, they come with a business requirement.

Perhaps they want an internal AI assistant.

Maybe they want developers to generate code.

Perhaps they want to perform medical research using a frontier open-source language model.

That business requirement dictates the AI workload.

The workload dictates the model.

The model dictates the precision.

The precision dictates the GPU memory.

Only then can we begin selecting GPUs.

I’ve found that simply changing the order in which I think about AI completely changed how I approach infrastructure design.

---

So What Exactly is Model Precision?

Every Large Language Model consists of parameters.

Billions of them.

These parameters are simply numbers that represent everything the model has learnt during training.

Programming.

Languages.

Mathematics.

Medicine.

Networking.

History.

Everything.

The interesting part is that these numbers don’t have to be stored using the same amount of memory.

Exactly the same model can exist in several different numerical formats.

For example:

Notice something?

The model hasn’t changed.

The knowledge hasn’t changed.

Only the amount of memory required to store each parameter has changed.

That one decision completely changes the infrastructure requirements. Modern quantisation techniques reduce memory footprint and improve inference efficiency by storing weights, activations and sometimes KV cache at lower precision while carefully balancing accuracy. 

Why Infrastructure Engineers Should Care

Let’s assume we want to deploy a 70-billion parameter model.

The memory calculation is actually very straightforward.

Model Memory = Parameters × Bytes per Parameter

Let’s calculate it.

FP16

70 Billion × 2 Bytes

≈ 140 GB

FP8

70 Billion × 1 Byte

≈ 70 GB

INT4

70 Billion × 0.5 Bytes

≈ 35 GB

Nothing about the model changed.

Nothing about the AI changed.

Only the precision changed.

Yet we’ve reduced the memory requirement from approximately 140 GB down to around 35 GB.

That’s a huge difference.

Suddenly the infrastructure design looks completely different.

The number of GPUs changes.

The number of GPU servers changes.

Whether tensor parallelism is required changes.

Even the amount of east-west traffic flowing across the AI fabric changes.

All because of one design decision.

But Model Memory Isn’t the Whole Story

One mistake I often see is engineers stopping their calculation there.

Unfortunately, model weights are only one part of GPU memory usage.

During inference, GPU memory is also consumed by:

• KV Cache

• Runtime overhead

• CUDA libraries

• vLLM

• Tensor Parallel communication buffers

• Memory fragmentation

• Future growth

The larger the context window becomes, the larger the KV Cache becomes.

The more users connected simultaneously, the more memory is required.

This is exactly why simply sizing GPUs based on the model weights can be misleading. Decoder-based LLMs also consume memory for the KV cache during inference, and its size grows with context length, making it an important part of capacity planning. 

My Rule of Thumb

When performing initial infrastructure sizing, I generally recommend allowing approximately 30% additional GPU memory beyond the calculated model weights.

Is it always exactly 30%?

No.

Some workloads require less.

Others, particularly those using very large context windows or heavy batching, may require significantly more.

However, I’ve found 30% to be a sensible and conservative starting point when designing production AI infrastructure.

Instead of calculating:

Model Memory

I calculate:

Required GPU Memory = Model Memory × 1.30

It’s a very small change.

But it produces much more realistic infrastructure estimates.

Example Calculations

Immediately you can see why this matters.

A 70B FP8 model isn’t really something I’d size at exactly 70 GB.

I’d be looking closer to 90 GB.

Likewise, a 70B INT4 model isn’t simply a 35 GB deployment.

I’d realistically budget somewhere around 45–50 GB of available GPU memory.

That extra headroom makes deployments far more resilient.

Does Lower Precision Mean a Worse Model?

This is probably one of the biggest misconceptions I hear.

People often assume INT4 automatically means poor quality.

It doesn’t.

Modern quantisation techniques such as AWQ, GPTQ and SmoothQuant have become remarkably good at reducing memory requirements whilst maintaining much of the original model quality.

Would I train a frontier model entirely in INT4?

No.

Would I happily deploy an internal company chatbot using an INT4 model?

Absolutely.

It all depends on the workload.

Which brings us back to the beginning.

The workload always dictates the infrastructure.

The Thought Process I Now Follow

Whenever someone asks me to design AI infrastructure, I mentally walk through exactly the same process.

Business Requirement

↓

AI Workload

↓

Model Selection

↓

Model Precision

↓

GPU Memory Calculation

↓

+30% Operational Overhead

↓

GPU Selection

↓

Tensor Parallelism

↓

GPU Server Count

↓

Network Fabric

↓

Storage

Notice something?

Networking doesn’t come first.

It supports everything that came before it.

That was probably one of the biggest mindset shifts I had whilst studying AI infrastructure.

Final Thoughts

The more I study AI infrastructure, the more I realise that GPUs aren’t really the starting point.

They’re simply the consequence of every design decision that came before them.

If someone tells me they’re deploying a 70-billion parameter model, my first question isn’t:

“How many H100s do you have?”

Instead, I ask:

“What precision are you planning to run the model at?”

Because once I know the answer to that question, I can begin estimating GPU memory, determine whether tensor parallelism is required, understand the likely traffic patterns across the AI fabric, and ultimately design the infrastructure that best supports the business requirement.

Everything else follows from there.