Over the last few years, the general approach towards building more powerful Large Language Models appeared relatively straightforward.
Increase the number of parameters.
Increase the amount of training data.
Add more GPUs.
Build a larger GPU cluster.
Consume more power.
And eventually…
You end up with a more capable model.
At least, that was the general assumption.
However, models such as DeepSeek and Kimi have demonstrated that increasing intelligence isn’t simply about throwing more GPUs at the problem.
It is also about using the available compute more intelligently.
DeepSeek-V3 contains approximately 671 billion parameters.
Kimi K2 contains approximately one trillion parameters.
Those are enormous models.
Yet DeepSeek-V3 activates only around 37 billion parameters per token, while Kimi K2 activates approximately 32 billion parameters per token. (arXiv)
Think about that for a moment.
A model can contain one trillion parameters…
But only use around 32 billion of them to process a particular token.
How is that possible?
The answer lies in understanding the difference between a Dense model and a Mixture of Experts model.
Let’s have a look.
First, What Is a Parameter?
Before comparing the two architectures, it is important to understand what we actually mean by a parameter.
A parameter is essentially a numerical value learned during the model’s training process.
These values influence how the model interprets language, identifies patterns and predicts the next token.
A model containing 70 billion parameters therefore contains approximately 70 billion learned numerical values.
These values must be stored somewhere.
During inference, they are normally stored within GPU memory.
This is why the number of parameters has such a major influence on the amount of GPU VRAM required to load a model.
As a simple example:
A 70-billion-parameter model running at FP16 or BF16 requires approximately:
70 billion × 2 bytes = 140 GB
That is only the model-weight memory.
We would still need additional capacity for the KV cache, temporary activations, inference framework, CUDA context and general operational overhead.
However, knowing how much memory is required to store the model is only one part of the story.
We must also understand how many of those parameters actually perform computation.
This is where Dense and MoE models begin to behave very differently.
What Is a Dense Model?
A Dense model activates the complete model for every token.
Imagine a company employing 100 people.
A customer phones the company with a networking question.
Instead of transferring the call to the networking team…
All 100 employees enter the meeting room.
Finance attends.
Human Resources attends.
Legal attends.
Marketing attends.
Everyone participates in answering the question.
Now imagine another customer phones with a payroll question.
Once again…
All 100 employees enter the room.
That would obviously be incredibly inefficient.
However, this is essentially how a Dense model works.
Every token passes through the complete set of model layers.
All the model’s parameters participate in the forward pass.
Nothing is selectively switched off.
If we have a 70-billion-parameter Dense model, approximately all 70 billion parameters contribute towards processing every token.
If we have a 405-billion-parameter Dense model, the complete 405-billion-parameter model participates.
Meta’s Llama 3.1 405B, for example, is described as a Dense Transformer containing 405 billion parameters. (Meta AI)
Other familiar Dense-model examples include:
Llama 3.1 8B
Llama 3.1 70B
Llama 3.1 405B
Mistral 7B
Gemma dense variants
Dense models are relatively easy to understand from an infrastructure perspective.
The complete model must be stored.
The complete model is also used.
This makes the relationship between model capacity and active computation fairly direct.
The Problem With Continually Scaling Dense Models
Let’s imagine we wanted to create a Dense model containing one trillion parameters.
At FP16, the model weights alone would require approximately:
1 trillion × 2 bytes = 2 TB of GPU memory
At FP8:
1 trillion × 1 byte = approximately 1 TB
Even after distributing the model across a large GPU cluster, every token would still need to pass through all one trillion parameters.
That means an enormous amount of matrix multiplication.
An enormous amount of memory traffic.
An enormous amount of power.
And potentially an enormous inference cost.
As the model becomes larger, the amount of active computation also grows.
This is one of the major limitations of scaling Dense models.
Researchers therefore needed a way of increasing model capacity without increasing active computation at exactly the same rate.
This is where Mixture of Experts comes in.
What Is a Mixture of Experts Model?
Let’s return to our company analogy.
This time, the company still has 100 employees.
However, instead of involving everyone in every customer request, the company introduces a receptionist.
A customer phones with a networking problem.
The receptionist analyses the request and sends it to the networking team.
A customer phones with a legal question.
The receptionist transfers the call to legal.
A customer asks about payroll.
The finance team deals with it.
The organisation still benefits from the combined knowledge and specialist capability of all 100 employees.
However, only the relevant employees perform work for each request.
This is essentially how a Mixture of Experts model operates.
An MoE model contains many specialist neural-network components known as experts.
A routing mechanism examines each token and determines which experts should process it.
Only the selected experts become active.
The remaining experts stay available…
But they do not perform the main computation for that token.
The Router
The router is one of the most important components within an MoE architecture.
Every token reaches the router.
The router calculates which experts are likely to be most useful for processing that token.
It may select the top two experts.
The top four.
The top eight.
The exact number depends on the model architecture.
The token is then sent to those selected experts.
This means different tokens can be handled by different parts of the model.
A programming-related token may be routed towards experts that have become particularly effective at processing code.
A mathematical token may be routed elsewhere.
A language-related token may use another combination.
It is worth being precise here.
The experts are not normally given simple human-readable labels such as “networking expert” or “finance expert.”
The specialisation emerges during training.
Nevertheless, the company analogy provides a useful way of understanding the underlying logic.
The model contains a large pool of capacity.
The router decides which part of that capacity should become active.
DeepSeek-V3
DeepSeek-V3 is one of the clearest examples.
The model contains approximately:
671 billion total parameters
However, only approximately:
37 billion parameters become active for each token
DeepSeek combines its MoE architecture with other efficiency techniques, including Multi-head Latent Attention, an auxiliary-loss-free load-balancing strategy and multi-token prediction. Its technical report states that the model was pre-trained on 14.8 trillion tokens before supervised fine-tuning and reinforcement-learning stages. (arXiv)
This means DeepSeek-V3 has access to the overall capacity of a 671-billion-parameter model.
But it does not perform computation across all 671 billion parameters for every token.
Only the selected path through the model becomes active.
Kimi K2
Kimi K2 takes this even further.
The model contains:
One trillion total parameters
Yet it activates approximately:
32 billion parameters per token
Kimi K2 was trained using the MuonClip optimiser and pre-trained on approximately 15.5 trillion tokens. Its architecture was designed particularly around knowledge, reasoning, coding and agentic capabilities. (GitHub)
Once again, the important distinction is between:
Total parameters
And:
Active parameters
The total number tells us about the model’s overall capacity and storage requirement.
The active number tells us more about the amount of model computation performed for each token.
Those are not the same thing.
Other Examples of MoE Models
DeepSeek and Kimi are not the only models using Mixture of Experts.
Other examples include:
Mixtral 8x7B
Mixtral 8x22B
DeepSeek-V2
DeepSeek-V3
DeepSeek-R1
Kimi K2
Kimi K2.5
Qwen MoE variants
This shows that MoE is not simply a temporary experiment.
It has become one of the most important architectures for scaling modern Large Language Models.
Does MoE Require Fewer GPUs?
This is where the subject becomes particularly important for AI infrastructure engineers.
It is tempting to look at DeepSeek-V3 and say:
“If only 37 billion parameters are active, we only need enough GPU memory for 37 billion parameters.”
That would be incorrect.
The complete 671-billion-parameter model still needs to be stored somewhere.
Likewise, Kimi K2’s complete one trillion parameters must still reside within GPU memory, host memory or another tier of the storage hierarchy.
Unless expert offloading is used, the entire model normally needs to be distributed across the GPU cluster.
Therefore, MoE does not necessarily reduce the number of GPUs required to fit the model.
In fact, a one-trillion-parameter MoE model may require more GPUs than a much smaller Dense model simply because there are more total weights to store.
This is the distinction we need to remember:
Total parameters influence memory capacity.
Active parameters influence computation per token.
A GPU provides both memory capacity and compute capability.
Those are two different resources.
Model Storage Is Not the Same as Model Computation
Let’s compare two imaginary models.
Model A: Dense
Total parameters:
400 billion
Active parameters:
400 billion
Every token causes the entire model to perform computation.
Model B: MoE
Total parameters:
One trillion
Active parameters:
32 billion
Model B requires significantly more memory because it must store one trillion parameters.
However, the main expert computation performed for each token may be far smaller because only 32 billion parameters become active.
This produces what initially appears to be a contradiction.
The MoE model can require more GPUs for memory…
While requiring less arithmetic work per token.
That is the real value of MoE.
It separates model capacity from active computation.
Does This Mean the Remaining GPUs Are Idle?
Not necessarily.
This is another area where we must be careful.
It would be easy to assume that if only a small number of experts are active, most GPUs are sitting completely idle.
That isn’t always what happens.
In a production environment, the inference platform may process many user requests simultaneously.
One token may be routed to one set of experts.
Another token may be routed to a different set.
Thousands of tokens may be moving through the model at the same time.
Batching and concurrency can therefore keep many experts and GPUs busy.
The objective isn’t to ensure that every expert participates in every token.
The objective is to distribute useful work efficiently across the cluster.
This is also why expert load balancing is so important.
If the router continuously selects the same small group of experts, those experts become overloaded while others remain underused.
A well-designed MoE architecture attempts to distribute tokens across the available experts while still selecting those most appropriate for the workload.
So Where Is the Compute Saving?
The compute saving comes from avoiding unnecessary matrix multiplication.
In a Dense model, every token passes through every Dense feed-forward block.
In an MoE model, only the selected expert blocks perform their main calculations for that token.
Fewer active parameters generally mean fewer floating-point operations per token than an equivalently sized Dense model.
That can lead to:
Higher inference throughput.
Lower computation per token.
Better compute efficiency relative to total model capacity.
Potentially lower inference cost per token.
Potentially lower energy consumed per generated token.
However, none of those outcomes should be treated as automatic.
MoE introduces its own overheads.
Tokens need to be routed.
Experts may be distributed across different GPUs or GPU servers.
Selected tokens may need to travel across the back-end network.
Expert imbalance can create bottlenecks.
Communication can become expensive.
The model may also be limited by memory bandwidth rather than pure arithmetic throughput.
Therefore, MoE is not “free performance.”
It exchanges some dense computation for routing complexity, communication and a more complicated distributed-inference design.
Power and Energy Are Not the Same Thing
This is another distinction that I believe infrastructure engineers need to understand.
Power tells us the rate at which electricity is being consumed at a particular moment.
It is measured in watts or kilowatts.
Energy tells us how much electricity has been consumed over a period of time.
It is measured in joules or watt-hours.
For example, a GPU cluster may draw 20 kilowatts while running inference.
However, that figure alone doesn’t tell us how efficiently it is producing tokens.
We also need to know the throughput.
Imagine the cluster consumes:
20,000 watts
And produces:
2,000 tokens per second
Because one watt is one joule per second:
20,000 joules per second ÷ 2,000 tokens per second
Equals:
10 joules per generated token
That is what we mean by energy per token.
It tells us how much electrical energy the infrastructure consumes to produce one token.
Does MoE Always Consume Less Power?
No.
An MoE cluster may still contain a very large number of GPUs.
Those GPUs consume power even when they are not performing peak tensor computation.
GPU memory consumes power.
The interconnect consumes power.
Network adapters consume power.
CPUs consume power.
System memory consumes power.
Fans and cooling systems consume power.
Simply storing an inactive expert in HBM does not reduce that GPU’s electricity consumption to zero.
Therefore, it would be incorrect to say:
“MoE models consume very little power because most experts are inactive.”
The more accurate statement is:
MoE can reduce the amount of active model computation required per token relative to an equivalently sized Dense model.
That reduction can improve energy efficiency.
But the total power draw depends on the complete deployment.
Energy per Token Is the Better Measurement
Let’s compare two deployments.
Dense Deployment
Average system power:
16 kW
Generation throughput:
800 tokens per second
Energy per token:
16,000 ÷ 800 = 20 joules
MoE Deployment
Average system power:
22 kW
Generation throughput:
2,200 tokens per second
Energy per token:
22,000 ÷ 2,200 = 10 joules
The MoE deployment is actually drawing more power at that moment.
Twenty-two kilowatts rather than sixteen.
However, it is producing far more tokens.
As a result, it consumes less energy for each generated token.
This is why simply comparing GPU power ratings or total cluster power isn’t enough.
A larger cluster can draw more power while still being more energy-efficient per unit of useful work.
For an inference platform, useful work may be measured as:
Tokens per second.
Requests per second.
Completed prompts.
Or joules per generated token.
GPU Utilisation Can Also Be Misleading
High GPU utilisation doesn’t automatically mean the infrastructure is efficient.
A GPU can report high utilisation while performing work that doesn’t produce proportionally high application throughput.
Likewise, relatively low utilisation may indicate poor batching, communication stalls, memory bottlenecks or insufficient request concurrency.
The important question isn’t simply:
“Are the GPUs busy?”
The better question is:
“How much useful output are we receiving for the compute, power and infrastructure being consumed?”
That is why production AI infrastructure needs to consider several measurements together:
GPU utilisation.
HBM utilisation.
Memory bandwidth.
Tokens per second.
Time to first token.
Inter-token latency.
Batch size.
Request concurrency.
Network utilisation.
Power consumption.
Energy per token.
Cost per token.
Looking at a single metric rarely tells the complete story.
Why DeepSeek and Kimi Are Achieving More With Less Active Compute
We can now return to the original question.
Why are models such as DeepSeek and Kimi achieving extremely strong results without activating their complete parameter count for every token?
MoE is one of the main reasons.
The models can contain enormous overall capacity.
Different experts can learn different patterns.
The router can select the experts most relevant to each token.
Only a small proportion of the complete model performs the major expert computation at one time.
This allows the model’s total capacity to grow much faster than its active compute per token.
However, MoE is only one piece of the overall design.
DeepSeek-V3 also uses Multi-head Latent Attention, multi-token prediction, load-balancing improvements, FP8-oriented training techniques, supervised fine-tuning and reinforcement learning. (arXiv)
Kimi K2 combines MoE with its MuonClip optimiser, large-scale pre-training and specific optimisation for coding and agentic workloads. (GitHub)
The performance comes from the entire system.
The model architecture.
The quality and quantity of training data.
The optimiser.
The attention mechanism.
The training strategy.
The precision format.
The inference software.
The GPU architecture.
And the network connecting the GPUs.
This is not simply a case of creating lots of experts and hoping for the best.
MoE Changes the Network Requirement
From a network infrastructure perspective, MoE also creates an important traffic pattern.
The experts may be distributed across multiple GPUs and multiple GPU servers.
Once the router selects an expert, the token representation may need to travel to the GPU hosting that expert.
After the expert has processed it, the result must be returned and combined.
This introduces all-to-all communication between GPUs during the expert-routing stage.
Dense tensor parallelism also creates east-to-west communication.
However, MoE introduces its own expert-parallel traffic pattern that can be irregular and sensitive to load imbalance.
This means the back-end network becomes extremely important.
High bandwidth is required.
Low latency is required.
Congestion needs to be controlled.
The network needs to move token representations between experts quickly enough that communication doesn’t cancel out the compute savings created by the MoE architecture.
This is why model architecture and infrastructure design cannot be considered separately.
The model dictates the communication pattern.
The communication pattern influences the network fabric.
And the network fabric can ultimately determine whether the theoretical efficiency of the model is realised in production.
Dense vs MoE Summary
Dense Model
All parameters are stored.
All parameters participate in processing every token.
The architecture is comparatively straightforward.
Compute increases directly as the model becomes larger.
Examples include Llama 3.1 8B, 70B and the 405B Dense Transformer. (Meta AI)
Mixture of Experts Model
All parameters normally still need to be stored.
Only selected experts participate in processing each token.
Total model capacity can be much larger than active computation.
Routing and expert-parallel communication introduce additional infrastructure complexity.
Examples include DeepSeek-V3, DeepSeek-R1, Kimi K2 and several Mixtral and Qwen MoE variants.
What Does This Mean for GPU Sizing?
When sizing GPU infrastructure for a Dense model, we mainly need to understand:
Total parameter count.
Precision.
KV-cache requirement.
Expected context length.
Batch size.
Concurrency.
Operational overhead.
When sizing infrastructure for an MoE model, we need all the above…
But we must also understand:
Total parameters.
Active parameters.
Number of experts.
Experts selected per token.
Expert placement across GPUs.
Expert-parallel strategy.
Routing overhead.
Expected load balance.
All-to-all network requirements.
Memory capacity tells us whether the model fits.
Active parameters help us estimate how much model computation occurs per token.
The parallelism strategy tells us how that work is distributed.
And expected concurrency tells us how many users the platform can serve.
Once again…
There is no single number that provides the complete answer.
Final Thoughts
One of the biggest mistakes we can make when learning AI infrastructure is to look at the number of parameters and assume it tells us everything.
It doesn’t.
A 405-billion-parameter Dense model and a one-trillion-parameter MoE model can behave very differently.
The Dense model may require less memory because it contains fewer total parameters.
Yet it may perform significantly more model computation per token because all 405 billion parameters participate.
The MoE model may require more GPUs simply to hold its complete one trillion parameters.
However, only 32 billion may become active for each token.
This is why we must separate three questions.
How much GPU memory is required to store the model?
How much computation is performed for each token?
How efficiently does the complete infrastructure convert power into useful output?
The first question is influenced by total parameters and precision.
The second is influenced by active parameters and model architecture.
The third is measured through throughput, latency, power consumption and energy per token.
This is the real lesson behind Mixture of Experts.
MoE doesn’t make an enormous model physically small.
It doesn’t make inactive GPUs consume zero power.
And it doesn’t remove the need for a high-performance back-end network.
What it does is allow model developers to increase overall model capacity without increasing active computation at exactly the same rate.
That is how models such as DeepSeek and Kimi can contain hundreds of billions or even one trillion parameters while activating only a relatively small proportion for each token.
They are not simply throwing more GPUs at the problem.
They are focusing on extracting more intelligence from every unit of compute.
And from an AI infrastructure perspective…
That changes everything.
Network Automator 
Leave a comment