DeepSeek shook up the U.S. stock market, and it’s still creating shock wavers around world. But the newest allegation is that DeepSeek actually used a particular process to put together its training data, and it’s one that some consider to be a little shady.

The new U.S. president’s AI and crypto czar David Sacks is one of those who is getting in on the action, saying in an interview with Fox News that there was “substantial evidence” that this kind of thing was going on.

”I think one of the things you’re going to see over the next few months is our leading AI companies taking steps to try and prevent distillation,” he said. “That would definitely slow down some of these copycat models.”

When you comb through these reports, there’s one word that keeps coming up again and again, and that’s “distillation.” What is distillation, and why is it important?

The Teacher/Student Model

In the AI world, distillation refers to a transfer of knowledge from on model to another. I came across this resource from Microsoft that describes it in greater detail.

Distillation is a technique designed to transfer knowledge of a large pre-trained model(the “teacher) into a smaller model(the “student”), enabling the student model to achieve comparable performance to the teacher model. This technique allows users to leverage the high quality of large LLMs, while reducing inference cost in a production environment, thanks to the smaller student model.

So in many cases, the distillation is being done to get the refined results from a big model onto a smaller, more efficient model. That may not be conventionally true in DeepSeek’s case, there’s something different going on there, but it can be very useful in, say, learning to apply robust AI to endpoint devices.

”Distillation represents a significant step forward in development and deployment of LLM/SLM at scale,” the analyst continue. “By transferring the knowledge from a large pre-trained model to a smaller, more efficient model, distillation offers a practical solution to the challenges of deploying large models, such as high costs and complexity. This technique not only reduces model size and operational costs but also enhances the performance of student models for specific task.”

Uses of Distillation in Autonomous Vehicles

One of the prime examples of this activity is to put sophisticated computer vision models into autonomous vehicles.

(This type of) learning has shown immense potential in various application domains, including autonomous driving, robotic control, and healthcare. In autonomous driving, split learning enables the efficient training and fine-tuning of AI models for tasks such as sensor fusion, object detection, and decision-making, all while minimizing energy consumption and ensuring real-time responsiveness.

To understand that, It’s important to know that the convolutional neural network or CNN is specifically made for computer vision and object detection.

Unlike other kins of neural nets, the CNN has particular metrics and layouts that allow the system to process what surround it in a visual field. So transmitting this knowledge to a more efficient model can be absolutely important for coming up with better self-driving models that are safer and more effective.

Other types of Distillation

The Microsoft piece also goes over various flavors of distillation, including response-based distillation, feature-based distillation and relation-based distillation. It also covers two fundamentally different modes of distillation – offline and online distillation.

The online method is more direct in real time, and the offline model is more a product of a pre-training process.

Then there’s self-distillation, where one model can do two things, and separate two process, to essentially learn from itself.

In any case, this term, distillation, is going to be useful because it gets to the heart of how we evaluate neural networks. What are the rules? Right now, the U.S. is trying to tighten export controls to keep the Chinese from doing this sort of thing, and making “imitations” of powerful LLM systms.

2025.1.31 – from Forbes