Continued Pretraining

When you ask a general-purpose LLM about specialized medical terminology or request complex reasoning in Japanese, the responses may fall short of expectations. When prompt engineering and RAG reach their limits, an approach that expands the model's knowledge itself is needed. Continued pretraining is a technique that trains a pretrained model on large volumes of domain-specific text to acquire specialized knowledge or language capabilities.

A Clear Distinction from Fine-Tuning

Often confused, continued pretraining and fine-tuning serve different purposes. Fine-tuning (SFT) uses question-answer pair data to adapt the model to specific task formats. In contrast, continued pretraining trains on unlabeled raw text using the same next-token prediction approach as the original pretraining. The goal is not task optimization but expanding the model's knowledge base itself. By analogy, fine-tuning is like "practicing job interview responses," while continued pretraining is like "reading hundreds of specialized textbooks." Typically, continued pretraining is performed first, followed by fine-tuning.

The Battleground for Japanese LLM Development

One of the areas where continued pretraining is most actively used is Japanese LLM development. Many open-source LLMs are primarily trained on English data, leaving their Japanese capabilities insufficient. To address this, the approach of performing extensive additional training with Japanese text on top of base models like Meta's Llama or Mistral has been widely adopted. Specifically, many of Japan's leading models -- CyberAgent's CALM, Preferred Networks' PLaMo, ELYZA's ELYZA-japanese-Llama, and StabilityAI's Japanese StableLM -- were developed through continued pretraining. By training on tens to hundreds of billions of Japanese tokens, they have dramatically improved Japanese fluency and cultural understanding.

Catastrophic Forgetting and Replay Strategies

The most critical issue to watch for in continued pretraining is catastrophic forgetting. In the process of adapting to new domain data, the model's original capabilities in English and general knowledge can degrade. For example, continued pretraining exclusively on Japanese data can significantly reduce reasoning ability in English. The primary countermeasure is the replay strategy, which mixes in a portion of the original training data (typically 5-20%) alongside the new domain data. Other important techniques include setting the learning rate lower than during initial pretraining and gradually adjusting data ratios.

When to Choose Continued Pretraining

Continued pretraining requires data on the scale of billions of tokens and substantial GPU compute time, so costs are far from trivial. It should be considered only when deep domain expertise or language capabilities are needed that RAG and fine-tuning cannot adequately address. For fundamental issues like "cannot understand specialized terminology" or "produces mixed-language output," continued pretraining is effective. But for something like "output in a specific format," fine-tuning is sufficient.