Data Leakage

In 2023, it was revealed that Samsung engineers had pasted confidential processor source code into ChatGPT to request optimization, forcing the company to impose a company-wide ban on AI chat tools. This incident symbolizes the data leakage risks of the AI era. Data leakage is a collective term for the problem of AI systems unintentionally exposing confidential or personal information contained in training data or user inputs.

Information Extraction from Training Data

One of the fundamental risks LLMs face is memorization and reproduction of training data. In the process of learning from trillions of tokens of text, models unintentionally memorize personally identifiable information (PII) such as phone numbers, email addresses, and physical addresses, as well as fragments of source code and internal documents. A 2023 study by Google's DeepMind team demonstrated that a simple prompt causing ChatGPT to repeat specific tokens could extract real individuals' personal information from the training data. Since larger parameter counts lead to greater memorization, the risk grows as models scale up.

Leakage Through Prompts

Another pathway is information leakage through user inputs. As in the Samsung case, when employees input business data into AI services, that data is sent to the service provider's servers. The concern is that such data may be used for additional model training or stored long-term in server logs. There is also the risk of confidential corporate instructions embedded in system prompts being exposed through prompt injection attacks. In RAG systems, incidents have been reported where inadequate access controls on the search database result in responses containing information that users should not have access to.

A Defense-in-Depth Approach

Effective countermeasures require defense at multiple layers. At the input stage, tools like Microsoft Presidio are used to automatically detect and mask PII. At the model stage, differential privacy is applied during training to suppress memorization of individual data points, or personal information is removed from training data in advance (data sanitization). At the output stage, generated text is filtered to check for PII. No single measure is sufficient -- combining multiple approaches is essential.

Actions Companies Should Take Now

For companies using AI in their operations, data leakage prevention is a top priority. The first step is to use enterprise plans that do not use input data for training offered by major AI providers (OpenAI API, Anthropic Claude API, Azure OpenAI Service, etc.). Next, establish an AI governance policy that clearly defines what can and cannot be input into AI, and communicate it to all employees. For highly sensitive operations, deploying local LLMs (such as Llama-based models) should also be considered. AI convenience and data protection are not mutually exclusive -- they can be achieved together with proper design.