Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
ISO/IEC 27565 — Privacy Technology — Guidelines for PII Protection in AI
Comprehensive guidelines for protecting PII throughout the AI system lifecycle
1. Introduction to ISO/IEC 27565
ISO/IEC 27565 provides comprehensive guidelines for protecting personally identifiable information (PII) throughout the lifecycle of artificial intelligence (AI) systems, from data collection and model training through deployment, inference, and retirement. As AI systems increasingly process vast quantities of personal data for training and operation, they introduce unique privacy risks that traditional data protection approaches do not adequately address. These include model inversion attacks that reconstruct training data from model outputs, membership inference attacks that determine whether a specific individual’s data was used in training, and the risk of unintended memorization of rare or unique records in training datasets.
Unlike traditional data processing where PII exists primarily in databases and files, AI systems embed PII patterns within model parameters, embedding vectors, and inference outputs. Privacy protection must therefore address not just input data but the model itself as a potential carrier of PII.
2. Privacy Risks Specific to AI Systems
The standard categorizes AI-specific privacy risks into three distinct groups: training data privacy risks (occurring during data collection and model training), model privacy risks (embedded in the trained model’s parameters and behavior), and inference privacy risks (arising when the model processes new data or generates outputs). Each category requires different mitigation strategies and engineering controls.
| Risk Category | Specific Attack/Threat | Affected Lifecycle Stage | Primary Mitigation | Effectiveness |
|---|---|---|---|---|
| Training data privacy | Data breach during collection or labeling | Data collection, preparation | Data minimization, access controls, encryption | High |
| Training data privacy | Unauthorized data inference via model | Model training | Differential privacy during training | High (provable) |
| Model privacy | Model inversion attack | Deployment, inference | Output perturbation, model pruning | Medium |
| Model privacy | Membership inference attack | Deployment, inference | Regularization, DP training, output restriction | Medium-high |
| Model privacy | Model extraction via API queries | Deployment, inference | Query rate limiting, output perturbation | Medium |
| Inference privacy | Attribute inference from model outputs | Inference | Output filtering, confidence score masking | Medium |
| Inference privacy | Unintended memorization | Training, inference | Deduplication, differential privacy, record suppression | High |
Membership inference attacks are particularly insidious because they can be executed with only black-box access to a model. An attacker can determine with high confidence whether a specific individual’s data was used in training — which itself can be a privacy violation if the fact of participation is sensitive (e.g., participation in a medical study).
3. Privacy-Preserving AI Engineering Practices
ISO/IEC 27565 provides detailed engineering guidance for implementing privacy-preserving AI systems across the entire lifecycle. For training data, it recommends data minimization through active learning techniques that select only the most informative samples for labeling, and systematic screening for rare or unique records that are at highest risk of memorization. For the training process itself, differentially private stochastic gradient descent (DP-SGD) is presented as the primary technical control, with guidance on privacy budget (epsilon) allocation based on the sensitivity of the application domain.
At deployment time, the standard recommends implementing inference privacy controls including output perturbation for API endpoints, query rate limiting to prevent extraction attacks, and confidence score masking to reduce information leakage. For models deployed in regulated domains such as healthcare and finance, on-device inference is encouraged to avoid transmitting raw PII to cloud-based inference endpoints whenever possible.
Differential privacy during training provides the strongest known mathematical guarantee against privacy attacks. An epsilon value of 8 or lower provides meaningful protection against membership inference, while values below 1 provide strong protection even against determined adversaries with auxiliary information.
4. Governance and Compliance for AI Privacy
The standard addresses the organizational and governance dimensions of AI privacy, emphasizing that technical controls alone are insufficient without proper governance structures. It recommends establishing an AI privacy review board that evaluates new AI use cases before deployment, conducts privacy impact assessments specific to AI characteristics (model reversibility, data retention in parameters, inference leakage potential), and maintains an inventory of AI systems with PII processing classifications. The standard also provides guidance on transparency obligations, including model cards and dataset documentation that disclose privacy-relevant characteristics such as training data sources, de-identification methods applied, privacy budget consumed, and known limitations regarding re-identification risk.
The European Union’s AI Act classifies AI systems used for biometric categorization, emotion recognition, and social scoring as high-risk, requiring conformity assessments that include privacy compliance verification. ISO/IEC 27565 provides the technical framework to satisfy these regulatory requirements, making it an essential tool for AI developers targeting the European market.
5. Frequently Asked Questions
Q: Can ISO/IEC 27565 be applied to open-source AI models?
A: Yes. The guidelines apply regardless of whether the model is proprietary or open-source. For open-source models, special attention should be paid to training data provenance and whether the model weights may contain embedded PII that could be extracted by fine-tuning or probing attacks.
A: Yes. The guidelines apply regardless of whether the model is proprietary or open-source. For open-source models, special attention should be paid to training data provenance and whether the model weights may contain embedded PII that could be extracted by fine-tuning or probing attacks.
Q: How does differential privacy affect model accuracy?
A: There is a fundamental trade-off between privacy and accuracy. The standard provides guidance on calibrating this trade-off based on application requirements. In practice, for many tasks, a privacy budget of epsilon=4-8 achieves acceptable accuracy (within 1-3% of non-private baseline) while providing strong privacy protection.
A: There is a fundamental trade-off between privacy and accuracy. The standard provides guidance on calibrating this trade-off based on application requirements. In practice, for many tasks, a privacy budget of epsilon=4-8 achieves acceptable accuracy (within 1-3% of non-private baseline) while providing strong privacy protection.
Q: Does the standard cover large language models (LLMs) and generative AI?
A: The standard was designed to be architecture-agnostic and applies to all AI/ML paradigms including LLMs and generative models. For generative AI specifically, additional considerations include prompt injection attacks that may leak training data, and the risk of generating PII in model outputs.
A: The standard was designed to be architecture-agnostic and applies to all AI/ML paradigms including LLMs and generative models. For generative AI specifically, additional considerations include prompt injection attacks that may leak training data, and the risk of generating PII in model outputs.
Q: What is the recommended approach for AI systems that need to process PII for legitimate purposes?
A: The standard recommends a tiered approach: minimize PII at the source, apply de-identification before training where possible, use privacy-preserving techniques (DP, federated learning) during training, implement strict access controls at inference time, and establish clear data retention and model retirement policies.
A: The standard recommends a tiered approach: minimize PII at the source, apply de-identification before training where possible, use privacy-preserving techniques (DP, federated learning) during training, implement strict access controls at inference time, and establish clear data retention and model retirement policies.
