⏱ 7 min read
In a previous article we outlined four levels of protection for working with artificial intelligence and sensitive information. The second of them, substituting identifiers before sending a document, is the one most organizations adopt first. It is accessible, it can be implemented in days, and it produces an immediate sense of control.
That sense deserves precision. Not because the technique is useless, but the opposite: it is useful, which is exactly why it is worth knowing where its protection ends. A layer that is well understood gets combined with others. A layer that is overestimated gets left on its own.
What follows is what published research shows today, and how to work with that information rather than around it.
First, how it works
The mechanism is straightforward. A process scans the document, locates identifiers, replaces them with markers or with substitute values, and stores the correspondence separately. The transformed version is sent. When the result returns, the substitution is reversed.
On how to substitute, one finding is consistent across the literature and worth adopting from the start: replacing with plausible values of the same type works better than inserting generic markers. A document containing a fictitious name retains its grammatical structure and the model processes it normally. A document littered with brackets breaks textual continuity and degrades the output.
This has an operational consequence: substitution must be consistent. The same entity, the same pseudonym throughout the document. Otherwise the ability to track a subject across paragraphs is lost.
The first limit: detection is uneven
Identifiers with fixed structure are reliably detected. Tax identification codes, national identity numbers, bank account numbers, phone numbers: all follow verifiable alphanumeric rules, many with check digits. That part of the problem is solved.
Names are another matter. They require entity recognition, and performance there depends heavily on the text. An independent evaluation published this year compared several detection systems across 42 test sets in 22 languages. Microsoft Presidio, the most widely used open source tool, scored an F1 of 0.431 on a synthetic dataset and 0.273 on medical text in its base configuration. OpenAI’s purpose-built filter reached 0.855 and 0.464 respectively.
A subsequent comparative exercise was harsher. Evaluating eight systems across ten unified datasets covering 48 entity types, the best of them scored an F1 of 0.14. The same OpenAI filter that reports 0.96 on its reference benchmark drops to a range of 0.18 to 0.65 when tested on out-of-distribution text.
The conclusion is not that the tools are bad. It is that vendor-published figures are measured on the kind of text the tools were tuned for. Change the domain and the result changes.
The second limit: Spanish has cases of its own
Two patterns common in Mexican documentation produce errors in opposite directions, and both matter.
Surnames with particles. Names such as “de los Santos Aguirre” or “del Río Vergara” fragment poorly. Models tend to treat the particle as a connector and exclude it from the entity, marking part of the surname and leaving the rest in the text. A partial name remains identifying, particularly when the surname is uncommon in the region.
Corporate names containing proper nouns. Denominations such as “Comercializadora Miguel Hidalgo, S.A. de C.V.” trigger the opposite error. The system labels the historical name as a natural person and replaces it with another personal name, turning the corporate name into a company that does not exist.
The outcome is defective under either intention. If the goal was to preserve the denomination, information that was not personal data and that the analysis required has been lost. If the goal was to conceal it for confidentiality reasons, that was not achieved either: an invented name protects no commercial relationship and corrupts the data in the process. In that case the correct treatment is a consistent label, not a different name.
Neither case is well covered by available test sets, most of which use peninsular or generic Spanish. It is a documented and still open gap.
The third limit: what remains after substitution
This is the central point and the least discussed.
A study published this year by researchers at ETH Zurich, with participation from Anthropic, systematically evaluated whether current models can re-identify people from text stripped of direct identifiers. The results are clear.
Cross-referencing profiles from a technical community against public professional profiles, an automated agent correctly identified 226 of 338 subjects at 90 percent precision. Compared with classical methods requiring structured data, the difference is of another order: where the prior approach recovered 0.1 percent of cases, the model-assisted approach recovered 45.1 percent while holding 99 percent precision.
The attack scales. Extrapolating to a universe of one million candidates, the method retains roughly 35 percent effectiveness. And even in scenarios where barely one in ten thousand subjects is identifiable, it holds around 9 percent.
The authors conclude that practical obscurity, meaning the protection afforded by the simple fact that re-identifying someone was too costly to attempt, no longer holds. Not because models exceed a capable human investigator, but because they do in minutes what would take that person hours.
And here is the nuance that changes practice
That same body of research records a result usually omitted when the above is cited. When re-identification was evaluated on anonymized court decisions, the models struggled, even though they achieved high rates on public biographies.
The difference is not in the anonymization technique. It is in the subject.
Re-identification requires reference material. If an abundant public trail exists about a person, professional profiles, publications, press coverage, appearances in registries, then the contextual fragments of a pseudonymized document have something to be matched against. If that trail does not exist, the same fragments lead nowhere.
This usefully reorders risk assessment. Residual risk is not a property of the document or of the tool. It is a property of the person described in it.
Two distinct regimes, not one
The analysis so far has focused on personal data. It is worth noting that much of what an organization should not send to an artificial intelligence tool is not personal data at all.
Pricing structure, margins, client lists, unsigned proposals, negotiated terms, commercial strategy, and information a third party shared under an obligation of confidentiality. None of it has a data subject to protect, and yet it is precisely what should not leave.
These are two regimes with different logics.
Personal data imposes a legal obligation. It applies regardless of what the organization prefers, the standard is set by law, and the test of sufficiency is effective dissociation.
Confidential information imposes a contractual obligation. It applies to the extent of what was signed, the standard is set by the contract itself, and the test of sufficiency depends on each relationship.
The distinction matters because the techniques described earlier perform differently in each case.
For personal data, replacing a name with another plausible name serves its purpose: the person ceases to be directly identifiable.
For confidential information, that same substitution may protect nothing. If a document describes the terms of a contract, changing the counterparty’s name does not conceal the contract. What reveals the commercial relationship is the content, not the name. Conversely, when the goal is to conceal the counterparty, the correct substitute is not another name but a consistent label that preserves traceability without inventing a company that does not exist.
An organization that classifies its information solely by data protection criteria leaves out half the problem.
How to work with this
None of these limits calls for abandoning the technique. They call for applying it with judgment. Six practices follow directly from the evidence above:
Classify by subject, not only by document. Before deciding on treatment, consider how publicly exposed the person described is. An executive with a visible professional presence requires more transformation than someone without a digital footprint, even when the document is the same.
Treat name detection as partial. Structured codes are detected reliably; names are not. Human review of a sample, with specific attention to surnames with particles and to corporate denominations, quickly reveals the error margin particular to your documents.
Distinguish natural from legal persons by design. Corporate names are not personal data and are often worth preserving. A prior catalog of the relevant corporate entities prevents the system from turning them into invented names.
Consider context, not only identifiers. A description such as “procurement lead of a given institution between two specific years” identifies without naming. When a document combines roles, periods and locations, generalizing those elements yields more than substituting one more name.
Protect the correspondence table as the asset it is. The moment a file exists translating keys to people, that file concentrates the entire sensitivity of the process. Its location, access control and retention period are part of the control, not an afterthought.
Review your confidentiality obligations beforehand, not afterward. Many commercial contracts include clauses that draw no distinction between processing channels. Running a client’s documentation through a third-party tool may constitute a breach of something already signed, regardless of whether the information contains personal data. That review belongs at the point of defining policy, not when a conflict arises.
When this layer stops being enough
The four layers described in the previous article were ordered by cost, from the simplest to the most expensive. That ordering may suggest they are interchangeable and that choosing comes down to budget. They are not.
The first two layers reduce exposure. The last two eliminate it or make it enforceable against a third party. The difference is not one of degree.
When information leaves the organization, even transformed, protection depends on how well it was transformed and how much context remained. The reviewed evidence shows that this bet carries a measurable and non-trivial margin of error. When information does not leave, or leaves under conditions someone signed and answers for, the problem changes in nature.
Four signals indicate that substituting identifiers is no longer the appropriate layer:
The subject has a broad public footprint. If the person or company described appears in registries, press coverage, professional profiles or publications, the contextual fragments remaining after substitution have something to be matched against. Here the transformation would have to be so aggressive that the document loses analytical usefulness.
The volume is recurring. Manual review of a sample works for ten documents a month. For a thousand, the detection error margin stops being an exception and becomes a rate. At that point local processing usually costs less than review.
The obligation is contractual and admits no nuance. If the client contract prohibits sharing their documentation with third parties without distinguishing between channels, no transformation cures the breach. What was sent, was sent. The only path is for processing to occur on your own infrastructure or under an agreement the client expressly accepts.
The document requires full analysis. Field extraction, classification and summarization tolerate substitution well. Reasoning about a case, detecting contradictions between clauses, or following a complex chronology requires the complete content. There the first layer gets in the way, and going directly to the second or third makes more sense.
None of this makes substitution useless. It places it: it is the layer that handles routine work with moderately sensitive information, and the interim step while the rest is implemented. Presenting it as the complete solution is what produces false confidence.
The two layers that follow are covered in In-house model or contract: two ways to use AI without exposing information.
On the applicable framework in Mexico
One recent change is worth noting. The new Federal Law on Protection of Personal Data Held by Private Parties took effect on 21 March 2025, and the functions previously exercised by the prior supervisory body passed to the Secretariat of Anti-Corruption and Good Governance.
Among the relevant changes, the law broadens the notion of controller to any natural or legal person carrying out processing of personal data, without requiring that they make decisions about such processing. The universe of obligated parties widens and expressly incorporates processors.
Mexican legislation articulates the concept of dissociation, understood as the procedure preventing data from being associated with its holder by its structure, content or degree of disaggregation. What the reviewed technical evidence adds to that definition is an operational criterion: if a third party with access to public sources can reconstruct the association, dissociation was not achieved, regardless of how many identifiers were substituted.
In summary
Substituting identifiers genuinely reduces exposure and is worth implementing. What it does not do is produce an anonymous document.
Knowing this does not weaken the practice; it orders it. It allows deciding which documents this layer covers on its own, which additionally require prior minimization, and which simply should not leave the organization in any form.
That decision, as we noted before, is not technical. It rests on a classification of information that each organization must perform, and that no tool can perform on its behalf. A classification that, to be complete, must account both for what the law requires protecting and for what the organization committed not to disclose.
Sources consulted
Lermen, S., Paleka, D., Swanson, J., Aerni, M., Carlini, N., Tramèr, F. Large-scale online deanonymization with LLMs. arXiv:2602.16800
Uppala, R. Evaluating OpenAI’s Privacy Filter: Cross-Lingual, Cross-Domain PII Detection Across 42 Benchmarks. arXiv:2608.02616
Nyffenegger, A., Stürmer, M., Niklaus, J. Anonymity at risk? Assessing re-identification capabilities of large language models in court decisions. Findings of the ACL: NAACL 2024
Decree issuing the Federal Law on Protection of Personal Data Held by Private Parties. Official Gazette of the Federation, 20 March 2025