How to Automatically Scrub Passport Data and Card Numbers from Logs

Recently we were investigating an incident in production where a crashed service dumped a full JSON with the user's request body into ELK. It contained a credit card number, home address, and phone number. Logging everything for debugging purposes is common practice, but when such dumps end up in storage, security teams come with some very uncomfortable questions. Writing regex patterns for every little thing gets old fast. They break on complex formats, miss names, and confuse random number sequences with real identifiers.

There are different ways to solve this problem. One of the most mature open-source tools available is Presidio, developed by the community under the data-privacy-stack organization.

What's Under the Hood and Why You Need It

Presidio helps find and mask personally identifiable information (PII) in texts, spreadsheets, and even images. The project was originally created by Microsoft engineers as an open library, and has since moved to a separate organization, gathering over 10,000 stars on GitHub.

The core idea here is simple: split the privacy protection task into two independent steps — analysis and anonymization.

Instead of one monolithic script, you get two separate modules:

1. presidio-analyzer — examines incoming text, finds entities like passport numbers, phone numbers, email addresses, or names, and returns the coordinates of findings along with a confidence score.
2. presidio-anonymizer — takes the analyzer's markup and applies transformation rules to it: replaces with placeholders, encrypts, masks with asterisks, or simply removes.

This separation is convenient when you need to log only the fact of data detection without modifying the text itself, or when you want to flexibly configure the masking method for different field types.

How Entity Detection Works

If you've tried parsing text with regular expressions alone, you know their main drawback: regex doesn't understand context. It will find a sequence of 16 digits, but won't know whether it's a card number or a product SKU in a warehouse.

Presidio combines multiple approaches at once:

• Regular expressions and checksums (for example, the Luhn algorithm for bank card numbers or IBAN validators).
• Pretrained NLP models based on spaCy or Hugging Face Transformers for named entity recognition (NER) — people, locations, organizations.
• Contextual word analysis. If a number sequence is preceded by words like "phone," "tel," or "call," the detector's confidence score increases.
• Dictionaries and stop word lists.

Python Example

Let's look at what a basic Python pipeline looks like. First, install the packages:

pip install presidio-analyzer presidio-anonymizer
python -m spacy download en_core_web_lg

First, run the analyzer to find vulnerable spots in the string:

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

text = "Привет, меня зовут John Doe. Мой рабочий email [email protected], а телефон +1-555-0199."

analyzer = AnalyzerEngine()
results = analyzer.analyze(text=text, entities=["EMAIL_ADDRESS", "PHONE_NUMBER", "PERSON"], language='en')

print("Найденные сущности:")
for res in results:
    print(f"Тип: {res.entity_type}, позиция: {res.start}:{res.end}, уверенность: {res.score:.2f}")

After the analyzer returns offsets and entity types, connect the anonymization module:

anonymizer = AnonymizerEngine()
anonymized_result = anonymizer.anonymize(
    text=text,
    analyzer_results=results
)

print("\nРезультат обработки:")
print(anonymized_result.text)

The output is a string where names are replaced with <PERSON> and contacts are replaced with <EMAIL_ADDRESS> and <PHONE_NUMBER>. If desired, you can configure hashing, partial character masking, or replacement with fake data via Faker.

Adding Custom Rules

Out of the box, the project is well-tuned for international formats: US social security numbers, Visa and Mastercard cards, international phone numbers, email. For local data like Russian passports, INN, or SNILS, you'll need to write your own recognizers.

It's quite simple to do using built-in classes:

from presidio_analyzer import Pattern, PatternRecognizer

# Паттерн для поиска СНИЛС (формат XXX-XXX-XXX YY)
snils_pattern = Pattern(
    name="snils_pattern",
    regex=r"\b\d{3}-\d{3}-\d{3}\s\d{2}\b",
    score=0.85
)

snils_recognizer = PatternRecognizer(
    supported_entity="RU_SNILS",
    patterns=[snils_pattern],
    context=["снилс", "страховой", "пенсионный"]
)

analyzer.registry.add_recognizer(snils_recognizer)

When the analyzer sees a match against the regular expression and finds words from the context list nearby, it raises the confidence score to maximum.

Other Library Features

In addition to working with plain text, the repository contains helper modules for related tasks:

• presidio-image-redactor — finds text in images and PDF scans via OCR (Tesseract) and paints over detected personal data with black boxes right in the pixels. Useful for processing passport scans before sending to external services.
• LLM pipeline integration. Presidio is often used as middleware before sending prompts to OpenAI or Anthropic: you scrub the user's personal data from the request, send the anonymized text to the model, and on the way back, restore the data to its place if necessary.
• REST API on FastAPI, packaged in Docker containers. You don't have to write the entire service in Python — you can deploy the analyzer as a microservice and call it from Go, Java, or Node.js backends.

Potential Challenges

The first nuance is resource hunger. If you connect heavy transformers like RoBERTa or BERT for more accurate entity recognition, inference will require memory and slow down data stream processing. Running a full NLP pipeline synchronously on hundreds of thousands of RPS will be heavy — you'll need to offload masking to async Kafka or Celery workers.

The second point is Russian language models. The basic en_core_web_lg from spaCy handles English well, but for Russian you'll need to either connect the ru_core_news_lg model or fine-tune your own NER recognizer for your text specifics.

Practical Applications

Presidio comes in handy when a company needs to comply with Federal Law 152, GDPR, or HIPAA, but rewriting the existing infrastructure from scratch is too expensive.

The library fits well into three scenarios:

• Scrubbing logs and traces before sending to monitoring systems (Sentry, OpenTelemetry, ELK).
• Preparing datasets for ML model training or analytics on real user data.
• Filtering incoming and outgoing messages in chatbots and RAG systems based on large language models.

If you're looking for a ready-made toolkit for data masking without having to write parsers from scratch, the repository is definitely worth bookmarking and testing locally with your own examples.