Incident Response Case Study: When Finding the Vulnerability Isn't Enough - Lessons from a Compromised AWS SES Key

Incident Response · Case Study · March 2026

How an EdTech company found one compromised AWS credential, thought the crisis was over, and why that assumption could have cost them everything. A real incident response case study with the lessons most companies learn too late.

💫 Key Takeaways

• Finding the source of one compromised credential does not mean you understand the full scope of a breach
• If an AWS SES key was exfiltrated, the attacker likely had access to every other secret stored alongside it
• Self-diagnosing a security incident is like self-diagnosing chest pain — you might be right, but the consequences of being wrong are catastrophic
• A proper incident response investigation covers containment, forensics, data impact assessment, and root cause analysis — not just patching the hole you found
• The average cost of an undetected data breach is $4.88M. A professional IR engagement costs a fraction of that
• Companies with self-hosted source control (GitLab, Gitea) face elevated risk — a compromised server means the entire codebase and CI/CD pipeline could be backdoored

Our first step in any incident response engagement is understanding the environment. We asked LinguaFlow's CTO to walk us through their tech stack, and what he described was a modern, moderately complex SaaS architecture:

We explained our standard incident response process: containment first, then forensic investigation, then remediation and hardening. We set up a secure communication channel and began scoping the engagement. The CTO seemed relieved. Someone was taking charge.

Then, less than four hours later, we got a second call.

We strongly recommended they continue with the investigation. We explained why finding one vulnerability is not the same as understanding the full scope of a compromise. But they were under budget pressure, their immediate problem (the spam emails) had stopped after the key rotation, and their team was confident they had found the root cause.

They thanked us for the quick response and said they'd be in touch.

This is the story I want to tell — not because LinguaFlow did something unusual, but because what they did is extraordinarily common. And it is precisely the wrong response to a security incident.

Let me be blunt: finding the exploit vector is not the same as understanding the full scope of a compromise. Not even close. It is the difference between finding the broken window the burglar used to enter your house and actually checking whether anything was stolen, whether they made copies of your keys, or whether they are still hiding in the basement.

In LinguaFlow's case, a developer had committed an AWS SES access key to a GitLab repository. They found it, rotated it, and removed the commit. Problem solved, right?

Here is everything that sentence does not tell you:

1. What else was in that commit — or in that repository?

In a typical Symfony application, environment variables and configuration files often contain clusters of secrets. The .env file in a Symfony project commonly holds:

• Database credentials (MySQL host, username, password, database name)
• AWS access keys (not just SES — often the same IAM key is used for S3, SQS, CloudFront, and SES)
• Redis and Memcached connection strings
• Application secrets (APP_SECRET used for CSRF tokens, cookie encryption, password reset tokens)
• Third-party API keys (payment processors, analytics, notification services)
• GitLab API tokens or deploy tokens used in CI/CD

If the SES key was committed to the repository, the question is not “was the SES key exposed?” The question is “what else was exposed, and for how long?”

2. How long did the attacker have access?

LinguaFlow detected the compromise because the attacker started sending spam through their SES account. But sophisticated attackers do not announce themselves. The fact that the key was being used for something noisy and obvious (spam) suggests one of two scenarios:

Without analyzing AWS CloudTrail logs, there is no way to distinguish between these scenarios. And CloudTrail logs have a limited retention window — if you wait too long, the evidence disappears.

3. Was the GitLab instance itself compromised?

This is the question that should have kept LinguaFlow's CTO up at night. Their GitLab was self-hosted, which means they were responsible for patching, hardening, and monitoring it themselves. If the attacker found credentials in a GitLab repository, they may have had access to GitLab itself. And a compromised self-hosted GitLab instance is an attacker's dream:

• Full source code access — every repository, every branch, every commit history
• CI/CD pipeline manipulation — the attacker could modify .gitlab-ci.yml to inject malicious code into every deployment
• Stored secrets — CI/CD variables often contain production database passwords, API keys, and deployment credentials
• Deploy keys and access tokens — used to push code and trigger deployments to production
• Container registry access — if they use GitLab's built-in registry, the attacker could push trojanized Docker images

4. What about persistence mechanisms?

Experienced attackers do not just exploit a vulnerability and leave. They establish multiple ways back in, so that even if the original vulnerability is patched, they retain access. Common persistence mechanisms in an environment like LinguaFlow's include:

Rotating the SES key addresses exactly one of these eight persistence vectors. The other seven remain completely uninvestigated.

5. What about the 2 million users?

LinguaFlow's MySQL database contained personally identifiable information (PII) for over two million users: names, email addresses, learning preferences, potentially payment information. If the database credentials were in the same .env file as the SES key — which they almost certainly were, since that is how Symfony's Doctrine ORM is typically configured — then the attacker may have had direct database access.

Without a forensic investigation, LinguaFlow cannot answer these questions:

• Was the user database accessed or exfiltrated?
• Were payment records compromised?
• Do they have a legal obligation to notify users under GDPR, CCPA, or other privacy regulations?
• Are their users now targets for credential stuffing attacks on other platforms?

Had LinguaFlow proceeded with the engagement, here is what a proper incident response investigation would have covered. This is not theoretical — it is the standard process we follow for every cloud credential compromise.

Phase 1: Containment (Hours 0–4)

Containment is not “rotate the key you found.” Containment is rotating every credential that could possibly have been exposed and cutting off every potential access path. For LinguaFlow's stack, that means:

• All AWS IAM access keys and secret keys — not just the SES key. Every key pair associated with every IAM user and role.
• MySQL database passwords — change the root password and all application user passwords. Audit the mysql.user table for unknown accounts.
• Redis and Memcached authentication — if using password-protected connections, rotate those passwords. If not password-protected (common with Memcached), assess network exposure.
• Symfony APP_SECRET — this single value is used for CSRF protection, cookie signing, and password reset tokens. If compromised, an attacker can forge sessions and generate valid password reset links for any user.
• GitLab personal access tokens, deploy tokens, and CI/CD variables — all of them, for all users and projects.
• SSH keys — on all servers, for all user accounts.
• Third-party API keys — payment processors, analytics, any integrated service.

Phase 2: CloudTrail Log Analysis (Hours 4–24)

AWS CloudTrail is the single most important forensic artifact in a cloud compromise. Every API call to AWS is logged (if CloudTrail is enabled — and if it is not, that is its own problem). For LinguaFlow's investigation, we would analyze:

• All API calls made with the compromised access key — not just SES calls. Did the key also call S3 GetObject? Did it list IAM users? Did it create new resources?
• IAM modifications — CreateUser, CreateAccessKey, AttachUserPolicy, CreateRole — any of these indicate the attacker was establishing persistent access.
• S3 activity — ListBuckets, GetObject, PutBucketPolicy — were backups, user uploads, or application data accessed?
• Lambda function creation — attackers frequently deploy Lambda functions for crypto mining or as persistent data exfiltration endpoints.
• EC2 and ECS modifications — new instances, modified security groups, changed IAM instance profiles.
• SQS queue access — did the attacker read messages from queues that might contain sensitive application data?
• Source IP analysis — correlating the IP addresses making API calls to identify the attacker's infrastructure and determine whether access came from expected or unexpected geographic locations.

Phase 3: GitLab Forensics (Hours 12–48)

Because LinguaFlow's GitLab was self-hosted, this phase is critical. A managed GitLab.com account has its own security team monitoring for abuse. A self-hosted instance is entirely the customer's responsibility. We would investigate:

• GitLab audit logs — login attempts, successful logins, repository access, CI/CD pipeline modifications, user creation, permission changes.
• CI/CD pipeline history — were any pipeline definitions modified? Were new jobs added that could exfiltrate secrets or inject code?
• Repository commit history — using git log and git diff to look for suspicious commits, especially to .gitlab-ci.yml, Dockerfiles, and deployment scripts.
• CI/CD variables — what secrets are stored as CI/CD variables? Were any accessed or modified?
• Webhooks — attackers sometimes add webhooks that send repository data to external servers on every push.
• Container registry — if LinguaFlow uses GitLab's container registry, we would verify the integrity of all Docker images used in production.

Phase 4: Infrastructure Forensics (Hours 24–72)

With Docker containers running on Apache, and caching layers in Memcached and Redis, there are multiple places an attacker could establish persistence or hide evidence:

• Docker image integrity — compare running images against known-good images from the registry. Look for modified layers.
• Running processes — check for unexpected processes inside containers and on host systems.
• Cron jobs — both inside containers and on the host OS. Attackers love cron jobs because they survive container restarts.
• File system analysis — look for web shells in the PHP application directory. A single malicious .php file can give an attacker full remote command execution.
• Apache configuration — check for modified virtual hosts, reverse proxy rules, or .htaccess files that could redirect traffic or expose sensitive endpoints.
• Network connections — analyze outbound connections for unexpected destinations (command-and-control servers, data exfiltration endpoints).

Phase 5: Data Impact Assessment (Hours 48–96)

This is the phase most companies want to skip — and the one that matters most for legal and regulatory obligations:

• Database access audit — review MySQL query logs (if enabled) or general logs for evidence of data access by unauthorized parties.
• S3 access logs — determine whether user-uploaded content (profile photos, assignment submissions, etc.) was accessed.
• PII inventory — precisely catalog what personal data was potentially exposed: names, email addresses, IP addresses, learning activity, payment data.
• Regulatory notification requirements — based on the data types and user locations, determine whether notification is required under GDPR (72-hour window), CCPA, or other applicable regulations.
• User risk assessment — if email addresses and hashed passwords were exposed, users on other platforms using the same credentials are at risk of credential stuffing attacks.

Phase 6: Root Cause Analysis & Hardening (Hours 72–168)

Only after all of the above is complete do we focus on the root cause and long-term remediation:

• How did the key get committed? Was it a developer mistake? A misconfigured .gitignore? A merge conflict that accidentally included the .env file?
• Secrets management implementation — move from .env files to AWS Secrets Manager, SSM Parameter Store, or HashiCorp Vault.
• Pre-commit hooks — implement tools like git-secrets, trufflehog, or gitleaks to prevent secrets from being committed in the first place.
• IAM least privilege — ensure AWS access keys have only the permissions they need, not broad AdministratorAccess or wildcard policies.
• GitLab hardening — enable two-factor authentication, restrict IP access, implement audit logging, and consider moving to GitLab's managed offering.
• Monitoring and alerting — set up AWS GuardDuty, CloudWatch alarms for IAM changes, and SES sending anomaly detection.

If you are reading this because your organization is currently dealing with a compromised AWS SES key, here is the immediate action checklist. Do these in order, do them now, and then call a professional incident response team.

After years of responding to cloud credential compromises, I have developed a list of indicators that tell me a breach goes deeper than the initial finding. If any of these apply to your situation, do not attempt to handle the incident internally:

Whether or not LinguaFlow's breach went deeper than the SES key, the point is this: they do not know, and now they cannot know. The window for effective forensic investigation closes rapidly. CloudTrail logs age out. Containers get redeployed. Memory-resident evidence vanishes. Every day without a proper investigation is a day where evidence is being lost and risk is compounding.

Here are the lessons that apply to every organization, not just LinguaFlow:

1. Never self-diagnose a security incident

Your internal team knows your systems better than anyone. That is genuinely valuable during an incident. But incident response is a specialized discipline that requires a different mindset than building and maintaining systems. Your developers will look for bugs. An IR specialist looks for evidence of adversarial activity — a fundamentally different lens. Bring in external incident response expertise immediately, even if your team is highly capable.

2. “Finding the bug” and “understanding the breach” are fundamentally different

Finding the bug means identifying how the attacker got in. Understanding the breach means answering: what did they access? How long were they inside? What did they change? What did they take? What did they leave behind? Finding the bug is step one of a multi-step process. Stopping at step one is like a doctor diagnosing a broken arm and sending you home without checking for internal bleeding.

3. The math is not complicated

4. Credential rotation must be comprehensive, not targeted

When one credential is compromised, assume they all are. This is especially true in environments that use .env files, CI/CD variables, or any other mechanism that stores multiple credentials together. Rotating the SES key but leaving the database password, the Redis password, the Symfony APP_SECRET, and the GitLab deploy tokens unchanged is not containment. It is wishful thinking.

5. Self-hosted source control is a crown jewel — treat it accordingly

If you choose to self-host your source control (GitLab, Gitea, Bitbucket), you are accepting responsibility for its security. That means: two-factor authentication for all users, IP-based access restrictions, regular security updates, comprehensive audit logging, network segmentation to limit access from other parts of your infrastructure, and regular security audits. If you cannot commit to all of these, use a managed service.

6. Prevention is cheaper than response

The entire LinguaFlow incident could have been prevented by any of the following:

• A pre-commit hook running gitleaks or git-secrets to block credential commits
• Using AWS Secrets Manager or SSM Parameter Store instead of .env files
• IAM policies scoped to minimum required permissions (so an SES key literally cannot access S3 or IAM)
• GitLab push rules configured to reject commits containing strings matching secret patterns
• A periodic cloud security review that would have identified these gaps before an attacker did
• Engaging a virtual CISO to establish security governance and ensure these controls are implemented and maintained

Every single one of these measures is less expensive than the incident response investigation that became necessary when they were absent.

Suspect a Credential Compromise? Don't Wait.

Every hour without a professional investigation is an hour of lost evidence and compounding risk.

Our incident response team is available 24/7 for emergency engagements. We provide: immediate containment guidance, full forensic investigation, data impact assessment, regulatory notification support, and root cause analysis with hardening recommendations.

Published: March 2026 · Author: Alexander Sverdlov

This article is based on a real incident response engagement with details anonymized to protect the client. Company names, individual names, and certain technical specifics have been altered. The investigation methodology, technical recommendations, and lessons learned are presented accurately. This article is for informational purposes only and does not constitute legal or professional advice. If you are currently experiencing a security incident, contact a qualified incident response firm immediately.