# Railway environment-variable breach - defensive detection pack (2026-10)

**Incident:** Railway disclosed on 2026-10-21 (no CVE) that an unauthorised party may have accessed encrypted customer
environment variables and, for some, obtained decrypted contents (API keys, database and private registry credentials).
Railway believes it is part of a series of attacks on platform companies aimed at compromising AI inference providers.
Railway rotated all bucket credentials, invalidated all Railway API tokens (2026-10-22 04:17 UTC), invalidated OAuth
sessions, reset GitHub credentials (reconnect required) and paused builds (staged restore in progress).

Primary sources (affected vendor):
- https://blog.railway.com/p/security-disclosure-wednesday-october-21st
- https://status.railway.com/incident/QH8L8B3E
- https://blog.railway.com/p/security-disclosure-recommendation-check-for-anomalous-aws-access
- https://blog.railway.com/p/security-disclosure-recommendation-check-for-anomalous-gcp-access
(The status page also links guides for OpenAI, Anthropic, OpenRouter, Grok, Stripe, GitHub and Postgres/MySQL/Mongo/Redis rotation.)

**Railway published no IOCs** (no IPs, domains, hashes, user agents, exposure start time). Everything here is behavioural.
The window start 2026-10-14 is a chosen hunting assumption, not a Railway-published date.

## Contents
| File | Purpose |
|---|---|
| `railway-hunting.kql` | Sentinel/Defender hunts (AWSCloudTrail, GCPAuditLogs, DB sign-ins). Parameters at top. |
| `sigma/*.yml` | 8 Sigma rules (4 AWS CloudTrail, 4 GCP audit). Validated with `sigma check`. |
| `check-railway-exposure.sh` / `Check-RailwayExposure.ps1` | Read-only: list Railway projects/envs/services and NAMES of secret-like variables; write rotation CSV. |
| `check-aws-new-iam.sh` | Read-only: IAM users/access keys created since `SINCE` (default 2026-10-14). |
| `wallet-check.md` | Checking transfers to a researcher-reported address. |
| `tests/` | Offline tests (shellcheck, mocked bash runs, pwsh parse, Pester, sigma check). |

## Reactive steps (do now)
1. Rotate **every** Railway variable (Railway's recommendation), prioritising cloud keys, AI provider keys, DB, registry, Stripe, GitHub.
2. AWS: deactivate exposed keys, create replacements, review CloudTrail per key **in every Region** for
   `RunInstances, CreateUser, CreateAccessKey, AttachUserPolicy, CreateRole, PutBucketPolicy`, check the credential report and Cost Explorer.
3. GCP: disable exposed SA keys (`gcloud iam service-accounts keys disable`), disable the SA if immediate cut-off needed (tokens live up to 1h),
   review audit logs for new SAs/keys, IAM bindings (esp. Token Creator), instances, buckets, project SSH keys, impersonation; check Vertex AI/GPU billing.
4. Check AI provider dashboards for unexpected usage; revoke keys.
5. Create a **new** Railway API token (old ones are invalid), redeploy buckets, reconnect GitHub.
6. If wallets/seed material were in variables, see `wallet-check.md`.

## Proactive
- Use static outbound IPs on Railway and allowlist them in cloud IAM conditions / DB firewalls; set `KnownGoodIPs` in the KQL.
- Prefer short-lived credentials (OIDC / workload identity) over long-lived keys; least privilege; spend caps on AI providers.
- Enable CloudTrail data events for Bedrock and GCP Data Access logs for `aiplatform.googleapis.com` (otherwise spike hunts see nothing).

## Usage
```bash
export RAILWAY_TOKEN=...          # NEW account/workspace token; or `railway login && railway link`
./check-railway-exposure.sh       # OUT=path.csv to change output
pwsh ./Check-RailwayExposure.ps1 -OutFile checklist.csv
SINCE=2026-10-14T00:00:00Z ./check-aws-new-iam.sh
bash tests/run-tests.sh           # needs shellcheck, jq, sigma-cli, pwsh (+Pester)
```
Exit codes: `0` nothing found, `1` action needed, `2` unknown/partial, `3` error (incl. no credentials).
Variable values are fetched into memory only to read the key names and are never printed or written.

## Limits
- **The KQL has not been run against a live tenant.** Column names (e.g. `GCPAuditLogs` request/auth fields, DB log tables) vary by connector; validate first.
- Sigma rules are broad behaviour matches; baselining/thresholds belong in the SIEM. Converted queries were not tested on live data.
- Railway checker: API mode uses Railway's public GraphQL API (`projects`, `variables`); projects in workspaces the token can't see are missed.
  CLI mode only covers the linked project/service (reported as partial, exit 2). Name matching is heuristic: rotate everything anyway.
- Project-scoped tokens (`Project-Access-Token` header) are not supported; use an account/workspace token.
- AWS checker covers IAM users/keys only, not roles, policies or other Regions' resources.