# Secret exfiltration tripwires: detecting a known string in many encodings Research into tooling that, given a known secret string, generates its representation in many common encodings so that simple string or regex matchers can detect the secret in outbound traffic or logged output — regardless of which naive encoding a misbehaving agent uses. ## Summary - **No off-the-shelf tool does this exactly.** A search of GitHub, PyPI, npm, and security tool indexes (awesome-lists, awesome-honeypots, awesome-yara, awesome-canaries) found no project whose stated purpose is "given a secret, emit N encoded forms suitable for use as grep or regex patterns against outbound traffic." The capability exists in fragments across several adjacent categories, none of which compose into the complete picture. - **The closest existing answer is YARA's `base64` / `base64wide` string modifiers** (added in YARA 3.6). A single string declaration with both modifiers generates all three base64 offset permutations automatically, in plain and UTF-16LE form. Combined with separate `ascii` / `wide` variants, this covers five of the most common encoding forms with no Python needed. - **Secret scanners** (gitleaks, trufflehog, detect-secrets, ggshield) solve the inverse problem: detecting *unknown* secrets matching known patterns. They are not designed to scan for a specific known literal in N encodings. Gitleaks (v8.20+) and TruffleHog do perform multi-encoding decodes before running their detectors, but only as a preprocessing step — not as a way to produce N encoded forms for downstream matchers. - **Canary token services** (Thinkst Canarytokens, OpenCanary) are callback canaries: detection fires when the token itself is accessed and phones home. They do not scan outbound streams for encoded representations of the canary value. They address a different threat model. - **Enterprise DLP** products (Microsoft Purview, Symantec DLP/Broadcom, Nightfall, Cyberhaven) perform encoding-aware matching internally as a black-box feature. The capability is real but not exposed as an API and not available for use in a self-hosted container sidecar. Symantec DLP explicitly does not decode base64 or ROT13 in all inspection paths due to processing overhead concerns. - Rolling this in ~100 lines of Python is feasible and is probably the right path for bot-bottle v1. The limiting factor is not the encoding logic — that is straightforward — but the false-positive rate from common base64 alphabet collisions and the zero coverage against any re-encoding that involves a key (encryption) or destroys byte boundaries (packet splitting). --- ## Encoding catalog The following encodings are the realistic candidates for a naive agent performing unintentional or low-sophistication exfiltration. For each, the "FP risk" column notes the false-positive risk in a pattern matcher scanning general HTTP request bodies or log lines. | # | Encoding | Notes | FP risk | |---|----------|-------|---------| | 1 | Raw UTF-8 bytes (literal) | Baseline. Exact substring match. | Lowest | | 2 | Base64 standard (RFC 4648 §4) with `=` padding | Up to 3 offset variants depending on byte alignment at encode boundary; YARA covers all three with `base64` modifier. | Low–medium; base64 collisions are rare for 16+ char secrets | | 3 | Base64 without padding | Same byte content, trailing `=` stripped. Matches appear inside longer base64 blobs; needs anchor-free search. | Medium | | 4 | URL-safe base64 (RFC 4648 §5) | `+` → `-`, `/` → `_`. Required by many JWT and OAuth token formats. A separate encoding class from standard base64. | Low | | 5 | Base64 of UTF-16LE encoding | Re-encode the secret as UTF-16LE, then base64. Produces a different ciphertext than UTF-8 base64. YARA `base64wide` modifier covers this. | Low | | 6 | Base64 of UTF-16BE encoding | Same as above but big-endian byte order. Rare in practice but trivial to generate; include for completeness. | Very low | | 7 | Double base64 | Base64 applied twice. Some log-shipping pipelines or config-encoding layers do this naively. | Very low | | 8 | Hex lowercase (`0-9a-f`) | Every byte becomes two hex digits; trivially detectable with a fixed-length hex string search. | Low | | 9 | Hex uppercase (`0-9A-F`) | Same encoded value, different case. Separate regex needed if your matcher is case-sensitive. | Low | | 10 | Percent/URL encoding of all bytes | `%xx` for every byte. Browsers and curl typically only encode special chars; a misbehaving agent might encode all bytes. | Low in raw bodies; medium in URL query strings | | 11 | Percent encoding of non-ASCII bytes only | The common case. Only high-bytes and special chars encoded; printable ASCII runs verbatim. | Medium; overlaps with normal URL encoding | | 12 | Double percent encoding | `%` → `%25`, so `%41` → `%2541`. Common WAF bypass; appears in logged URLs after server-side decode. | Very low | | 13 | JSON string escaping | `"` → `\"`, `\` → `\\`, non-ASCII → `\uXXXX`. Relevant when a secret is serialized into a JSON payload body. | Medium; very common in request bodies | | 14 | HTML/XML entity encoding | `&`, `&#xXX;`, `&#DDD;`. Relevant for form POST bodies and SOAP/XML egress. | Medium in HTML context; low in JSON | | 15 | UTF-16LE raw bytes | Interleaved NUL bytes; `ABC` becomes `41 00 42 00 43 00`. Visible in PCAP or raw log hex dumps. YARA `wide` modifier covers this. | Low in text content; medium in binary/multipart streams | | 16 | UTF-32LE raw bytes | Four bytes per character. Unusual in web payloads but trivial for an agent to produce via Python's `encode('utf-32-le')`. | Very low | | 17 | ROT13 | Caesar cipher, shift 13, printable ASCII only. Shifts letters; digits and special chars unchanged. Weak obfuscation, cheap to detect. | Medium; common English words collide | | 18 | ROT47 | Extends ROT13 over printable ASCII range 33–126. Transforms digits and symbols too. Less collision-prone than ROT13. | Low | | 19 | gzip + base64 | `gzip(plaintext)` then base64-encode the binary output. Output is always recognizable by the `H4sI` base64 prefix for RFC 1952 gzip magic bytes. | Low; `H4sI` prefix is a cheap anchor | | 20 | zlib/deflate + base64 | `zlib.compress()` (DEFLATE with zlib header) then base64. Output starts with `eJy` or similar zlib magic prefix in base64. | Low; magic prefix detectable | | 21 | Leetspeak / character substitution | `e` → `3`, `a` → `@`, `o` → `0`, `i` → `1`, etc. No fixed mapping; generates many variants. High FP against non-secret content. | High; impractical to enumerate exhaustively | | 22 | Reversed bytes | Secret reversed character-by-character. Trivial, occasionally used as a confusion layer. | Low | | 23 | Space-separated characters | `SECRET` → `S E C R E T`. Defeats substring search; requires regex `S\s+E\s+C\s+R\s+E\s+T`. | Very low | | 24 | Null-separated characters (wide variant) | Like space-separated but with literal `\x00` bytes. Same as UTF-16LE for ASCII-only secrets. | Very low | | 25 | Base32 (RFC 4648 §6) | Used in TOTP seeds, some DNS exfil channels. Alphabet is `A-Z2-7`. Longer output than base64. | Low for secrets ≥ 10 chars | **Diminishing returns note.** Encodings 1–13 cover the vast majority of realistic naive-exfil vectors for an agent using standard Python or shell tools. Encodings 14–25 are worth including in a comprehensive scanner but individually contribute little marginal risk reduction. --- ## Adjacent category survey ### Secret scanners The major open-source secret scanners — gitleaks, TruffleHog, detect-secrets, and ggshield — all solve the *inverse* problem: given a body of text, find strings that look like secrets using entropy analysis or regular expressions trained on known credential formats. They are not designed to answer "does this payload contain *this specific known value* in any encoding?" The distinction matters. These tools look for patterns that match the structural form of, e.g., an AWS access key (`AKIA...`). They are not designed to take a user-provided literal and report all byte-equivalent encodings of it. That said, two of these tools have encoding-aware *preprocessing* steps that are directly relevant: - **Gitleaks** ([github.com/gitleaks/gitleaks](https://github.com/gitleaks/gitleaks)) added a `--max-decode-depth` flag in v8.20.0. When set to a non-zero value, it recursively decodes segments of the input before running regex detectors, supporting three encodings: hex, percent-encoding (URL encoding), and base64. The purpose is to find secrets that have been naively encoded before committing. This is functionally what a content-tripwire needs to do, but hard-coded to Gitleaks' own detector ruleset rather than user-supplied literals. The flag is off by default. - **TruffleHog** ([github.com/trufflesecurity/trufflehog](https://github.com/trufflesecurity/trufflehog)) decodes four encoding types before running its detectors: UTF-8, UTF-16, Base64, and Escaped Unicode (`\uXXXX` form). It also detects secrets hidden in archived and compressed files. A Truffle Security blog post from October 2024 documents this in detail ([trufflesecurity.com/blog/secret-scanning-encoded-and-archived-data](https://trufflesecurity.com/blog/secret-scanning-encoded-and-archived-data)). Same caveat as gitleaks: the decode step feeds the existing pattern detectors, not a user-supplied literal search. - **detect-secrets** ([github.com/Yelp/detect-secrets](https://github.com/Yelp/detect-secrets)) and **ggshield** ([github.com/GitGuardian/ggshield](https://github.com/GitGuardian/ggshield)) do not appear to have multi-encoding decode steps; they operate on the input text as-is. None of these tools expose a "match this literal in N encodings" API. The closest workflow would be to feed a custom gitleaks rule that matches pre-computed encoded variants, but that requires generating those variants externally (i.e., the exact gap this research note addresses). ### Canary token services Canary token services operate on a fundamentally different detection model and should not be confused with matcher canaries. **Callback canaries** work by embedding a unique URL or resource reference in a document, credential file, or environment variable. When an agent (or attacker) reads and uses the credential, the canary service receives an HTTP callback. The detection signal is the *access of the resource*, not the presence of an encoded form in an outbound byte stream. - **Thinkst Canarytokens** ([canarytokens.org](https://canarytokens.org) / [github.com/thinkst/canarytokens](https://github.com/thinkst/canarytokens)) offers AWS key canaries, Azure login canaries, PDF canaries, and many others. All rely on callback detection. A Canarytokens bypass issue ([github.com/thinkst/canarytokens/issues/36](https://github.com/thinkst/canarytokens/issues/36)) specifically documents that an attacker who extracts the canary value and uses it without triggering the callback URL (e.g., by sending the raw credential string to an external API over a non-canary channel) can bypass the detection entirely. This is the exact gap that encoding-aware content inspection would close. - **OpenCanary** ([github.com/thinkst/opencanary](https://github.com/thinkst/opencanary)) is Thinkst's self-hosted daemon that mimics network services (SSH, FTP, Telnet, HTTP, SMB, etc.) and alerts when they are probed. It is a network-layer honeypot, not an outbound content scanner. Detection is interaction-based, not encoding-aware content matching. - **IndicatorOfCanary** by HackingLZ ([github.com/HackingLZ/IndicatorOfCanary](https://github.com/HackingLZ/IndicatorOfCanary)) is conceptually the nearest to what is needed: it is a red-team tool for *detecting the presence of canary tokens inside files* before using those files, to avoid triggering callback alerts. It searches for known canary IoCs (callback domain patterns) in file metadata and content. It is the adversary-side mirror image — red team detecting canaries before they can be tripped — but it shows the art of the possible for encoding-aware document inspection. **The gap**: no canary service offers a "here is your secret; here are 12 encoded forms of it; ingest these into your egress scanner" API. ### Enterprise DLP Enterprise DLP products do perform encoding-aware content matching, but as an internal, closed-source capability: - **Symantec DLP (Broadcom)** ([broadcom.com](https://www.broadcom.com/products/cybersecurity/information-protection/data-loss-prevention)) An official Broadcom knowledge base article explicitly states that Symantec DLP is not able to inspect and alert on base64 and ROT13 encoded files in all inspection paths, citing processing overhead as the reason ([knowledge.broadcom.com/external/article/184415](https://knowledge.broadcom.com/external/article/184415/is-symantec-data-loss-prevention-dlp-abl.html)). This is a documented limitation, not marketing copy. - **Microsoft Purview DLP** ([learn.microsoft.com/en-us/purview/dlp-policy-reference](https://learn.microsoft.com/en-us/purview/dlp-policy-reference)) supports custom sensitive information types and trainable classifiers but the encoding-awareness of its content matching engine is not publicly documented at the rule-authoring level. No public API exists for generating encoded variant patterns. - **Nightfall AI** ([nightfall.ai](https://www.nightfall.ai/)) uses deep-learning classifiers rather than regex, with 100+ AI-based detectors. It offers a REST API that accepts arbitrary strings and files and returns findings. Its encoding-awareness is model-dependent and not configurable by the caller. No "user-supplied literal + encoding sweep" mode is documented. - **Cyberhaven** ([cyberhaven.com](https://www.cyberhaven.com/)) is notable for its data-lineage approach: it tracks data transformations (copy, compress, rename, convert) and ties exfiltration events to original sensitive files even after transformation. This is a more powerful model than pure byte matching but requires a full endpoint agent and cloud backend. Not suitable for a container sidecar. The enterprise DLP space confirms that encoding-aware detection is a solved problem at enterprise scale, but the implementations are either closed-source SaaS products, require endpoint agents, or are not configurable with user-supplied literals. ### Pentest / red-team encoding generators Several red-team tools generate many encodings of a payload, treating encoding as a *generation* problem rather than a detection problem. They are directly useful for producing the encoding catalog needed to build tripwire patterns. - **hURL** ([github.com/fnord0/hURL](https://github.com/fnord0/hURL)) is a command-line encoder/decoder supporting URL encoding, double URL encoding, base64, HTML entities, ASCII-to-hex, integer-to-hex, ROT13, and SHA family hashes. It is packaged in Kali Linux (`apt install hurl`). It does not produce a "all encodings of this string" output in one command — each encoding is a separate invocation flag — but the encoding catalog it covers aligns well with the practical catalog above. - **CyberChef** ([gchq.github.io/CyberChef](https://gchq.github.io/CyberChef) / [github.com/gchq/CyberChef](https://github.com/gchq/CyberChef)) is the GCHQ "cyber Swiss Army Knife," a browser-based tool with 400+ encoding/decoding/transformation operations. It can be scripted via the `cyberchef-node` npm package ([github.com/nicowillis/cyberchef-node](https://github.com/nicowillis/cyberchef-node)) to generate many encodings programmatically. The community recipe list ([github.com/mattnotmax/cyberchef-recipes](https://github.com/mattnotmax/cyberchef-recipes)) is a good reference for the encoding chains real attackers use. CyberChef is the best single reference for what an exhaustive encoding catalog looks like in practice. - **Burp Suite Intruder** (PortSwigger, commercial with community edition) has a payload processing rule chain in its Intruder module that can apply sequences of encoding transformations (URL, HTML, base64, ASCII hex, built-in strings) to a wordlist. Not scriptable outside Burp; primarily useful for interactive enumeration during a pentest. - **wfuzz** ([github.com/xmendez/wfuzz](https://github.com/xmendez/wfuzz)) supports encoder plugins (base64, urlencode, md5, sha1, double-urlencode, html, etc.) that can be chained with the `@` syntax in payload specs. It is a brute-force fuzzer, not a pattern generator, but its encoder catalog is a useful reference list. None of these tools emit "N regex patterns for detecting this secret in any of its encoded forms in an outbound stream." They are all generation tools for attacks, not detection tools for defense. --- ## YARA string modifiers — the closest existing answer YARA ([virustotal.github.io/yara-x](https://virustotal.github.io/yara-x) / [yara.readthedocs.io](https://yara.readthedocs.io/en/stable/writingrules.html)) has the most complete existing treatment of "match this string in multiple encoding forms" via its text-string modifier system. This was designed for malware detection in binary files and network captures, but the same logic applies to outbound traffic inspection. ### Available modifiers Four modifiers apply directly to the encoding problem: - **`ascii`** — match the string as raw ASCII/UTF-8 bytes. This is the default when no modifiers are specified. - **`wide`** — match the string in UTF-16LE form (each ASCII byte interleaved with a NUL byte). Designed for detecting strings in Windows PE binaries. - **`base64`** — generate all three base64 offset permutations of the string and search for any of them. The three permutations arise because base64 encodes 3 bytes at a time; depending on where a string starts within the 3-byte boundary, its encoding shifts by 0, 1, or 2 base64 characters. YARA computes all three at compile time and emits patterns for each, so the rule author does not need to pre-compute them. - **`base64wide`** — same as `base64`, but applied to the UTF-16LE form of the string. Covers the case where the secret was stored as a wide string (UTF-16LE) before being base64-encoded. Modifiers can be combined on a single string declaration. A rule that covers all four of these forms simultaneously looks like: ```yara rule secret_tripwire { strings: $s = "my-secret-value" ascii wide base64 base64wide condition: $s } ``` YARA will generate and search for (at minimum) seven patterns from this single declaration: raw UTF-8, raw UTF-16LE, and three base64 variants of UTF-8, and three base64wide variants of UTF-16LE. A fifth modifier, **`xor`** (added in YARA 3.8), searches for single-byte XOR obfuscated variants of the string across all 255 non-zero keys. The `xor` modifier cannot be combined with `base64` or `base64wide` in a single string declaration (it causes a compiler error). To cover both XOR and base64, two separate string declarations are required. **Custom base64 alphabets:** The `base64` and `base64wide` modifiers accept an optional 64-character custom alphabet string. This covers URL-safe base64 (`-_` substituted for `+/`) and any custom alphabets. ### Limitations of YARA for this use case - YARA does not natively cover hex encoding, percent encoding, JSON string escaping, gzip+base64, ROT13, or the other entries in the encoding catalog above. Those would require pre-computing the encoded forms externally and writing them as explicit hex-pattern strings in the rule. - YARA operates on files or byte buffers passed by the calling application; it does not natively hook network streams. Integration with a proxy or a log-scanning pipeline requires an application layer to call `libyara` or the `yara-python` bindings on each captured request body. - YARA's `base64` modifier has a documented minimum-length constraint: strings shorter than three characters cannot be base64-matched reliably due to the offset permutation math. This is unlikely to matter for real secrets but worth noting. ### DissectMalware/base64_substring The tool `base64_substring` ([github.com/DissectMalware/base64_substring](https://github.com/DissectMalware/base64_substring)) generates a YARA rule to find base64-encoded files containing a specific keyword, by enumerating all three offset permutations and emitting them as a YARA rule. This predates YARA's built-in `base64` modifier and is largely superseded by it, but the repository is useful as a reference for the permutation math. --- ## DIY sketch There is no off-the-shelf tool that takes a known secret and emits N patterns for outbound stream matching. The remaining question is how much work it would be to write one. A minimal `tripwire-encode` script in Python (~80–120 lines) would: 1. Accept a secret string on stdin or as a CLI argument. 2. Emit one encoded form per line (or a JSON object mapping encoding names to encoded values) for encodings 1–20 from the catalog above. 3. The encoding logic for each form is 1–4 lines of Python using the standard library (`base64`, `codecs`, `urllib.parse`, `gzip`, `io`); no third-party dependencies are required. 4. For the YARA output mode, emit a `.yar` rule with one string declaration per encoding (or use `ascii wide base64 base64wide` for the first four, then add explicit hex-string patterns for the remaining forms). A companion `tripwire-grep` script (~30–50 lines) would: 1. Accept the secret (or the pre-computed encoding list) and a stream on stdin. 2. Compile the encodings into a single `re.search` call or `bytes.find` loop. 3. Exit non-zero and print the matching line/offset if any form is found. For a proxy integration, the same encoding set can be compiled once at container startup and injected into a mitmproxy addon or a small filter script that wraps the proxy log output. **Rough size estimate:** the full encoding catalog (25 entries) in Python requires roughly 80 lines of encoding logic, 30 lines of CLI scaffolding, and 40 lines of grep/match logic. Total under 200 lines. **Important caveat on ROT13 and leetspeak.** These encodings have non-trivial false-positive rates for short secrets and for secrets containing common English substrings. If the secret contains only alphanumerics, ROT13 and ROT47 patterns should be included but flagged at warning level rather than error level in the output. **The honest limitation of this entire approach** is that it is effective only against naive, deterministic encodings where a single fixed plaintext always produces the same fixed ciphertext. Any exfiltration method that involves: - A secret key (AES, ChaCha, XOR with a non-trivial key), - Compression with a novel dictionary, - Splitting the secret across multiple messages or packets, - Steganography (hiding data in image pixels, audio samples, whitespace), ...will produce output with no detectable relationship to the input plaintext. For bot-bottle's threat model — an untrusted but not adversarially sophisticated agent that might accidentally or naively encode secrets before logging or posting them — naive-encoding detection covers the realistic concern. A deliberately adversarial agent with knowledge of its own monitoring is a different threat model entirely. --- ## Integration point in bot-bottle The natural choke point for encoding-aware content inspection in bot-bottle is the body of outbound HTTP/HTTPS requests, visible at the egress proxy layer. The egress-guard note documents smokescreen (Stripe, [github.com/stripe/smokescreen](https://github.com/stripe/smokescreen)) and squid as sidecar proxy options. Both can be configured to pass request bodies through a content-inspection hook. Smokescreen has an experimental MITM mode with detailed HTTP logging; squid supports `icap_service` ICAP integration for content inspection. Either could be extended with a small filter script that runs the encoded-form search against each captured body. **SSH is opaque to a proxy.** SSH connections carry encrypted payloads; the proxy cannot inspect their contents. Egress SSH (e.g., git push to a permitted remote) is outside the reach of this defense unless the container is configured to log git command arguments before they execute — which would catch a literal secret passed on the command line but not a secret embedded in a commit object or passed via an environment variable to a subprocess. For v1, this defense should be scoped explicitly to HTTP/HTTPS request bodies and to logged shell command output, and SSH egress should be called out as a gap. --- ## Limitations and bypass surface This defense catches secrets in transit only when the encoding is deterministic and pre-computable. The following techniques defeat it entirely: - **Symmetric encryption with a key** (AES-CBC, ChaCha20, XOR with a 16+ byte key). The ciphertext has no statistical relationship to the plaintext. Detection requires key recovery, which is out of scope for a content scanner. - **Splitting across messages or packets.** If a secret is split as `SECRET[0:4]` in one request and `SECRET[4:]` in another, no single- message content scanner will fire. Stateful stream reassembly would be required, which is a much harder problem. - **Compression with a novel or pre-shared dictionary** (LZ4, Brotli, zstd). Standard gzip/zlib produce recognizable magic-byte prefixes in base64, but a custom compression configuration can produce output that is entropy-high and format-unrecognizable. - **Steganography.** Encoding data into image LSBs, audio samples, whitespace-as-binary in a text file, or Unicode homoglyphs. The output looks like a legitimate artifact; no regex will find it. - **Hash-chained exfiltration.** Sending the SHA-256 of secret fragments in sequence; a receiver who knows the secret can verify, but no string scanner can reconstruct the original. - **Protocol-level encoding not visible at the HTTP body layer.** DNS exfiltration (encoding data in query subdomains), ICMP covert channels, or TCP sequence number steganography all operate below HTTP and are not intercepted by an HTTP proxy at all. The egress-guard note covers DNS separately via a controlled resolver. In the bot-bottle context, the primary realistic concern is an agent that naively embeds a secret in a log line, a curl argument, a JSON body, or a shell heredoc without specifically intending to obfuscate. All of the above bypass techniques require deliberate, adversarially-motivated engineering from a process that is assumed to be untrusted but not intentionally malicious. The encoding-aware tripwire is an appropriate control for this threat model, provided its scope is communicated clearly.