Keyword filtering

The internal 'MAAT网络流处理配置统一描述框架' engineering manual (v3.1.20, author 郑超, 2021) documents MAAT's Redis-synced rule-compilation framework and its RuleScan/Hyperscan-based pattern-matching engine (libmaatframe.so / librulescan), and records that RuleScan's fast-scan feature caused a production outage on 2019-03-19 and has been disabled ever since.

A MESA Lab monthly report describes a 'web-proxy' engineering task that modified the open-source Ultraviolet web-proxy project to add keyword- and specific-URL-blocking, alongside deployment work (certificate issuance, redirect handling) and a document listing which sites the proxy is permitted to relay; the purpose (internal filtered access vs. a broader capability) is not stated in this excerpt.

An internal MESA Lab research note directly measures and compares China's ('CN') HTTP censorship middlebox against Russia, India and an unlabeled 'HZ' system, plus open-source Snort2/Snort3/Suricata: China is characterized as inspecting Host- and keyword-based triggers (example trigger given: a request containing the parameter 'q=ultrasurf') across ALL ports rather than just 80/443, responding with a triple RST or an extra RST+ACK; an 8-technique HTTP-request-mangling evasion comparison table credits China's middlebox as vulnerable only to request-line whitespace insertion and HTTP-version tampering, fewer categories than the other three systems tested.

pg_valve_deal.cpp / pg_valve_main.cpp source shows the 'valve' (pangu_valve) component implements a MAAT-fed, hierarchical (region/group -> dataset -> domain-ID) rule-dispatch engine: each incoming rule line carries a keyword field plus expr_type/match_method/is_hexbin matching flags, an ADD/DEL action, and a percentage-based 'Droprate' throttle parsed out of a semicolon-delimited region string (DOMAIN_ID=...;config_id=...;Droprate=...) — i.e. a keyword/domain-based engine that can partially throttle traffic, not just binary-block it.

An internal 'MAAT Configuration Description Manual (String)' fully documents MAAT's rule-compilation model: per-field string matches (substring/prefix/suffix/exact/regex/AND-of-substrings/offset-anchored substrings) grouped into up to 8 AND/NOT clauses per compiled rule (conjunctive normal form), each carrying an action code (0=block, 1=monitor-only, 2=whitelist), blacklist and logging flags, and a floating-point execution-order field for safe rule reordering, plus the C scanning API (Maat_full_scan_string) and the file/JSON formats used to push rule updates to production.

Internal MAAT engineering Q&A notes detail the rule engine's filter hierarchy: a 'region' config matches on keyword / regex / IP / extended-IP(CIDR) / numeric / file-digest / text-similarity / FQDN fields; regions roll up via AND/OR into 'group' configs; groups roll up into an 8-clause-max 'compile' config in conjunctive normal form. Rule config is pushed via a Redis single-primary/multi-replica tree, and MAAT itself is not distributed — each production front-end box runs its own single MAAT instance that receives pushed config and calls the shared rulescan library to execute the actual scan.

A weekly operational report documents newly-added blocking rules for 25 mobile apps (18 hosted overseas, mostly on Hong Kong-region Azure/Alibaba/Tencent cloud IPs), where analysts extract each app's domains, IPs, and distinctive URL paths (e.g. '/tigase/getLastChatList', '/user/getUserMoney') and log 'system disposition: newly added blocking action' for every one; matching raw keyword-object exports using the identical URL-path style confirm these hand-extracted paths are loaded directly as live filter-list entries in production.

The AppSketch/MAAT signature system contains a systematically dated, patch-versioned catalog of FQDN-based detection signatures for dozens of individually named commercial VPN apps (e.g. V2VPN, V2Netvpn, Turbo VPN, LetsVPN, AdGuard VPN, Avira Phantom VPN, TouchVPN, FinchVPN, Opera VPN, Ultrasurf VPN, GoFly VPN), each modeled as an 'app' object in category 'networking' / subcategory 'tunnels' whose default deny_action drops the flow while sending both an ICMP-unreachable and a TCP RST to the client. At least 35 such distinct VPN-app signature objects appear in this batch alone, dated July-November 2024, with repeated 'Patch0N' revisions to the same app as its domains rotate.

Commit history for the K18 (Kazakhstan) argus-ntc console reveals its concrete feature set: a scheduled "网页关键字定时器" (webpage-keyword timer/scheduler) for keyword filtering, ASN/IP block-list configuration pages, a "BGP泛收" (BGP wide-collection) page, an SSL-interception config toggle, a file-scanning results page with MALWARE TYPE/MALWARE NAME columns, app-identification entries including a WhatsApp rename, and a VoIP business-config approval workflow, with blocking actions relabeled from "阻断" (block) to "封堵(丢弃)" (interdict/drop).

A ~11,000-row categorized domain blocklist (CSV: domain, numeric category code, Chinese label) tags domains under content categories including 涉赌 (gambling-related) and 涉黄 (pornography-related), including combined multi-category rows (e.g. '涉黄;涉赌'), consistent with a keyword/content-category filter list feeding a DPI/domain-blocking rule table such as MAAT's.

A 616-entry keyword list of URL/API paths (e.g. "index.php/index/ajaxlogin", "api/User/login", "user/signup") targets login/registration endpoints across a large number of distinct web backends, demonstrating HTTP-path-level keyword-filtering signature capability at scale; the specific target set looks oriented at domestic app/site fingerprinting (loan/gambling-adjacent path names appear) rather than confirmed circumvention-tool targeting, so attribution of intent is uncertain.

A TSG management-UI format-reference doc enumerates every native blocklist/policy-object import type: IP (with CIDR/range and port-range), FQDN, URL, Account (email address), Subscriberid, and Keywords/HTTP-Signature (matched against User-Agent, Cookie, Set-Cookie, or Content-Type headers, with optional hex-encoded byte patterns) — confirming Subscriberid is a first-class, directly importable policy-matching key alongside network-layer identifiers.

An exported keyword/domain filter-list object contains 48,874 rows of domain-blocking entries, with each domain listed twice as both an exact-match '$domain' pattern and a wildcard '*.domain' subdomain pattern, demonstrating the scale of a single production domain-blocklist object within the platform.

An internal "business log loading interface" spec enumerates the platform's full censorship/surveillance taxonomy as three parallel log streams (管控/blocking, 监测/monitoring, and 一般/general) each covering the same roughly 13 categories -- IP blacklist, DNS spoofing, URL, website, specific-certificate, webpage-keyword, email-keyword, FTP-keyword, search-term, email, VPN, instant-messaging, and social-app -- fed via HTTP POST/Avro to a "front-end big data platform," with source/destination geolocation fields explicitly keyed to a carrier-supplied "疆外" (outside-Xinjiang) IP-location database.

A 616-row keyword/URL-path signature list targets login/register/wallet/loan API endpoints (e.g. /api/User/login, /api/Qianbao/tixian [wallet-withdraw], /api/Order/loanuse) characteristic of scam-loan and phishing apps, demonstrating the same URL-path keyword-filtering rule format used for censorship targets is applied to a distinct fraud-detection ruleset -- direct primary evidence of the keyword-filtering rule format itself.

A TSG/sapp signature bundle dated 2024-06-17 defines a single 'Express VPN_Patch01' application (app_id 15190) matched by three independently OR'd signature layers -- an FQDN blocklist (expressvpn.com and ~7 domain variants), a destination-IP blocklist of roughly 150 addresses, and a raw UDP payload byte-pattern match anchored by offset/depth against ExpressVPN's Lightway protocol handshake bytes -- tied to an explicit enforcement action of dropping matched traffic.

A raw keyword/domain blocklist export (1,052 rows) includes wildcard/suffix-matched entries such as '*.gfwu.cn', '$gfwu.cn', '*mxstsg.com', and '*90tsg.com' alongside apparently unrelated commercial domains -- a concrete instance of the keyword-filter-list format used to feed TSG's matching engine, though this excerpt alone does not establish the list's overall purpose (anti-fraud vs. political/content blocking).

A production TSG maat.conf shows the MAAT rule-matching engine running four parallel instances (STATIC, DYNAMIC, APP_SIGNATURE_MAAT, CAPTURE); the APP_SIGNATURE_MAAT instance is explicitly configured against app_sketch_tableinfo.conf / app_sketch_maat.json -- directly confirming MAAT is the execution engine underlying the AppSketch signature system -- and is tagged with an ACCEPT_TAGS datacenter value 'xjlhs', indicating this specific instance is scoped to a Xinjiang deployment.

Beyond ExpressVPN, TSG/sapp's signature pipeline maintains individually dated, continuously-updated detection rules (IP, FQDN, and in some cases WireGuard-specific payload signatures) for dozens of distinct named commercial VPN products spanning mid- through late-2024 -- including FlyVPN, VPNHero, RapidVPN, JourneyVPN, quarkVPN, AwardVPN, VPNLite, AdGuardVPN, WolfVPN, DelightVPN, VPNlat, VPNBrazil, NotVPN, VPNTurkey, jumpjumpVPN, BetternetVPN (with a dedicated WireGuard variant), TurboVPN, Psiphon 3, CyberGhost (WireGuard-specific), NordVPN, VPN Unlimited, and SuperUnlimitedVPN -- indicating a long-tail, actively-maintained blocklist covering niche/regional VPN apps, not just a handful of major providers.

A MESA lab measurement of national censorship middleware (China, Russia, an unnamed "HZ" censor, and India) finds China's GFW performs both Host-header-based and keyword-based filtering across ALL ports, not just HTTP/HTTPS 80/443, and blocks matching connections with 3x injected RST packets or an additional RST,ACK -- a broader and more aggressive posture than the other three censors measured, none of which monitor all ports.

K18_NTCS_WEB/NTC (git.mesalab.cn) is the Java/Spring web console for Kazakhstan's (K18) National Traffic Control System. Its domain model implements per-protocol keyword filtering (App/ASN/DNS/FTP/Mail/P2P/SSL keyword configs), an HTTPS proxy-MITM object (PxyObjTrustedCaCert.java), and explicit content-manipulation templates for HTTPS Redirect and Replace (complex/IP-based) plus Hijack/Insert actions, all managed through this customer-facing K18 control panel.

A repo path-labeled "K18_NTCS_WEB" (backend service "argus-service", originally developed as "maat_service") defines per-protocol raw-log and business-rule types spanning HTTP, SSL, DNS, SSH, FTP, Mail, P2P, and VoIP, and the VPN/tunnel protocols PPTP, L2TP, IPsec, and OpenVPN, plus a dedicated keyword-based URL log (NtcKeywordsUrlLog) and a RADIUS collection log (NtcCollectRadiusLog) — showing the K18 (Kazakhstan)-labeled monitoring platform logs keyword-hit URLs and carrier RADIUS data alongside full protocol-specific traffic logs.

The GitLab group itself is named "K18_NTCS_WEB" (K18 = Kazakhstan), and its "nfs" web app implements per-protocol keyword-filter configuration classes (App/FTP/Mail/P2P/SSL keyword configs), an OpenVPN IP-list config, RADIUS-based logging/reporting entities, MAAT rule-sync beans, and an explicit "IP spoofing" business feature with its own "PXY仿冒地址池" (proxy spoofed-address pool) and dedicated policy-log support -- the single strongest piece of evidence in this batch tying named keyword-filtering plus IP-spoofing capabilities directly to the Kazakhstan deployment.

A ~6,000-line undifferentiated URL classification/whitelist dataset in the leak includes the URL 'https://raw.githubusercontent.com/getlantern/lantern' placed directly adjacent to known GFW-circumvention and Google-mirror sites (pac.itzmx.com, www.guge.xxx, g.alexyang.me, www.meiguge.com), suggesting Lantern's own GitHub source repository is catalogued within a circumvention-tools/mirror-site cluster of this URL database rather than filed as an ordinary developer/tech reference.

A 616-row "Keyword" list of URL path fragments (login/register/loan-related API endpoints such as api/Order/loanuse, api/index/jiekuang, plus generic JS/CSS resource paths) with at least one entry annotated "（暂停）" (paused), demonstrating that MAAT/AppSketch signature curation extends to URL-path-level keyword matching for a specific app category (apparent online-lending/finance apps) with an active edit/curation workflow (entries can be individually paused).

Internal MAAT engineering specification (v3.1.20, MESA Lab, revision history spanning 2014-2021) documents the config schema underlying sapp's rule-matching engine: per-rule action types (0=block/阻断, 1=monitor/监测, 2=whitelist/白名单), string/regex/IP/numeric/digest match types, and a geographic+ISP "tag" targeting system that lets the identical rule set be selectively activated per city district and carrier (e.g. Beijing/Chaoyang + China Telecom vs. Shanghai/Pudong), enabling narrowly-scoped rule rollout/testing before wider deployment.

The 2019 chenguanlin/gie_server repo implements a standalone "Gram Index Engine" (GIE_server) service that links directly against libmaatframe.so and bundles Maat_rule.cpp/.h, indicating an n-gram/substring pre-indexing layer sits alongside MAAT's Hyperscan/Rulescan pattern-matching engine for DPI signature matching.

MAAT's compiled rule format includes explicit HTTP URL/request-body/response-body matching rule categories (NTC_HTTP_URL, NTC_HTTP_REQ_BODY, NTC_HTTP_RES_BODY) and email content/header matching categories (NTC_MAIL_BODY, NTC_MAIL_HDR), and its scanner backend supports two swappable regex engines -- a Hyperscan adapter (adapter_hs) and a 'Rulescan' adapter (adapter_rs) -- corroborating this corpus's existing understanding that MAAT falls back from Hyperscan to Rulescan above roughly 50K rules.

A MAAT/rulescan crash-debugging log shows the engine's getCfgId() lookup called with domain=".twitter.com", table_id=12, type="SNI", confirming MAAT performs SNI-field domain matching against a configured blocklist table (table_id 12 = SSL/SNI domain tables DF_SSL_REGION/DJ_SSL_REGION; table_id 22 = HTTP URL tables DF_HTTP_URL/DJ_HTTP_URL seen in the same crash series), and that the underlying librulescan.so string-scan library is prone to native crashes (SIGABRT) under real traffic.

The PanGu/mesa_plug bundle (2019) shows the plugin framework's config layout with an explicit http_url_filter.conf plugin config and a dedicated ntc_bgp_plug submodule alongside asn_tableinfo.conf and IP-deny table config, confirming both URL-based content filtering and BGP-feed-driven ASN/prefix blocking are first-class, separately-configured plugins within the DPI stack (not just SNI/IP list matching).

An internal MITM-attack training/research presentation demonstrates a live keyword-filtering test through a trusted-root-CA MITM proxy (mitmproxy) against HTTPS traffic to scholar.google.com: after installing the MITM root certificate on the client, searches containing a prohibited keyword (drugs, 毒品) fail to load while normal searches succeed, in both forward-proxy (via an Aliyun VPS relaying through the circumvention tool Clash) and transparent-proxy configurations. The same deck describes a separate built tool, "video_server," that MITM-intercepts and downloads WeChat Channels (微信视频号) video content, and covers SSH/RDP MITM techniques (including the SSH Terrapin attack) more broadly.

A crash/perf bug report reveals Geedge's MAAT rule engine (libmaatframe.so, using Intel Hyperscan for literal/regex matching) running with 545,441 FQDN blocking rules and 404,141 IP address blocking rules loaded in a single instance as of 2024; loading 500,000 new FQDN rules via the full Hyperscan path took ~95 minutes, addressed by an automatic fallback to a second engine ("Rulescan") above 50,000 rules.

Root-caused an Ethiopia SNI-block-failure incident to the detection mechanism itself: APP_SKETCH's FQDN-scanning module -- which exists specifically to identify Psiphon3 and Freegate via top-N SNI matching -- was CPU-expensive enough to saturate cores, triggering sapp's fail-open DDoS bypass and letting some target connections through uninspected. A signature-structure fix roughly doubled throughput (33K/s to 73K/s new connections) and disabling the fail-open bypass restored reliable blocking.

A Fujian-site performance incident (single-core CPU pinned >90%, packet loss) traced to a large keyword-object list (~400 hex-encoded keywords, described as connected-vehicle-related) being scanned per-packet by components named libmaatframe, librulescan, and fw_http_plug. Removing unused keyword objects resolved the packet loss.

Fujian's anti-fraud "big screen" dashboard treats circumvention-tool use as a first-class flagged category alongside fraud: warning reasons include "visited a fraud-linked site, used an accelerator" and "new user of AiJiaSu" (a named accelerator app), each tied to the individual's phone number and refreshed ~every minute.

The Fujian (domestic China) deployment uses a punycode/IDN-aware keyword-filtering policy object named '中文涉诈域名' (Chinese fraud-related domains) with wildcard item matching; a bug ticket shows the policy engine failing to match an IDN punycode domain correctly against this object, revealing the underlying components: a 'verify-policy' microservice, and libraries 'libmaatframe' and 'librulescan' handling rule evaluation.

Geedge fingerprinted several V2Ray-based VPN apps by their fixed API/bootstrap domains rather than protocol characteristics — blocking "V2 Pro" via pro.mucacherry.org, "V2 VPN" via v2.mucacherry.org, and "V2Net VPN" via api.v2net.live plus 5 rotating free-node domains (free1-5.v2n3t2.online / v2n3t.online) the app uses to fetch node lists.

PanGu defines explicit business/service IDs for webpage keyword blocking (0x28) and keyword monitoring (0x98), with a dedicated NTC_KEYWORDS_URL_LOG log table storing the URLs recovered from keyword-hit traffic -- confirming keyword-based content filtering is a named, first-class TSG business rule with its own audit log, separate from domain/IP/SNI blocking.

The same import-format reference document specifies wildcard-keyword bulk-import formats spanning IP/CIDR ranges, FQDN, URL, email Account, Subscriber ID, raw hex-encoded byte keywords (isHexbin flag), APN, and — notably — "Http Signature" imports scoped to a specific HTTP field (User-Agent, Cookie, Set-Cookie, or Content-Type), letting an operator add an arbitrary HTTP-header-based match/block rule via a simple wildcard text import rather than custom engineering per rule.

A captured 'getcatlist' API response enumerates an 83-category licensed URL-classification taxonomy (BrightCloud/Webroot-style category names and IDs) that includes a dedicated 'Proxy Avoidance and Anonymizers' category (catgroup 'Security') alongside categories like 'Government' and 'Hacking' -- showing the platform layers a licensed third-party URL-reputation feed on top of AppSketch's custom signatures specifically to catch circumvention/anonymizer traffic by category rather than only by named signature.

The same TSG QA suite includes purpose-built "hijack" test fixtures for at least seven content types (APK, EXE, GIF, HTML, JPEG, PNG, SVG) alongside JS/CSS insertion scripts and Lua replace/insert manipulation scripts, and its UI/API test cases reference a "Manipulation" policy object with replace/hijack/deny(drop) actions triggered by "Keywords" objects inside "http_signature" matching and an explicit "no_sni" traffic-matching condition; separate pcap test fixtures (esni.pcap, ssl_ech_false.pcap, ssl_no_sni.pcap, ssl_tls13.pcap) show this is specifically regression-tested against ESNI/ECH and SNI-less TLS 1.3 traffic.

Geedge runs an ongoing, largely manual weekly signature-extraction program against commercial VPN apps (LetsVPN, Betternet, VPNLITE, and a ~80-app batch), combining server-IP lists, FQDN patterns, and payload/SNI/Host/User-Agent signatures, tested for blocking efficacy and false-positive rate before rollout. VPNLITE was fingerprinted via TCP direct-connect on ports 18000/3320/8099 plus TLS to randomized .xyz/.info subdomains on Cloudflare ranges (104.21.0.0/16, 172.67.0.0/16).