NTC_HTTP_COLLECT ('http_url_discovery', in the codebase since at least 2014-2018 per version tags) passively harvests every HTTP request URL and Referer header seen in monitored TCP traffic and streams each hit (with source/destination IP:port, transport proto, extracted domain, capture-node IP, and found_time) to Kafka topic 'AIM'. A companion filter list (http_url_filter.conf) suppresses roughly 100 file-type suffixes — not just static assets like .jpg/.css/.js but also office, archive, and executable extensions (.doc/.docx/.pdf/.zip/.rar/.exe/.dll) — from this particular URL-discovery feed, consistent with those downloads instead being routed to a separate full-object capture pipeline (see the ObjectScanner finding from this same batch).
MESA_load_profile_string_def(g_item.profile_path, PLUGIN_NAME, "kafka_topic", g_item.topic_buf, sizeof(g_item.topic_buf), "AIM");
Defense implications
- Any plaintext HTTP request URL/Referer your client sends is logged into a bulk feed unless its path ends in one of ~100 filtered suffixes; transports that never expose a plaintext HTTP Host/URL/Referer on the wire (TLS-wrapped, or non-HTTP entirely) bypass this specific dragnet.
Cited artifacts
mesalab_git—PanGu/ntc_http_collect.bundle(git-commit-log)
Provenance pointers only — no leak content is rehosted. Raw material: Enlace Hacktivista.
Related findings
An internal TSG operations/troubleshooting manual lays out TSG's full traffic pipeline (NIC -> mrzcpd/marsio capture driver -> sapp DPI engine -> firewall/proxy(KNI->TFE)/active-defense/WAN-NAT policy branches) and shows engineers using maat_redis_tool to pull the live Redis-synced blocking policy tables (TSG_SECURITY_COMPILE, TSG_OBJ_IP_ADDR, TSG_OBJ_APP_ID) and filter them by numeric policy/object ID to debug why a block rule isn't firing.
The internal 'MAAT网络流处理配置统一描述框架' engineering manual (v3.1.20, author 郑超, 2021) documents MAAT's Redis-synced rule-compilation framework and its RuleScan/Hyperscan-based pattern-matching engine (libmaatframe.so / librulescan), and records that RuleScan's fast-scan feature caused a production outage on 2019-03-19 and has been disabled ever since.
An internal design note for a 'Shadowsocks traffic parsing/restoration module' describes decrypting captured Shadowsocks payloads back to the original HTTP request/response, given a known pre-shared key (AES-256-CFB, MD5-derived key, IV embedded in the stream).
A dedicated internal "运营商前端分析团队" (Carrier Front-end Analysis Team) produces weekly, carrier-specific traffic-analysis reports explicitly titled "新疆移动流量分析报告" (Xinjiang Mobile Traffic Analysis Report) and "新疆联通流量分析报告" (Xinjiang Unicom Traffic Analysis Report), with instances dated from at least July 2022 through March 2024. One instance (Xinjiang Mobile, 2023-05-15) reports total carrier bandwidth of 4.4Tbps (1Tbps access bandwidth), 27.34PB of weekly throughput, 123 distinct identified application-layer protocols, and a section (4.1, pages 17+) explicitly tracking server IP/location for 28 named overseas apps (BBC, Discord, ESPN, Facebook, Gmail, Google, Hulu, Instagram, Netflix, Reddit, Skype, Telegram, Twitter, Uber, WhatsApp, Wikipedia, YouTube, Zoom, etc.).
A 2018 MESA Lab monthly report describes work on the "GPS"/"先导" (Pioneer) active-probing projects: adding IPv6 scanning support and simultaneous 3-domain scanning to MAAT, alongside active-probing experiments (nslookup/dig/nmap) and root-DNS-server BGP-anycast measurement -- indicating MAAT's rule engine is paired with an active network-scanning subsystem, not purely passive/inline matching.
An internal 'MAAT Configuration Description Manual (String)' fully documents MAAT's rule-compilation model: per-field string matches (substring/prefix/suffix/exact/regex/AND-of-substrings/offset-anchored substrings) grouped into up to 8 AND/NOT clauses per compiled rule (conjunctive normal form), each carrying an action code (0=block, 1=monitor-only, 2=whitelist), blacklist and logging flags, and a floating-point execution-order field for safe rule reordering, plus the C scanning API (Maat_full_scan_string) and the file/JSON formats used to push rule updates to production.