A Flink-based "FileChunkCombiner" pipeline in the TSG OLAP stack (galaxy/tsg_olap/file-chunk-combiner) reassembles file chunks parsed from proxy and session metadata (ParseProxyFileMetaFlatMapFunction, ParseSessionFileMetaFlatMapFunction) captured off monitored traffic, then persists whole reconstructed files to HBase and object storage (an "HosSink" and an OSS sink), confirming TSG performs full file-content extraction and durable storage from intercepted sessions, not just flow/metadata logging.
File tree: src/main/java/com/zdjizhi/FileChunkCombiner.java, src/main/java/com/zdjizhi/function/map/ParseProxyFileMetaFlatMapFunction.java, src/main/java/com/zdjizhi/function/map/ParseSessionFileMetaFlatMapFunction.java, src/main/java/com/zdjizhi/sink/HBaseSink.java, src/main/java/com/zdjizhi/sink/HosSink.java, src/main/java/com/zdjizhi/sink/OssSinkByCaffeineCache.java
Defense implications
- Any file transferred in plaintext, or through a session a TSG deployment can decrypt via MITM, is subject to full reconstruction and durable storage, not just flow-level logging — reinforces that end-to-end confidentiality must survive interception, not merely transport-level obfuscation.
Related findings
TSG/CM ships with pre-built, first-class 'Learning Object' entries specifically for Freegate (Object ID 18) and Psiphon3 (Object ID 19), plus a generic 'Top Server IP' object (ID 20) -- default product features, not customer-commissioned custom signatures. The Psiphon3 object auto-learns and dynamically updates a live blocklist that reached roughly 70,000 IPs at one deployment before a database issue temporarily dropped it to ~50,000.
Crash-dump stack traces reveal TSG's core packet-processing engine (sapp) architecture: a custom multi-threaded C engine using 'marsio' as the DPDK-style packet-I/O driver, a plugin system for protocol handlers (confirmed: plug/protocol/http/http.so), and a libdocumentanalyze component that actively decompresses gzip content and parses ZIP/document formats found inside HTTP bodies -- i.e. inspection goes beyond headers into reconstructed application content.
certstore's own commit history documents its transparent-TLS-MITM mechanics directly: it writes the client's observed SNI into the SAN field of the leaf certificate it mints on the fly, supports ECC issuance (secp192r1/secp256r1) for those forged certs, and reads its Trusted/Untrusted decryption-keyring configuration from MAAT's DECRYPTION_KEYRING table -- confirming the interception pipeline end-to-end: client SNI in, matching forged certificate out, gated by MAAT-synced keyring policy.
TSG implements heuristic tunnel-protocol detection distinguishing Teredo (IPv6-over-UDP, default port 3544) from GTP tunneling based on UDP payload inspection, used to decide 'innermost' session attribution for firewall matching.
A 2020 MESA Lab monthly report describes building an encrypted-DNS gateway device and a self-hosted DoH server that was wired directly into a live "TSG platform WAN net interface", alongside a passive DoH-service-discovery pipeline that found 39 new DoH-serving domains in a single day of backbone ("科技网") traffic; a related note documents actively probing known DoH-serving IPs' upstream resolvers via a proxy network across 196 countries to check which still function.
A production feature spec ("加密协议JG") requires detection-and-control capability for QUIC, ESNI, and ECH, with a per-connection log field c_encrypt_type explicitly distinguishing ClientHello encryption (0=unencrypted, 1=ESNI, 2=ECH) and a DF_QUIC_REGION rule table matching QUIC by SNI. The system computes a per-rule "CT" (穿透/penetration) rate from paired mirror-vs-functional RST-packet logs (TF-RST-LOG / TF-MIRROR-RST-LOG) to grade each block rule's effectiveness, and runs continuous active-verification probes against its own rules, deliberately varying the probe 4-tuple each run "to ensure it doesn't hit the blacklist" so the verification traffic itself reaches the target.