zhanghongqing/knowledge-log is a Flink ETL pipeline that builds relationship graphs from DNS and connection logs (Ip2IpGraphProcessFunction, DnsGraphProcessFunction/DnsRelationProcessFunction) and persists them to ArangoDB (a graph database) plus ClickHouse, indicating infrastructure for graph-based flow-correlation analysis (e.g. clustering related proxy/VPN infrastructure by IP-to-IP or DNS relationships) separate from the per-packet DPI engine.
REPO PATH: zhanghongqing/knowledge-log.bundle; src/main/java/com/zdjizhi/etl/connection/Ip2IpGraphProcessFunction.java, src/main/java/com/zdjizhi/etl/dns/DnsGraphProcessFunction.java, src/main/java/com/zdjizhi/etl/dns/DnsRelationProcessFunction.java, src/main/java/com/zdjizhi/utils/arangodb/ArangoDBSink.java
Related findings
A MESA Lab task tracker records the assignment "TSG: determine how many clients are behind an IP address," and a companion technical design document details the method: identify distinct TLS clients sharing one public IP using a <JA3 fingerprint, server domain, server IP> 3-tuple (JA3 alone collides across different apps), then use TLS Session Ticket reuse/lifetime sequences per identified client to detect multiple concurrent devices (i.e. NAT) behind that IP.
A published MESA Lab / IIE-CAS paper ("ExitSniffer", CCIS-2021) and two filed patents describe a tool that actively builds 2-hop Tor circuits through every exit relay to decoy websites and diffs the decoy's observed source IP against the relay's public consensus IP to expose non-public "hidden node" infrastructure behind Tor exits (96% coverage of all exits in about 50 seconds), plus a complementary passive method -- a government-funded ("国家信息安全项目") controlled Tor relay that logs real predecessor-hop IPs -- that found hidden nodes behind 71 of 6,850 tracked routing nodes; the same research program's dataset separately logs 10,412,582 real anonymous-user access records against 1,627,920 distinct clearnet domains visited through Tor.
A MESA thesis proposal ("移动互联网信令数据关联分析方法研究") builds a SAPP platform plugin that parses LTE GTP-C control-plane signaling (S11 interface) to extract session/TEID identifiers and correlate them with user-plane IP sessions, explicitly to resolve subscriber identity that is otherwise unavailable because "the data plane and signaling plane are separated" in mobile networks. A later revision of the same proposal confirms the parser and the ID-to-IP correlation logic are both implemented as SAPP plugins.
Internal engineering doc describes "整形平台" (internally versioned as "Nirvana" — GitLab repos reshape/nirvana_client and reshape/nirvana_platform), a Kafka/Redis-backed, fully-async C/S platform sitting downstream of sapp that performs cross-session, cross-link correlation to reconstruct complete file content, VoIP audio, and session metadata. Its explicit "单向流对准" (single-direction flow alignment) feature reunites the client-to-server and server-to-client halves of one session when they were captured separately (e.g. asymmetric routing, or a passive mirror tap seeing only one direction).
A dedicated, actively-maintained TSG_OLAP Flink streaming job (galaxy/tsg_olap/sip-rtp-correlation, 200+ commits through late 2024) fuses one-way SIP signaling records with their corresponding RTP media-stream records into unified bidirectional VoIP call records in real time, keyed by address/session state with configurable expiry -- a production system for reconstructing complete VoIP call metadata (parties, duration/state) from separately-captured signaling and media flows.
A MESA Lab experiment re-identifies individual users across time solely from which domains they visit, using one-hot/frequency "domain access vectors" (Jaccard, overlap-coefficient, cosine, Euclidean) computed against real production HTTP logs pulled from a ClickHouse table named tsg_galaxy_v3.connection_record_log_http_domain. The best-performing configuration (Alexa top 100-10000 domain list, overlap coefficient) distinguished the same user's adjacent-hour sessions from other users' sessions with high separation (~0.19 vs ~0.88 average difference).