hezhengjie/videoportaldetection is a Spark/ClickHouse-backed MESA Lab pipeline (cn.ac.iie package) that mines DNS/HTTP/SSL/AV traffic logs with frequent-pattern-mining and a precision/recall evaluation module to automatically discover new audio/video-service domains (source resource file named "AV domains to be detected"), i.e. an automated FQDN-discovery system for media-streaming services that likely feeds a blocklist/classification pipeline rather than relying on static signature lists alone.
src/main/resources/待检测AV域名.txt 添加recall评价指标 添加多入口挖掘模块
Defense implications
- Traffic to a self-hosted or uncommon media/CDN domain may get flagged by frequency/co-occurrence pattern mining even without an explicit signature; rotating or randomizing outbound domains alone is not sufficient if the underlying traffic pattern (DNS+SSL+HTTP fingerprint shape) resembles known streaming services.
Related findings
The same internal research note's second research point develops an ML-based detector for Geneva-style automated censorship-evasion traffic; simple flow-level features (flow size, max packet size, RST/SYN/FIN flag counts, forward init-window bytes, inter-arrival timing) achieve near-perfect (ROC-AUC ~1.00) classification of Geneva-generated evasion traffic against CICIDS2017 and MAWI backbone background traffic using decision trees, LightGBM, XGBoost and random forest, with abnormal flow size (~150 bytes vs. 1000-30000 bytes typical) identified as the single most discriminative feature.
A 2020 MESA Lab monthly report describes building a sapp plugin that extracts packet-sequence features specifically "for DoH (DNS-over-HTTPS) service discovery," alongside a broader CSTNET DoH measurement-report effort and configuring a DNS-to-DoH gateway -- confirming sapp is used to fingerprint DoH traffic via statistical sequence features rather than plaintext DNS content.
MESA Lab researcher notes on an encrypted-video-identification project describe adding SSL-layer information output to more precisely trace a flow's true source/identity, and considering reinforcement learning so the identification model adapts as network conditions change, working within/around sapp's plugin limitations.
An internal schema doc describes an 'Unknown Protocol Identification Database': an Elasticsearch port-asset table tracking active/passive/fused protocol-type guesses and banner text per IP:port, feeding a MySQL clustering pipeline (cluster_info/cluster_task) that groups unclassified traffic by a 'fingerprint' field into named-protocol clusters -- an unsupervised discovery pipeline for identifying and naming new/unknown protocols at scale, distinct from MAAT/AppSketch's signature-matching against already-known protocols.
A thesis-project assignment for hidden-service (VPN/Proxy/Tor) identification via heterogeneous graph neural networks on flow logs sources its one day of training data from a database explicitly named 'tsg_galaxy_p19' — tying the internal TSG naming convention to the taxonomy's P19/WMS-UTR Pakistan site codename — accessed over an internal 'Information Harbor' (信息港) VPN, with ground-truth hidden-service IP labels supplied by the commercial IP-intelligence service spur.us.
A master's thesis-in-progress supervised within this ecosystem (advisor 杨嵘, senior engineer) is explicitly tied to the national key R&D program "海量公害 网页、图片、视频流量识别技术" (massive nuisance webpage/image/video traffic identification technology) and a VoIP engineering project; it builds cross- platform (YouTube/Facebook/Bilibili) encrypted-video identification that matches a single per-video traffic fingerprint across different CDNs/resolutions/codecs via trend-similarity, fuzzy-search, and meta-learning matching, explicitly framed around detecting "非法视频" (illegal/harmful video) spread across platforms.