ThinThread is a compelling story about modern traffic analysis: how to use meta data to gather and analyze intelligence without reading the messages themselves.
A few years ago, I wrote a series about wireless intercept during World War 2. Part of that story was about radio traffic analysis – how to gather intelligence from the flow of messages without knowing their content.
For example, what party is sending the signal and to whom? What is the frequency and the call sign? Where is the message coming from and where is the destination? Are there code names for participants or locations? Are there networks of participating stations that can be identified? And so on.
Recently, I watched a Netflix movie called A Good American which tells a story about modern traffic analysis. If you are technically inclined, you might enjoy the details of an NSA tool that was called ThinThread. Basically, this was a tool developed to inspect digital message traffic between billions of individuals and draw complex relationship graphs. Certain patterns in these graphs provided alerts about possible terrorist or illegal activities.
ThinThread was all about mathematics and algorithms to sift through large sets of data and find the stuff that might be important, without downloading and storing a lot of irrelevant material. Developed by the small Signals Intelligence Automation Research Center, ThinThread cost $3 million and was said to achieve more than subsequent efforts costing $3 billion.
Modern traffic analysis needs to handle volume, velocity and variety of data collected from e-mail, phone calls, social media, etc. With the pervasive adoption of encryption, traffic analysis is more important than ever. Typically legacy NSA systems focused on keywords which can be obscured by encryption.
ThinThread was designed to focus instead on traffic meta data rather than messages. If the meta data revealed a suspicious pattern, efforts would then switch to content. This saved a lot of time and storage space.
ThinThread – Typical Story of Large Organization Behavior
A Good American focuses a lot on bureaucratic and political infighting within NSA and government. Astonishingly, ThinThread was cancelled three weeks before 9/11, when it might have been quite useful. The story reminds me a lot of how the US was not able to take full advantage of available SIGINT during the run up to Pearl Harbor.
You can find lots of good stories online about the controversy around this apparent NSA failure. Just search on “ThinThread” and read a few.
As a documentary, A Good American is fairly weak overall. But it shines in its compelling conversations about and descriptions of the technologies and approaches to modern traffic analysis – and how to automate analysis and use relationship graphs to figure out key patterns. Here is a good description of the ThinThread technology process, including how to protect privacy during data collection and still be effective.