The Dark Side of Data The NSA ThinThread Tale

Transcription

1 The Dark Side of Data The NSA ThinThread Tale Thomas A. Drake Knowpari Systems LLC 18 Oct 2011

2

3

4

5 Or?? Or?? Orwell ian??

6 NSA in Action!

7 Purpose Provoke thought, reflection, as well as introspection from what matters Invoke consideration for alternative approaches and thinking that make sense Stoke calls for asking the right questions that generate focused discussion on informed decision-making and strategic implementation Internet time delivery (4x) in 12 minutes! We live in an asymmetric world Volatile, Uncertain, Complex, Ambiguous!

8 Remember Industrial Age versus Information and Knowledge Age world views Highly disruptive Social data structures Moving from transactional and stovepiped to relational and networked The thinking that got us here today is insufficient for what we need tomorrow - with thanks to Einstein

9 A Muddle of a Maze?! Consider this

10 Back to the Future?! How do you obtain the best possible information and information systems for making truly fact-based, informed decisions in the boardroom, the executive suite or even the halls of government? What if the best becomes a threat to the established way of doing things? What happens to decision making? What happens to your future? In my recently concluded past, the government wanted to put me in prison for telling the ThinThread tale and exposing government wrongdoing and illegalities. Government found out everything they could about me and turned me into an Enemy of the State. I was data framed!

11 Dark Side of Personal Data So Imagine for the Moment

12 Bottomline Sold out national security to big business and also violated the Constitution. All was so unnecessary. American ingenuity and the Constitution were quite sufficient to protect and defend the country with the best and under the Law. There was NO, I repeat, NO need to go to the dark side.

13

14 Context Setting Data absent context is data with no meaning! And all data is value laden! Lose the context, make up the meaning incredibly seductive to manage and manipulate data and for other ends. What do you do with your data?! How do you frame your data?!

15 A Republic If You Can Keep It Democracy is two wolves and a lamb voting on what to have for lunch. Liberty is a well-armed lamb contesting the vote. Benjamin Franklin What expired on 9/11? -- The Constitution? Corporatized democracy through soft tyranny? Those who give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety. Benjamin Franklin Is data free or the precursor for taking away freedom?

16 The Web of Data Challenges and Opportunities

17 The Information Age: Change Intelligence was once a closed ballgame Spies Secret electronic surveillance of the radio waves Private networks - Large, open networks like the Internet are changing the game - Cheap - increasingly widespread and available - Increasingly easy to hide No secrets worth knowing on Internet!

18 More challenges Major problem regarding information sharing Knowledge is control for some and controlling the data avoids potential competition and loss of power What I know data that you don t know data -- coin of the realm information hoarding Human factors as well as institutional and organizational operating behaviors had become major obstacles

19 Even More Challenges

20 Were Falling Way Behind Default approach to access: Pre-select source data (immediate bias) Analyst queries were essentially stabs in the dark Data spilling on the floor Missing all kinds of real-time and stored intelligence Classic approaches were utterly failing Was losing the battle with massive data Needed a smart approach Had to consider data in 3 dimensions just not 2! - The missing Z axis

21 Needed a Whole New Solution An analyst could spend all their time looking in database A, when the answer was in database B all along Had a glut of information, a dearth of knowledge, and incoherent capabilities Data selection had become a virtual toss of the dice Traditional tools could not manage multi-gigabytes and terabytes of live data streaming in real-time with storage requirements now at the multi-petabyte and multi-exabyte level and approaching the Zettabyte range! Could not catch up to the rate at which we were falling behind Needed a system that minimized the need for apriori knowledge, provided for rapid discovery, worked at near line speeds, and pointed the analyst to the relevant content Necessity became the mother of invention - American ingenuity and invention took on the core challenge in the 1990s SIGINT Skunk Works lab and ThinThread was the result.

22

23 THINTHREAD Designed as All the data, all the time Built relationships on world-wide scale Provided analytic focus to target development Provided manageable, relevant content Enabled detection of network changes Automatically compiled target activities Captured the information needed to build behavior profiles for inferencing Formed basis for analysis across any number of variables

24 What ThinThread Did

25 Discovering and Protecting Guarding Privacy While Finding the Threat - High fidelity, high integrity data

26 Discovering and Sharing Information Relationships Across Organizations C R O S S - D O M A I N C O R R E L A T I O N Business A Business B Business C Business D

27 Velocity as the Core Discovery and Real-Time Warning Driver High-Velocity, Volume, Variety Data Sources Build Entity Graph Selected Graph Analysis Actionable Intelligence ALL-SOURCE METADATA KNOWLEDGE (data at rest) ASSOCIATED ALL-SOURCE CONTENT Enabled by knowledge-based, rules-driven automation

28 Designed as a Complete and Trusted Information Management Approach In-depth network security to protect information (information assurance) Data masking to protect privacy and/or organizational equities Source masking to protect sensitive sources of information and to facilitate the down-grading of classification User masking to facilitate collaboration and risk-taking, and to protect organizational equities Near real-time sharing of data to capitalize on demonstrable relevance, without the need to disclose sensitive content or identities

29 The Dark Side of Innovation At worst? -- The throwing out of ThinThread was due to multibillion dollar buy versus make money interests and political connections, all surrounding personal gain and institutional self-interest. At best?? -- The throwing out of ThinThread was due to blatant ignorance and disregard for policy under the Federal Acquisition Regulations and the 4 th Amendment of the Constitution. Either way, same result critical breakthrough IT, security, and defense innovation and ingenuity denied to the American people with an incalculable loss of intelligence. The great historical tragedy of what could have been including the disruption, even the prevention, of 9-11.

30 What was Lost with the Loss of ThinThread Data Going Darker?

31 Whither Privacy?

32 Surveillance The Rise of the National Security State And you wonder what Section 215 of the Patriot Act is all about?? How is your data being used or abused? How do you frame your data?! Do You Know Where Your Data is Today? Or Where It Went Yesterday? Or Where It Is Going Tomorrow?!

33

34 What Do We Do?

35 What Do We Do Part 2 and You! Data as a Weapon? Internet as the Delivery Platform?

36 What Future Do We Want to Keep?