A NOVEL BIG DATA ARCHITECTURE IN SUPPORT OF ADS-B DATA ANALYTIC DR. ERTON BOCI

Transcription

1 Place image here (10 x 3.5 ) A NOVEL BIG DATA ARCHITECTURE IN SUPPORT OF ADS-B DATA ANALYTIC DR. ERTON BOCI Big Data Analytics HARRIS.COM #HARRISCORP

2 Agenda With 87,000 flights per day, America s ground based radar system has hit a growth ceiling. The FAA has undertaken the implementation effort of the Next Generation Air Transportation System (NextGen) initiative that would transform today s air transportation and ensure increased safety and capacity in the National Airspace System (NAS). The first building block of the NextGen initiative is the implementation of the Automatic Dependent Surveillance Broadcast (ADS-B) system. 650 radio stations nationwide terrestrial infrastructure In March 2014, the FAA completed the nationwide infrastructure upgrade, enabling air traffic controllers to track aircraft with greater accuracy and reliability, while giving pilots more information in the cockpit. SBSS 2

3 Nationwide Radio Station Infrastructure Deployment Source: FAA, "2014 NextGen Implementation Plan" SBSS 3

4 ADS-B Services The ADS-B system requires the provision of essential and critical services. Critical services consist of ADS-B Surveillance of 1090ES and UAT aircraft ADS-B Rebroadcast (ADS-R) Wide Area Multilateration (WAM) Essential services consist of Traffic Information Services Broadcast (TIS-B) Flight Information Service Broadcast (FIS-B). ADS-B services in the United States are based upon two noninteroperable data link technologies 1090 MHz Extended Squitter (1090ES): this data link is applicable primarily to commercial aviation aircrafts Universal Access Transceiver (UAT): this data link operates at 978 MHz and is applicable primarily to general aviation aircraft SBSS 4

5 ADS-B Services (2) SBSS 5

6 ADS-B CAT033 Messages ADS-B system ingests, processes and stores terra-bytes of compressed data each month In high interference environments such as busy En Route, Terminal and Surface Service Volumes (SVs) each radio station employs multiple 1090ES Rx/Tx antennas. More than 2,500 Rx/Tx channels constitute the deployed infrastructure of radio stations across the US. Rx channels receive aircraft 1090ES and/or UAT messages and generate the radio CAT033 messages. These messages are forwarded to the ADS-B control stations via the network. ADS-B Control station processes CAT033 messages and stores into the Data Recording and Logging (DLR) ADS-B sub-system. CAT033s and other type of messages, ingested and generated by ADS-B system, are archived on a regular basis in a large storage system. SBSS 6

7 Big Data - Hadoop Three main resources of Hadoop Hadoop Distributed File System (HDFS) The MapReduce programing platform Applications that ingest, store and use data, and manage the hardware that runs Hadoop SBSS 7

8 Hadoop Distribution Vendors Evaluation conducted 12/2013 Cloudera Hortonworks MapR Product Name CDH 4 Data Platform (HDP) M7 (NoSQL) / M5 / M3 distribution is free Proprietary Code Open source with some proprietary Open source Hadoop Not built on Hadoop HBase HCatalog Geospatial Address no no no Single Point of Failure yes yes yes Metadata Architecture Centralized Centralized Distributed Management Tools Cloudera Manager Ambari MapR Control System Tableau Yes Yes Yes Extras Accumulo Accumulo The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. It uses fine-grained security controls allowing organizations to control data at the cell-level without degrading performance. Real-time RTD, RTQ, RTS Stinger and Storm Dremel Training near company headquarters Pricing Herndon, VA Reston, VA Herndon, VA Cloudera Enterprise Data Hub 24x7: CDH + Annual subscription (per node) for Cloudera Manager and Technical Support, including RTD, RTQ, RTS, Navigator, and BDR. Support available 24 hours a day, 7 days a week. GSA discount - yes Both Enterprise and Standard support packages available. Additional packages: Project Jumpstart - 1 year Enterprise - 20 nodes/250tb - 1 seat developer support - Hadoop Class (for 8) - 2 weeks consulting POC Jumpstart - 1 seat developer support - Hadoop Class (for 8) - 1 week consulting MapR software comes in 1 and 3 year subscriptions for their M5 and M7 distributions GSA discount - yes Five major Hadoop software distribution vendors offer commercial software distribution based on Apache Hadoop 1 : Cloudera Hortonworks IBM MapR Technologies Pivotal Focused on: Cloudera Hortonworks MapR Technologies 1 Source: Gartner 2015, Market Guide for Hadoop Distributions SBSS 8

9 ADS-B Data Lake Architecture SBSS 9

10 Hardware Cluster Infrastructure Control Nodes Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9 Node 10 Data Nodes Node 11 Control Node Services: Zookeepers Namenodes Hive Metadata Server Activity Monitors Resource Manager Job History Hive Gateway Hive Metastore Ozzie Server Data Node Services: HDFS Hive Hive Gateway SBSS 10

11 HortonWorks Distribution Hue Query Editor SBSS 11

12 ZDW, ZNY and ZDC Radio CAT033 geospatial density analysis Data Lake architecture (VMs cluster) was used to Process 17 billion records, or one month s amount of CAT033 log data, generated from 291 Rx 1090 channels deployed in 74 radio stations Process time about 35 minutes in 8 VMs. Data grid resolution of ~500m was used for this density analysis. The output file consists of ~2.9 million density data points. MapInfo geospatial tool was used to Render density data points into a grid file. Generate an pdf image at 300dpi resulting in a 30MBytes pdf file. SBSS 12

13 BOS, JFK and IAD air traffic corridor High Resolution maps allow for close-ups without a need for re-run Proper use of color schema naturally reveals the areas (airways) with high traffic density SBSS 13

14 Radio Station CAT033 Density Analysis 1090 MHz Channel orientation North East South West Number of CAT033 messages (03/31/2014) 2.2 million 1.6 million 1.2 million 2.1 million Radio station-specific density analysis filters results based on the radio station channels and frequency Combined density of the 90-degree 1090MHz channels designed to provide full 360 degree coverage SBSS 14

15 Radio CAT033 density analysis IAD SBSS 15

16 Radio CAT033 density analysis IAD (2) Hotspot identified Density map requires alignment of ~1.5km to match with jetway J75 SBSS 16

17 Radio CAT033 density analysis IAD (3) SBSS 17

18 Real-Time Querying Alternatives (Emerging Technologies) Data Sources/ File Systems Companies (Sponsor, Partner) Apache Hive (Hadoop) HDFS, HBase, Amazon S3 Apache Software Foundation Apache Drill HBase, Mongo, HDFS MapR Cloudera Impala Berkeley Spark/ Shark Amazon Redshift HDFS, HBase Hive Proprietary (not open source) Multiple- Tableau, etc. API(s) Java REST? Hive & JDBC, ODBC driver Query Languages Production Ready Hive SQL ANSI SQL:2003 Hive & Subset of ANSI-92 SQL Multiple Scala, Java & Python Hive SQL Multiple- Tableau, etc. JDBC and ODBC Based upon PostgreSQL Yes No Yes Yes No (in beta) Metrics Time(s) 1822 n/a 345 disk 337memory 538 disk 447 memory Reason for this row: who s going to maintain this technology? Join Query, Median Response Time (s); 533,287,121 rows. No User Defined Function (UDF) support. SBSS 18

19 Conclusion and Q&A Hadoop-based architecture, ADS-B Data Lake, is mature enough to handle demanding use-cases in support of large scale ADS-B data analysis. Enables ingestion and analysis of billions of CAT033 records in a relatively short time. Future work Evaluate new open source applications, such as Spark, in an effort to further reduce computation time. Evaluate open-source applications to enable data transfer in and out of Hadoop cluster. Evaluate open source applications that enable workflows in an attempt to automate the analysis. Support additional use-cases. SBSS 19