Spectrum Collaboration Challenge (SC2)

Transcription

1 Spectrum Collaboration Challenge (SC2) Phase 1 Entrance Hurdles Revision 1 9/1/2016 Defense Advanced Research Projects Agency Microsystems Technology Office 675 North Randolph Street Arlington, VA

2 Document Change Summary Section Description Date 2 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 1

3 1 Introduction Participation in the Spectrum Collaboration Challenge (SC2) is open to most all interested parties. However, access to the Colosseum testbed requires DARPA to set limits on the number of teams that can participate in any phase of the program. DARPA also needs to ensure that participating teams have the necessary skillset to field competitive Collaborative Intelligent Radio (CIR) solutions that perform properly on the testbed and meet the objectives of the SC2. For Proposal Track teams, this is determined by the quality of their proposals. Open Track participants will be required to achieve a sufficiently high score across a series of hurdles to secure one of the available seats. Entrance Hurdles will evaluate a team s ability to develop software defined radios and demonstrate applicable machine learning techniques. Open Track teams may join at any phase of the competition. This document defines the Entrance Hurdles for Open Track teams for Phase 1 of the SC2. 2 Entrance Hurdle Rounds The SC2 Entrance Hurdles for the Open Track are organized in two rounds: Threshold and Ranking. The Threshold round will be held in November Problems are graded pass/fail based on whether a team s solution reaches a performance level specified in this document. A team must score a passing grade on all Threshold problems to proceed to the Ranking round. The Ranking round will be held in January Performance on each problem will be compared to give a team ranking for each problem. Team rankings for each problem will be combined using the Schulze voting algorithm with problem weights TBD to give an overall team ranking. 1 The overall ranking will determine the teams admitted to the competition. If the number of Open Track teams passing the Threshold round is small enough that all can be accommodated in the competition, the Ranking round will be canceled. The Threshold problems are designed to assess basic competence in the field. The Ranking problems are open-ended and are considerably more difficult. We recommend that teams start work on the Ranking problems as early as possible. The Ranking problems are designed to 1 Schulze, Marcus. A new monotonic, clone-independent, reversal symmetric, and condorcet-consistent singlewinner election method. Social Choice and Welfare, Vol. 36. No. 2, Feb 2011, pp We apply the algorithm by treating each problem as a voter that rank orders the teams. The Schulze algorithm finds the overall ranking such that given the overall ranking for Team_k, for any pairwise comparison against another team Team_m (without loss of generality assuming the ranking for Team_k is higher than that of Team_m) then the sum of the weights of the tests in which the rank of Team_k is greater than that of Team_m is necessarily greater than the sum of the weights for the tests in which the ranking was reversed. 3 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 1

4 address issues that are relevant for the Competition. Therefore, work on them will be valuable even if the Ranking round is canceled. In each round, the problems cover 3 key areas of Collaborative Intelligent Radio technology: reconfigurable radio, RF environment understanding, and reasoning and contextualization. 3 Schedule The schedule for the Phase 1 Entrance Hurdles is as follows. 1 Sep 2016 Revision 1 of this hurdles document released Draft of Threshold problems Early Oct 2016 Revision 2 of this document released Final Threshold hurdle problems and logistics Draft Ranking hurdle problems Threshold hurdle software bundle released (see Section 4) Mid Oct 2016 DARPA notifies teams that submitted to the Proposal Track of the result of their proposal (selected or non selected) 1 Nov 2016 Threshold hurdle submission window opens 22 Nov 2016 Threshold hurdle submission window closes Early Dec 2016 DARPA announces whether Ranking round will occur. Revision 3 of this document released [if needed] Final Ranking hurdle problems and logistics Ranking hurdle software bundle released (see Section 4) 3 Jan 2017 Ranking hurdle submission window opens [if needed] 17 Jan 2017 Ranking hurdle submission window closes [if needed] 24 Jan 2017 DARPA releases list of teams joining the competition (both Open Track and Proposal Track) 4 Logistics for participation All interactions between teams and DARPA required to complete the Hurdles will occur remotely via and the SC2 website. No in-person meetings or webinars are planned. Details of the logistics for Threshold hurdles will be provided in Revision 2 of this document. At present, it is expected that: Hurdles will execute as a pair of Docker containers exchanging data over a (virtual) local network using Apache Thrift RPC. One container from DARPA, called the Test Infrastructure or TI container, contains the test sequencing and scoring processes, the 4 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 1

5 DARPA agents specified in the problems, and the test data. The other container implemented by the team provides the solution to the problem. DARPA will provide the teams a software bundle containing an executable instance of the Test Infrastructure Docker container along with source code and sample test data, to enable local testing. DARPA will specify a cloud service with which its TI container is known to be compatible (e.g. an AWS Instance and a specific Linux AMI) although teams are free to run the TI container wherever desired. For submission, teams will deliver their executable Docker container to DARPA SC2 Team, who will execute it on the specified cloud service in conjunction with a clean copy of the TI container containing new test data. Teams may only submit once. Failure due to a bug or platform compatibility issue is treated the same as poor performance the team will not pass the Hurdles. This models what will happen in the competition. Teams are encouraged to follow careful software engineering practices and perform pre-submission testing. DARPA will return the resulting score to the team as soon as the test has completed, but no log files or other internal information. After the submission window has closed, DARPA will distribute the TI container used for the submission tests so teams can execute it locally and assess why their software performed as it did. 5 Threshold Round Hurdles 5.1 Reconfigurable radio All teams will need some comfortability with radio design, even if that is not a team s strongest skillset. A large collection of open source tools are at your disposal to help with radio design. This hurdle is meant to ensure a team has an understanding of radio concepts, and can leverage available tools to solve radio problems. Problem: Develop a receiver for the following waveform. Channel bandwidth: Modulation: Symbol rate: Max timing offset: Max frequency offset: Channel: Inter-packet gap: 4 MHz QPSK 10 6 symbols per second ± 20 ppm ± 100 khz Flat fading, no multipath random duration, no carrier signal during gap 5 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 1

6 Packet format: There is a fixed 12 byte packet header at the start of each packet. The packet length is variable. The packet length field consists of four copies of an 8 bit packet length. The packet length includes the length of the header. Packet fields will be in big endian format. Bytes 0-3 Bytes 4-7 Bytes 8-11 Bytes 12-Packet Len Preamble (4 bytes) Sync Marker (4 bytes) Length (4 bytes) Payload (N bytes) The preamble and Sync Marker fields will have fixed contents as shown below First transmitted bit (Bit 0) Last transmitted bit (Bit 63) Preamble: 0x Sync Marker: 0x1ACFFC1D DARPA will distribute a transmitter for this waveform as part of the sample Test Infrastructure container. Any questions about the waveform specification can be answered by studying the transmitter code. Success condition: the receiver achieves a bit error rate (uncoded) better than 1x10-5 at a receiver Eb/No of 12.6 db. Testing: The Test Infrastructure container will send a stream of I/Q data representing 4x10 6 complex samples per second (4 samples per QPSK symbol) containing encoded packets plus inter-packet gaps and noise. The team s container should return the decoded packets to the TI container. This is not a real-time test: the receiver may take arbitrary execution time. 2 The bits in the preamble and sync marker fields should be included in the decoded packets sent to the TI container. However, those bits will not be considered when computing the achieved bit error rate. 5.2 RF Environment Understanding Teams will need to be able to apply AI pattern matching capabilities in the domain of radio signals. This threshold hurdle is meant to assess basic radio signal recognition. The Ranking version of the problem will extend to incorporate clustering challenges. Problem: Develop a classifier that can identify the occupied range and type of six simultaneous non-overlapping signals within a 3 MHz bandwidth channel. Each signal will be continuous in time and fixed in frequency for the duration of the test vector. The signals that may be present 2 Execution time must be reasonable. The test will be halted and declared a failure if the team s container has not returned a result approximately 10 minutes after receiving the last I/Q sample, or after the execution time exceeds 100x of real time, whichever is longest. 6 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 1

7 are: analog FM, QPSK, and GMSK. Multiple or no instances of a particular signal type may be present. Each signal is present at the same total power as each of the others. AWGN is present in the 3 MHz channel. The SNR seen by any individual signal type will be greater than or equal to 15 db. The scoring metric is as follows: Divide the band into 30 bins of 100 khz each The classifier identifies which bins are occupied by signals and which ones are just noise. The Probability of Detection PD is the number of occupied bins that the classifier reported as occupied, divided by the number of occupied bins. The Probability of False Alarm PFA is the number of unoccupied bins that the classifier reported as occupied, divided by the number of unoccupied bins. The classifier should additionally report the type for each bin that it believes is occupied. PT is the fraction of type reports that are correct. 3 The score is S = PD * (1 PFA) * PT. Success condition: S 0.68 Testing: The evaluation server will send the team s container an I/Q trace sampled at 2x Nyquist representing at least 40 msec worth of samples. The container will return a vector where each entry corresponds to a 100 khz bin and contains one of (empty, FM, QPSK, GMSK). DARPA will distribute sample I/Q traces as part of the TI container. The symbol rate and other aspects of the embedded signals may change from the sample I/Q trace to the one used in the submission test. 5.3 Reasoning and contextualization Teams will need to be able to build CIR networks that coexist gracefully in the spectrum with other networks. This requires the ability to learn and predict other networks spectrum access behavior. This hurdle assesses basic AI learning capability in the context of avoiding collisions with other spectrum users. Problem: Build an agent that plays a simplified spectrum sharing game with an agent provided by DARPA. Your agent is called the Player agent A; the DARPA agent is called D. The game consists of many turns in which each agent will vie for spectrum resources (e.g., channels) represented by integers. In each turn, two things happen simultaneously: 3 False alarms are thus penalized twice, since they increase PFA and decrease PT. This is intentional: false outputs from a classifier tend to harm downstream learning algorithms more than missed detects do. 7 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 1

8 The DARPA agent makes its choice of spectrum resource by choosing an integer between 1 and M: its output DO. The Player agent emits a pair of integers: its output AO (its choice of spectrum resource) and its prediction of the DARPA output DD OO, both between 1 and M. After these actions, A gets a payoff for avoiding a spectrum resource collision (AO DO). A increases its payoff if it predicts D s exact selection (DD OO =DO) since successful prediction enables optimal spectrum sharing. This is repeated over many turns. For A to do well, it must observe the history of D s behavior and improve its ability to predict D s actions. The DARPA agent D models a network whose spectrum access decisions depend, in some undisclosed way, on what it sees in the environment (e.g., the actions taken by A). D is an automaton with a finite number of states N. For each state qq 1.. qq nn in D, there is a state transition matrix giving the probability of the next state as a function of the player agent s output: Transitions from state q Player output AO Next state for D 1 2 N 1 p11 p12 p1n 2 p21 p22 p2n... M PM1 PM2 PMN All the pij 0. The sum of each row of the matrix is 1. One state of D is designated as the initial state, where D starts round 1. Associated with each state q is an integer O(q) [1..M] that specifies the output from D. Note that multiple distinct states may output the same integer, and there are no accepting or final states the automaton never restarts during the game. The sequence of events in each turn is as follows. 1. D starts the turn in some state q 2. A sends AO and PDO to D 3. D sends O(q) to A 4. A s score is increased according to the payoffs described below. 5. D transitions to a new state, with probabilities determined by the row for AO from the state transition matrix for q. 8 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 1

9 The payoff for A for each turn is as follows. +1 point if AO DO -12 points if AO = DO +3 points if PDO = DO These payoffs were chosen for M=10. They are likely to be adjusted if the final version of this problem uses a different value of M. Success condition: Total payoff > X after Y turns played against a DARPA agent of Z states. One set of parameters under consideration is (X,Y,Z) = (3000, 10000, 20). Testing: In each turn, the team s container sends its prediction and its output to the TI container, then the TI container sends what the DARPA agent s output was for that turn. The metric is computed across all turns starting from the first. 6 Ranking Round Hurdles DARPA will release a draft of the Ranking problems as part of a future revision of this document. At present the problems are expected to cover: Reconfigurable Radio: Develop a waterfilling waveform with error correction and implement a transmitter and receiver for it. The waveform should operate correctly despite sharing the channel with a time varying interfering signal. RF Environment Understanding: Develop a classifier that can identify the PHY type and MAC type, and cluster observed signals by network ID, for a spectrum trace containing packets sent by different networks that may collide with each other. Reasoning and Contextualization: The game of section 5.3 is expanded in the following ways: there are more than 2 agents in the environment, and the automaton periodically changes in ways that preserves portions of the prior transition matrices. 9 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 1