Spectrum Collaboration Challenge (SC2)

Transcription

1 Spectrum Collaboration Challenge (SC2) Phase 1 Entrance Hurdles Problem Description Revision 4 11/22/2016 Defense Advanced Research Projects Agency Microsystems Technology Office 675 North Randolph Street Arlington, VA

2 Document Change Summary Section Description Date 4. Logistics Updated 10/17/16 5. Threshold Hurdles Updated 10/17/16 4. Logistics Removed Docker reference 11/7/ RF Environment Deleted power constraint 11/7/16 3. Schedule Extended hurdle submission deadline 11/22/16 2 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 4

3 1 Introduction Participation in the Spectrum Collaboration Challenge (SC2) is open to most all interested parties. However, access to the Colosseum testbed requires DARPA to set limits on the number of teams that can participate in any phase of the program. DARPA also needs to ensure that participating teams have the necessary skillset to field competitive Collaborative Intelligent Radio (CIR) solutions that perform properly on the testbed and meet the objectives of the SC2. For Proposal Track teams, this is determined by the quality of their proposals. Open Track participants will be required to achieve a sufficiently high score across a series of hurdles to secure one of the available seats. Entrance Hurdles will evaluate a team s ability to develop software defined radios and demonstrate applicable machine learning techniques. Open Track teams may join at any phase of the competition. This document defines the Entrance Hurdles for Open Track teams for Phase 1 of the SC2. 2 Entrance Hurdle Rounds The SC2 Entrance Hurdles for the Open Track are organized in two rounds: Threshold and Ranking. The Threshold round will be held in November Problems are graded pass/fail based on whether a team s solution reaches a performance level specified in this document. A team must score a passing grade on all Threshold problems to proceed to the Ranking round. The Ranking round will be held in January Performance on each problem will be compared to give a team ranking for each problem. Team rankings for each problem will be combined using the Schulze voting algorithm with problem weights TBD to give an overall team ranking. 1 The overall ranking will determine the teams admitted to the competition. If the number of Open Track teams passing the Threshold round is small enough that all can be accommodated in the competition, the Ranking round will be canceled. The Threshold problems are designed to assess basic competence in the field. The Ranking problems are open-ended and are considerably more difficult. We recommend that teams start work on the Ranking problems as early as possible. The Ranking problems are designed to 1 Schulze, Marcus. A new monotonic, clone-independent, reversal symmetric, and condorcet-consistent singlewinner election method. Social Choice and Welfare, Vol. 36. No. 2, Feb 2011, pp We apply the algorithm by treating each problem as a voter that rank orders the teams. The Schulze algorithm finds the overall ranking such that given the overall ranking for Team_k, for any pairwise comparison against another team Team_m (without loss of generality assuming the ranking for Team_k is higher than that of Team_m) then the sum of the weights of the tests in which the rank of Team_k is greater than that of Team_m is necessarily greater than the sum of the weights for the tests in which the ranking was reversed. 3 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 4

4 address issues that are relevant for the Competition. Therefore, work on them will be valuable even if the Ranking round is canceled. In each round, the problems cover 3 key areas of Collaborative Intelligent Radio technology: reconfigurable radio, RF environment understanding, and reasoning and contextualization. 3 Schedule The schedule for the Phase 1 Entrance Hurdles is as follows. 1 Sep 2016 Revision 1 of this hurdles document released Draft of Threshold problems 17 Oct 2016 Revision 2 of this document released Week of 17 Oct 2016 Final Threshold hurdle problems and logistics Threshold hurdle software bundle released (see Section 4) DARPA notifies teams that submitted to the Proposal Track of the result of their proposal (selected or non selected) 1 Nov 2016 Threshold hurdle submission window opens 22 Nov 2 Dec 2016 Threshold hurdle submission window closes Early Dec 2016 DARPA announces whether Ranking round will occur. Revision 3 of this document released [if needed] Final Ranking hurdle problems and logistics Ranking hurdle software bundle released 3 Jan 2017 Ranking hurdle submission window opens [if needed] 17 Jan 2017 Ranking hurdle submission window closes [if needed] 24 Jan 2017 DARPA releases list of teams joining the competition (both Open Track and Proposal Track) 4 Logistics for participation and test procedure All interactions between teams and DARPA required to complete the Hurdles will occur remotely via and the SC2 website. No in-person meetings or webinars are planned. For detailed instructions on how to participate in the Threshold Hurdles, see Instructions-and-Logistics.pdf The following summary is provided to facilitate the problem descriptions in the next section. 4 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 4

5 Hurdle tests will execute as a pair of LXC (Linux Containers) containers exchanging data over a (virtual) local network using Apache Thrift RPC or bare TCP, depending on the hurdle. One container from DARPA, called the Test Infrastructure or TI container, contains the test sequencing and scoring processes, the DARPA agents specified in the problems, and the test data. The other container implemented by the team provides the solution to the problem. DARPA has distributed a software bundle containing an executable instance of the Test Infrastructure Docker container along with source code and sample test data, to enable local testing. For submission, teams will deliver their executable LXC container to DARPA SC2 Team, who will execute it in conjunction with a clean copy of the TI container containing new test data. DARPA will return the resulting score to the team as soon as the test has completed, but no log files or other internal information. After the submission window has closed, DARPA will distribute the TI container used for the submission tests so teams can execute it locally and assess why their software performed as it did. 5 Threshold Round Hurdles 5.1 Reconfigurable radio All teams will need some comfortability with radio design, even if that is not a team s strongest skillset. A large collection of open source tools are at your disposal to help with radio design. This hurdle is meant to ensure a team has an understanding of radio concepts, and can leverage available tools to solve radio problems. Problem: Develop a receiver for the following waveform. Channel bandwidth: Modulation: Symbol rate: Max timing offset: Max frequency offset: Channel: Inter-packet gap: 4 MHz QPSK 10 6 symbols per second ± 20 ppm ± 100 khz Flat fading, no multipath random duration, no carrier signal during gap Packet format: There is a fixed 32 byte packet header at the start of each packet. The packet length is variable. Header fields are in big endian format. 5 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 4

6 Bytes 0-3 Bytes 4-7 Bytes 8-11 Bytes Bytes Bytes 32 - End Preamble Sync Marker Packet Length Packet Count CRC Payload (4 bytes) (4 bytes) (4 bytes) (16 bytes) (4 bytes) (N bytes) The preamble and Sync Marker fields have fixed contents as shown below First transmitted bit (Bit 0) Last transmitted bit (Bit 63) Preamble: 0x Sync Marker: 0x1ACFFC1D The packet length field consists of four copies of an 8 bit packet length. The packet length includes the length of the header. The packet count field consists of four copies of a 32 bit packet length. The Cyclic Redundancy Check (CRC) field is used to assure correct scoring. Participants need not check the CRC and may treat it as part of the payload if desired. DARPA is distributing a transmitter for this waveform as part of the sample Test Infrastructure container. Any questions about the waveform specification can be answered by studying the transmitter code. Success condition: The receiver must achieve a bit error rate (uncoded) better than 1x10-5 at a receiver Eb/No of 12.6 db. Testing: The Test Infrastructure (TI) container will send a stream of I/Q data representing 4x10 6 complex samples per second (4 samples per QPSK symbol) containing encoded packets plus inter-packet gaps and noise. The team s container should return the decoded packets to the TI container as a TCP stream. This is not a real-time test: the receiver may take arbitrary execution time. 2 The bits in the preamble and sync marker fields should be included in the decoded packets sent to the TI container. They are used by the scoring algorithm to detect the start of packets in the decoded TCP stream. However, the bits in the preamble and sync marker fields will not be considered when computing the achieved bit error rate. For each packet detected in the reported stream, the scoring algorithm first checks the CRC field. If the header CRC doesn t match, the packet is discarded. All bits of discarded packets are considered to be in error. Next, the scoring algorithm uses the packet counter field to identify the ground truth version of the packet for a bit by bit comparison. The participant s final bit 2 Execution time must be reasonable. The test will be halted and declared a failure if the team s container has not returned a result approximately 10 minutes after receiving the last I/Q sample, or after the execution time exceeds 100x of real time, whichever is longest. 6 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 4

7 error rate is determined as a function of the total number of correct bits discovered during the bit by bit comparisons and the total number of ground truth bits transmitted. 5.2 RF Environment Understanding Teams will need to be able to apply AI pattern matching capabilities in the domain of radio signals. This threshold hurdle is meant to assess basic radio signal recognition. The Ranking version of the problem will extend to incorporate clustering challenges. Problem: Develop a classifier that can identify the occupied range and type of six simultaneous non-overlapping signals within a 3 MHz bandwidth channel. The channel is divided up into 30 bins of 100 khz each. Signals may have different bandwidths, but each signal occupies an integer number of bins. There is a guard band of at least one frequency bin between signals. Each signal is continuous in time and fixed in frequency for the duration of the test vector. The signals that may be present are: analog FM, QPSK, and GMSK. Multiple or no instances of a particular signal type may be present. Each signal is present at the same total power as each of the others. Additive White Gaussian Noise (AWGN) is present in the 3 MHz channel. The signal to noise ratio seen by any individual signal type will be greater than or equal to 15 db. The scoring metric is as follows: The classifier identifies which bins are occupied by signals and which ones are just noise. The Probability of Detection PD is the number of occupied bins that the classifier reported as occupied, divided by the number of occupied bins. The Probability of False Alarm PFA is the number of unoccupied bins that the classifier reported as occupied, divided by the number of unoccupied bins. o A guard band bin (adjacent to an occupied bin) is not considered unoccupied. Therefore, the classifier s report for guard band bins does not affect the score. The classifier should additionally report the type for each bin that it believes is occupied. PT is the fraction of type reports that are correct, ignoring guard band bins. 3 The score is S = PD * (1 PFA) * PT. Success condition: S 0.68 Participants are encouraged to set their detection thresholds such that sidelobes from signals extending beyond the adjacent bin are not counted as detections. 3 False alarms are thus penalized twice, since they increase PFA and decrease PT. This is intentional: false outputs from a classifier tend to harm downstream learning algorithms more than missed detects do. 7 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 4

8 Testing: The TI container sends the team s container an I/Q trace sampled at 2x Nyquist representing at least 40 msec worth of samples. The team s container returns a vector where each entry corresponds to a 100 khz bin and contains one of (empty, FM, QPSK, GMSK). DARPA has distributed sample I/Q traces as part of the TI container. The symbol rate and other aspects of the embedded signals may change from the sample I/Q trace to the one used in the submission test. 5.3 Reasoning and contextualization Teams will need to be able to build CIR networks that coexist gracefully in the spectrum with other networks. This requires the ability to learn and predict other networks spectrum access behavior. This hurdle assesses basic AI learning capability in the context of avoiding collisions with other spectrum users. Problem: Build an agent that plays a simplified spectrum sharing game with an agent provided by DARPA. Your agent is called the Player agent A; the DARPA agent is called D. The game consists of many turns in which the agents vie for spectrum resources (e.g., channels) represented by integers. In each turn, two things happen simultaneously: The DARPA agent makes its choice of spectrum resource by choosing an integer between 1 and M: its output DO. The Player agent emits a pair of integers: its output AO (its choice of spectrum resource) and its prediction of the DARPA output DD OO, both between 1 and M. After these actions, A gets a payoff for avoiding a spectrum resource collision (AO DO). A increases its payoff if it predicts D s exact selection (DD OO =DO) since successful prediction enables optimal spectrum sharing. This is repeated over many turns. For A to do well, it must observe the history of D s behavior and improve its ability to predict D s actions. The DARPA agent D models a network whose spectrum access decisions depend, in some undisclosed way, on what it sees in the environment (e.g., the actions taken by A). D is an automaton with a finite number of states N. For each state qq 1.. qq nn in D, there is a state transition matrix giving the probability of the next state as a function of the player agent s output: Transitions from state q Player output AO Next state for D 1 2 N 1 p11 p12 p1n 2 p21 p22 p2n... M PM1 PM2 PMN 8 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 4

9 All the pij 0. The sum of each row of the matrix is 1. One state of D is designated as the initial state, where D starts round 1. Note that there are no accepting or final states the automaton never restarts during the game. The sequence of events in each turn is as follows. 1. D starts the turn in some state labeled q 2. A sends AO and PDO to D 3. A s score is increased according to the payoffs described below. 4. D sends the integer q and the payoff to A 5. D transitions to a new state, with probabilities determined by the row for AO from the state transition matrix for q. The payoff for A for each turn is as follows. +1 point if AO DO -12 points if AO = DO +3 points if PDO = DO These payoffs were chosen to ensure that random guessing gives a low score when M=10. Success condition: There will be 10 subtrials of 30,000 turns each. Payoff earned in the first 29,000 turns is discarded. The score for A in a subtrial is the total payoff accumulated in the last 1,000 turns of the subtrial. To pass the hurdle, the team's score must exceed 2,000 for at least 6 out of the 10 subtrials. Testing: In each step, the DARPA agent sends a Thrift remote procedure call to the team s agent containing the DARPA agent s output for the previous turn and the team s payoff from the previous turn. The team s agent returns its output for the current turn and its prediction of the DARPA agent s output for the current turn. In the first turn, the DARPA agent sends a remote procedure call without any arguments. 6 Ranking Round Hurdles DARPA will release a draft of the Ranking problems as part of a future revision of this document. At present the problems are expected to cover: Reconfigurable Radio: Develop a waterfilling waveform with error correction and implement a transmitter and receiver for it. The waveform should operate correctly despite sharing the channel with a time varying interfering signal. 9 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 4

10 RF Environment Understanding: Develop a classifier that can identify the PHY type and MAC type, and cluster observed signals by network ID, for a spectrum trace containing packets sent by different networks that may collide with each other. Reasoning and Contextualization: The game of section 5.3 is expanded in the following ways: there are more than 2 agents in the environment, and the automaton periodically changes in ways that preserves portions of the prior transition matrices. 10 Spectrum Collaboration Challenge Phase 1 Entrance Hurdles Revision 4