THE OPEN TRACEABLE TIME PLATFORM AND ITS APPLICATION IN FINANCE AND TELECOMMUNICATIONS. Michael J. Wouters. National Measurement Institute, Australia

Transcription

1 THE OPEN TRACEABLE TIME PLATFORM AND ITS APPLICATION IN FINANCE AND TELECOMMUNICATIONS Michael J. Wouters National Measurement Institute, Australia E. Louis Marais National Measurement Institute, Australia Amitava Sen Gupta National Physical Laboratory, India Ahmad Sahar bin Omar National Metrology Institute of Malaysia Piyaphat Phoonthong National Institute of Metrology Thailand Abstract The Open Traceable Time Platform (OpenTTP) is an open source project that is developing software and low-cost hardware for disseminating legally traceable time and frequency. We describe the software which is the core of OpenTTP and the reference hardware implementation. We then focus on an important application, the provision of a source of traceable time of day via the Network Time Protocol (NTP). The performance of the OpenTTP system as a time server is characterized from a metrological perspective. Practical time-of-day applications in finance and telecommunications are described. 1

2 Key words GPS, time-transfer, traceability, finance, telecommunications Corresponding author Michael J. Wouters National Measurement Institute, 36 Bradfield Rd, Lindfield, NSW, 2070 Australia Tel

3 I. Introduction The objective of the Open Traceable Time Platform (OpenTTP) project is to create a fully open platform for developing time and frequency services with timing signals traceable to national standards. The OpenTTP project comprises both hardware and software, with the aim of providing a complete, functioning system that can be used out of the box. All software and the reference hardware design are freely available via GitHub [1]. The OpenTTP project has its roots in the experiences of the National Measurement Institute Australia (NMIA), in providing remote time and frequency services to domestic users over the past 20 years. Initially, remote calibration services were provided for laboratory frequency references, with custom hardware and software developed to enable this, because there was no suitable commercial platform [2]. These services eventually grew to include other applications such as time of day and network time auditing services. In these applications, metrological traceability of the timing signals to the national standard is either legally required or desired for assurance of accuracy and reliability. Traceability can be defined in a metrological context as the property of a measurement result whereby the result can be related to a reference through a documented unbroken chain of calibrations, each contributing to the measurement uncertainty [3].The emphasis on traceability is a key aspect of the OpenTTP project. An open source approach has been embraced to encourage co-operation amongst national metrology institutes but also to better support traceability and auditability of the system. Traceability is enhanced by giving the user complete access to the measurement recording and analysis chain. Auditability is achieved by full access to data collected by the system, at the level of raw data reported by the GPS receiver. In this paper, we describe the techniques used to provide traceability to remote users and then introduce the OpenTTP reference platform. We explain the design decisions that were made and characterize the performance of the system. The emphasis in our discussion is on time rather than frequency because time-of-day applications are of the most interest. We focus on system accuracy 3

4 since a key aspect of traceability is being able to state uncertainties at all points in the traceability chain. Finally, some industrial applications in finance and telecommunications are discussed. II. GPS time-transfer GPS time-transfer is the core technique used in OpenTTP to establish traceability between a remote time standard and the national standard. It has been used by national timing laboratories since 1980 for comparing remote clocks [4] and is still the principal means by which the clock comparisons needed for the calculation of Co-ordinated Universal Time (UTC), the international time standard, are made. Figure 1. Principle of GPS time-transfer. The timing signals available from GPS are used by a GPS timing receiver to produce a one pulse-per-second tick that can be measured against a local clock A. When the same measurement is made at B, the difference A-B can be obtained. The principle of GPS time transfer is illustrated in Figure 1. GPS positioning is based on timing signals derived from the atomic clocks on board each visible GPS satellite and present in the GPS data stream. These timing signals are extracted by a GPS receiver and can be used to produce a one pulse per second (pps) output that is linked to the atomic clocks on board each GPS satellite which can then be measured against a local clock. Clocks at two ground stations can then be linked over continental 4

5 distances by using GPS as a common reference. Noise is introduced into the link by the requirement for each station to estimate the continuously varying delay between it and the GPS satellite clock. GPS time-transfer can achieve measurement uncertainties of better than 1 ns for time of day on a carefully calibrated link [5] and relative uncertainties of better than 1 part in for frequency when averaging over a few days [6]. Other Global Navigation Satellite Systems (GNSS) such as BeiDou and GLONASS may also be used for time-transfer. Currently, OpenTTP supports the use of GPS only. III. The OpenTTP reference platform The reference platform was designed as part of an Asia Pacific Metrology Programme initiative aimed at assisting developing national metrology institutes. Total system cost was a significant constraint, being limited to $US2000 per unit, and therefore compromises that limited performance were necessarily made in the choice of some system components. However, most of the software is not tied to the specific hardware used and can be run readily on different platforms. A. Hardware The principal components of the OpenTTP reference platform and the main timing signals are shown in Figure 2. Commercial modules were used where available, mainly to reduce development time but also to allow for easier customisation. The main custom component in the reference platform is the motherboard to which the various modules are connected 5

6 Figure 2. Principal components of the OpenTTP reference platform and timing signals. A computer is required to collect and process data and enable network-based applications such as serving time of day via the Network Time Protocol (NTP). Small, single-board ARM-based computers (SBCs) are sufficiently powerful for our application and have the advantage of providing a simple interface between external timing signals and the microprocessor via a number of General Purpose Input/Output (GPIO) pins. Candidate SBCs were narrowed down to the BeagleBone Black (BBB) and Raspberry Pi 2 (RPi), two mainstream SBCs that are well-supported by their respective communities and that might be expected to have a usefully long product cycle. The BBB was chosen because of its two real-time coprocessors that were attractive for timing functions but in the end were not used. One other difference between the BBB and RPi is that the Ethernet interface (PHY) is a discrete chip on the BBB whereas it is part of the RPi system-on-chip (SOC) and running on the SOC s internal USB bus. NTP performance might therefore be expected to be slightly poorer on the RPi because the PHY is contending with other USB devices but this was not tested. The time-interval counter is implemented in a field-programmable gate array (FPGA). An FPGA development board, the Opal Kelly XEM6001, was chosen because of its convenient application programming interface which provides an easy-to-use USB interface for configuration and collecting data. The FPGA is configured to provide a multi-channel counter/timer and also provides some signal 6

7 switching functions. The multi-channel counter is comprised of 32 bit, 200 MHz counters, so that the time interval measurements have a resolution of 5 ns. The relatively coarse time resolution was a compromise in the design, but has the potential for improvement in the future. The reference clock for the system is a Jackson Labs LTELite GPS-disciplined oscillator (GPSDO). Figure 3 shows its measured time deviation (TDEV) when compared with a Microsemi 5071A cesium standard. The maximum TDEV is 20 ns, at an averaging time of about 200 s. The oscillator in this GPSDO is a temperature compensated crystal oscillator (TCXO). The TCXO is sensitive to temperature variations so the GPSDO was thermally shielded to improve its stability. This low-end device was one of the main design compromises made due to the limited budget. The most significant drawback of the TCXO is its limited holdover in the absence of a GPS signal. When unlocked, and operated in an environment where the temperature is controlled to ±0.5 C, the GPSDO drifts by about 1 ms over 5 days. Figure 3. Time deviation (TDEV) of the LTELite GPS-disciplined oscillator with respect to a cesium standard. An NVS Technologies NV08C-CSM GNSS timing receiver is used for time-transfer (the GPSDO has a separate GPS receiver). The NV08C is a 32 channel single-frequency receiver that tracks GPS, GLONASS and Galileo (BeiDou is also tracked in the newest version of the receiver) in various combinations. In the OpenTTP reference platform, it is configured to track GPS only. 7

8 All schematics for the electronics and supporting documentation are available from GitHub [1]. B. Software In common with other ARM-based SBCs, the BeagleBone Black runs under Linux, an open source operating system that has become ubiquitous in embedded devices. Linux offers many advantages including full control via the command line so that it is easily managed remotely. The particular Linux distribution we are using is Debian GNU/Linux 9 with a 4.14 version kernel. For NTP service, we use the NTP reference implementation, ntpd, version 4.2.8p10. The core of OpenTTP is its extensive suite of custom software. This includes logging and configuration scripts for various GNSS receivers and counter/timers, and mktimetx, the software used to produce GNSS time-transfer files. All custom software is available from GitHub [1]. IV. Stability and accuracy of time-transfer To evaluate the stability of time-transfer, we first established a baseline for performance by operating the NV08C receiver on the same antenna as a Septentrio PolaRx4TRPRO timing receiver, our gold standard receiver, in a common-clock (Microsemi 5071A cesium standard) configuration. For the comparison, C/A code measurements made at the L1 frequency are used and a counter/timer with subns resolution is used in conjunction with the NV08C, rather than the FPGA-based counter in the OpenTTP system. The measured noise floor of the NV08C receiver (Figure 4) is about four times higher than in a similar comparison between two high-performance receivers, illustrating one of the compromises made in using a low-cost receiver. Practical time-transfer involves post-processing the raw data with corrections for the effects of the ionosphere and satellite orbits and clocks. This can be done in situ with predictions broadcast by the GPS system, but better accuracy can be obtained using post-processed GPS data products. OpenTTP supports the latter option via generation of suitable Receiver INdependent EXchange (RINEX) observation files [7], but does not currently provide software for the post-processing. CGGTTS format 8

9 data files (a format commonly used in the international timing community [8]) using broadcast GPS data products can be produced via mktimetx, however. The results of single-frequency time-transfer using broadcast clocks and orbits between an OpenTTP unit and a hydrogen maser over a 2000 km baseline are shown in Figure 4. The remote time-transfer receiver is a Septentrio PolaRx5. Version 8 of r2cggtts [9] was used to produce 30 s interval observations which were then compared in common view. Typically, 6 to 8 satellites were visible in common-view. An unweighted average of the differenced observations was used. For comparison, the GPSDO stability is also shown. The timetransfer link reproduces the behaviour of the GPSDO out to averaging times of about 3000 s, but then exhibits a bump, probably due to imperfect modelling of the ionosphere. The accuracy of time-transfer depends, in part, on the calibration of system delays. The procedures for this are straight-forward and well-established. Typically, the delays can be measured with a total uncertainty of about 2 ns. The long-term stability of the measured delay is more difficult to quantify. The Time Department of the International Bureau of Weights and Measures currently assigns a drift of approximately 1 ns/per year to its receiver calibrations, clamped at a maximum of 10 ns. We do not as yet have sufficient experience with the receivers we use in the OpenTTP system to characterize their long-term stability. Figure 4. Time deviation (TDEV) of time-transfer. The combined GPS receiver noise floor, measured on an antenna shared with a Septentrio PolarRx4TRPRO, sets the performance baseline. Practical time-transfer between an H-maser and the OpenTTP GPSDO on a 2000 km baseline is also shown. 9

10 V. Stability and accuracy of the computer s time Time on the system computer is maintained via a software clock in the Linux kernel. Physical clocks such as counters available in the CPU provide a source of ticks for the software clock to count. The frequency and phase offset of the software clock is controllable by user-space applications, allowing the software clock to be synchronized to an external reference. The 1 pps from the GPSDO provides the precise reference for adjusting the system computer s time via a GPIO pin on the BBB. The GPIO input signal is mapped to an interrupt. When an interrupt occurs and is handled by the Linux kernel, the event is timestamped with the current system time. The measured offset between the 1pps and system time can then be used to adjust the software clock. The stability and accuracy of the reference 1 pps, however, is degraded by delays and jitter. This can be mitigated by filtering the measurements to remove outliers. Filtering can either be done in the Linux kernel, or by using ntpd to process the measurements. Generally, the filtering by ntpd is more effective, so ntpd s filtering was selected. To characterize the stability of the system time, we first logged the measurements collected by the PPS device (/dev/pps0) under minimally loaded conditions (all OpenTTP-related processes were disabled) to establish the baseline stability, and then under typical operating conditions (Figure 5). The measurement necessarily includes a component of jitter in fetching of a timestamp so must be considered as an upper bound on the stability. System loading only has a noticeable effect on TDEV at averaging times longer than 1000 s. This is likely due to the action of the kernel s scheduler on long-running processes. There is a knee in TDEV at 16 s because ntpd s adjustments of the time are made every 16 s. Quantifying the accuracy of the system time is not straightforward because the delay between the 1 pps raising an interrupt and the instant it is time stamped, the interrupt latency, is only accessible via software. Measurements made on the BBB with one GPIO being used to generate an interrupt on another GPIO suggest that at a 1 GHz CPU clock frequency, an upper bound on the latency is about 5 10

11 µs [10]. The latency can be compensated via ntpd, which has a facility for delay via its configuration file. The latency will also increase with the system Input/Output load. In Figure 5, a step increase in the NTP load also suggests that the interrupt latency is about 5 µs. Under heavy loading, the total latency is thus about 10 µs. Figure 5. Time deviation (TDEV) of the SBC system time with respect to the GPSDO 1 pps signal, with a minimal I/O load and a normal I/O load In the case of one of the collaborators, a national network of NTP servers is used as a secondary source of time. The records of NTP transactions are retained and these provide additional verification of correct time of day at the level of tens of milliseconds. Finally, time stamps obtained from the host operating system are subject to latency as well. An upper bound on this latency can be obtained by running a program which repeatedly queries the system time in a very tight loop, storing the time stamps for later output and analysis. The latency observed on the BBB is about 1 µs. VI. Effects of NTP load on the system performance In the installations we have made to date, NTP load has been light and has no measurable effect on the accuracy of the system time. Nonetheless, for complete characterization it is necessary to test the system s performance over the full range of operating conditions it might encounter. 11

12 The system behaviour was measured as a function of the NTP load. For the tests, the OpenTTP unit had a direct network connection to the NTP load generator, a Linux PC running a custom program to generate NTP requests. The BBB has Fast Ethernet (100 Mbit/s), so the network interface will be saturated at an NTP load of about packets/s. In the first test, the load was linearly increased to packets/s over a period of two hours. At about packets per second, the unit starts to drop NTP requests. By about packets per second, it is dropping about 1% of NTP requests and the BBB CPU is running at nearly 100% load. No significant change to the stability of the system time was observed during this test. We therefore adopt packets/s as the maximum load for the device. Clearer information about the response of the system time to load can be obtained by a step impulse test. The response of the system time to a step impulse load of NTP requests per second is shown in Figure 6. There is an immediate step of about 5 µs, approximately equal to the interrupt latency, which is then damped out over the next s. When the load is removed at t = s, the expected negative offset is seen. Figure 6. Response of the system time to a step impulse load of NTP requests per second at t = 0 s. The load is removed at t = s. The accuracy of the time as seen by an NTP client can also be measured as a function of server load. The network delay might be expected to increase as the load increases, increasing the formal 12

13 uncertainty. For the test, the unit was polled as the load was linearly ramped as in the first test, and the four NTP timestamps (Section IX) recorded. In each poll, two NTP requests are made, with the first discarded because the first delay tends to be biased high. Figure 7 shows the test results, with the total delay separated into the network delay and server delay. At up to packets/s the network delay stays relatively constant at about 200 µs, while the server delay increases with the load. The uncertainty in the NTP measurement, given by half of the network delay, is therefore independent of load up to our maximum specified load. Figure 7. NTP measurement delays as a function of server load. VII. Evaluation of uncertainties For the outputs of the system to be traceable, we need to assess the measurement uncertainties in those outputs, linked back to the national standard. Two kinds of outputs can be identified: the precise timing outputs of the GPSDO; and time of day, as realized by the BBB kernel clock using those precise timing outputs and made available via NTP. The link back to the national standard is made via GPS time-transfer, as discussed in Section II. The assessment of the uncertainty of GPS time-transfer links has been described elsewhere [11], and we will not examine this in detail. Calibration and measurement capabilities published in the Key Comparison Database by various national metrology institutes claim uncertainties for GPS time transfer between their local UTC realization and a remote clock in the range 10 to 50 ns. Because we 13

14 are principally interested in time of day, where the uncertainties are of the order of microseconds, we will simply note that a 50 ns uncertainty makes a negligible contribution to the overall uncertainty. For time of day applications, we want the uncertainty in the time stamps read from the software clock. Six sources of uncertainty are formally identified (Table 1). To characterize the GPSDO and system time stability, we use TDEV. The choice of 5 µs for TDEV of the software clock is conservative. The antenna and cable delay for the GPSDO is assumed to have been measured and compensated. Combining the various error components in quadrature, the expanded uncertainty in the system time (k = 2) is then 13 µs. Component Raw estimate U i (µs) Reducing factor k i Standard uncertainty u i (µs) GPSDO TDEV GPSDO cable delay software clock TDEV interrupt latency time stamping latency system loading combined uncertainty 6.5 expanded uncertainty 13 Table 1. Uncertainty budget for the system time. VIII. Application: NTP service NTP service is the simplest application of the system. The requirements on accuracy can be quite modest in some cases. One application in the telecommunications industry that we have supported is provision of traceable time for charging and billing of telephone calls. In Australia, there is an industry code of practice that specifies the tolerances on the time stamping of the beginning of a call 14

15 and its duration [12]. The tolerances are more than one second (Table 2), and this requirement is easily satisfied. Parameter Untimed call Timed call Call start time error +8.5,-5.5 s +8.5, -5.5 s Call duration error not applicable +1.5,-2.5 s Table 2. Telephone call charging and billing accuracy IX. Application: auditing of NTP devices Auditing of third party NTP devices is a significant application of the OpenTTP system and one that illustrates the flexibility that is available. A customer may have existing NTP infrastructure that they want to make traceable to the national standard, or may wish to have independent evidence available of correct functioning of that infrastructure. Auditing may also provide evidence of correct operation of dependant NTP clients, where the OpenTTP system is also used to provide time. This is one current application of the system: it provides time to a device in a trading system, and this device is then queried to verify its time is correct. The basic idea is to use NTP to query time on the customer s device. The limitation on the achievable accuracy is the network delay. In an NTP exchange, four time stamps are created: T 1 when the request is sent (according to the client) T 2 when the request is received (according to the server) T 3 when the reply is sent (according to the server) T 4 when the reply is received (according to the client) These four timestamps allow an estimate of the round trip network delay δ = (T 4 T 1 ) (T 3 T 2 ), (1) 15

16 where the delay T 3 T 2 can be identified as the processing delay at the server. The mean client-server offset (Δ) is = [(T 2 T 1 ) + (T 3 T 4 )]/2 (2) The estimate of the client-server offset assumes that the network delays in each direction are symmetric and consequently cancel out. The formal uncertainty in the offset is therefore half of the round trip delay. In the above, the small correction to the processing delay T 3 T 2 due to the frequency offset of the remote device is neglected. From the data in Figure 7, with a processing delay of 10 ms, the correction is only 1 µs for a fractional frequency offset of 1 part in 10 4, which is negligible compared with the network delay of 200 µs. The data collected during NTP auditing can be messy and therefore challenging to analyse and interpret. For example, if an audited device is rebooted there will be transient, non-representative behaviour. Information in the standard NTP response about server stratum and accuracy can be used to identify and filter out such events. Figures 8 to 10 show typical data obtained for NTP measurements on a GPS-referenced NTP server connected to the auditing system via a network switch. From the manufacturer s specifications, NTP time on the server is expected to be within 10 µs of UTC. The NTP server was polled every two minutes. In Figure 8, maximum and minimum offsets are determined by assuming all of the delay appears in either the forward or return path. Asymmetry in this very simple network setup is evident, along with bimodal structure in the one way network delay (Fig. 10) and skewing of the distribution of the mean offset (Fig. 9). The uncertainty in this example has a systematic component of about -70 µs (from the mean offset) and a statistical component of about ±150 µs (from the one way network delay). 16

17 Figure 8. Maximum and minimum offsets of a GPS-referenced NTP server, measured using NTP. Figure 9. Distribution of the mean offset of a GPS-referenced NTP server, measured using NTP. 17

18 Figure 10. Distribution of the one-way network delay to a GPS-referenced NTP server, measured using NTP. The NTP measurement performance can be evaluated in the context of synchronization tolerances applying to transactions taking place in selected financial regulatory environments. With the exception of a 100 µs tolerance required in some situations by MiFID II, the observed 150 µs uncertainty is sufficient to satisfy typical requirements (Table 3). Market Regulator Document Tolerance (ms) USA FINRA Regulatory Notice Australia ASIC Market Integrity Rules 20 (Securities Markets) European ESMA MiFID II RTS ,1 Union Table 3. Synchronization requirements in selected financial markets. The 100 µs tolerance is required where the gateway to gateway latency of the trading system is less than 1 ms - an example of such a system is a high frequency algorithmic trading system. In this application, direct measurement of a signal such as 1 pps would be necessary; the OpenTTP system has an external 1 pps input for this purpose. 18

19 From a risk management perspective, operation in passive mode as an auditing device has the advantages that the auditing device is not as critical a piece of infrastructure as the NTP devices themselves and change management such as the updating of software is simpler. A less complicated user interface is also required, reducing the need for development and maintenance of non-core software. One further example of an auditing application is worth mentioning because it emphasizes the flexibility of our system. In the application, the audited device is NTP-synchronized but cannot respond to an NTP query. Instead, the device s time can be queried via a HTTP-based management interface. A simple program, installed on our system, can generate the necessary requests, with an NTP-like measure of the round trip delay to estimate the uncertainty. Easy customization is one of the key strengths of OpenTTP. X. Conclusion The OpenTTP system has been described and its operation characterized from a metrological perspective. While it does not have state of the art time-transfer performance, it is suitable for the medium accuracy (> 100 ns) applications we see in telecommunications and finance. Its development is ongoing, and we encourage the timing community to participate in its further evolution. Acknowledgments The Asia Pacific Metrology Programme (APMP) generously provided funding for the development and manufacture of prototype hardware through a Technical Committee Initiative supported by the APMP Technical Committee for Time and Frequency. References M. J. Wouters, E. L. Marais, P. T. H. Fisk, R. B. Warrington, Development and Applications of a Traceable Time-Transfer System, in Proc. of 28 th European Frequency and Time Forum, Neuchâtel, Switzerland, 2014, pp

20 3. JCGM 200:2012, International vocabulary of metrology Basic and general concepts and associated terms (VIM), 3 rd ed., W. Allan and M. Weiss, Accurate time and frequency transfer during common-view of a GPS satellite in Proc. of the 34th Annual Frequency Control Symposium, Philadelphia, Pennsylvania, USA, 1980, pp Z. Jiang, A. Czubla, J. Nawrocki, W. Lewandowski and E. F. Arias, Comparing a GPS time link calibration with an optical fibre self-calibration with 200 ps accuracy, Metrologia, vol. 52, 2015, pp G. Petit, J. Leute, S. Loyer, and F. Perosanz, Sub frequency transfer with IPPP: Recent results, Proc. of 31 st European Frequency and Time Forum, Besancon, France, 2017, pp RINEX the Receiver Independent Exchange Format v3.02, P. Defraigne and G. Petit, CGGTTS-Version 2E : an extended standard for GNSS Time Transfer, Metrologia, vol. 52, 2015, pp. G1-G22 9. P. Defraigne, and K. Verhasselt, Multi-GNSS time transfer with CGGTTS-V2E, in Proc. of 32 nd European Frequency and Time Forum, Turin, Italy, APMP TCTF CMC Guideline Remote Frequency Standards, Australian Communications Industry Forum Industry Code ACIF C518:2006 Call charging and billing accuracy. 20