Mobile stereo video broadcast

Transcription

1 Mobile stereo video broadcast Gozde B Akar M. Oguz Bici Anil Aksay Antti Tikanmäki Atanas Gotchev 1 MOBILE3DTV project has received funding from the European Community s ICT programme in the context of the Seventh Framework Programme (FP7/ ) under grant agreement n This document reflects only the authors views and the Community or other project partners are not liable for any use that may be made of the information contained therein.

2 MOBILE3DTV Project No Mobile stereo video broadcast Gozde B Akar, M. Oguz Bici, Anil Aksay, Antti Tikanmäki, Atanas Gotchev Abstract: In this report, the DVB-H channel is studied for its capabilities to transmit stereo video in different formats and to provide adequate error protection for such content. The following stereo video representations are considered: two channel stereo, encoded either independently with H.264 encoder (simulcast) or by multi-view coding (MVC) and video plus depth both encoded by H.264. These encoding approaches are implemented and combined with an IP streaming application. DVB-H specific tools, such as MPE-FEC encapsulator and decapsulator and physical layer simulator complement the software environment built to run end-to-end video transmission simulations. In the experimental tests, various physical channel models are used to represent different reception conditions. These are put against different stereo video encoding modes and MPE-FEC rates. The preliminary results show the feasibility of the DVB-H channel to handle encoded stereo video streams. They also illustrate the problems of transmission in bad channel conditions, which should be addressed by optimizing the MPE-FEC rates and developing novel application layer error protection schemes Keywords: Error characteristics, stereo, error resilience, 2D+depth

3 Executive Summary The DVB-H standard has been developed to provide broad and flexible broadcast of reach data and to overcome the challenges of mobile reception environment. While largely based on the terrestrial digital video broadcasting (DVB-T) standard, it also introduces time-slicing for lower power consumption and special error protection through combining multiple protocol encapsulation with forward error correction codes (MPE-FEC). As such, it looks quite capable of broadcasting 3D video to mobile users. In this report, the DVB-H channel is studied for its capabilities to transmit stereo video in different formats and to provide adequate error protection for such content. In order to make the content self-readable, the report incorporates parts of the previous report on most typical error patterns of the DVB-H channel. The following questions are addressed and studied in the report: ensuring transmission of stereo video in different formats by designing a proper streamer for such content; building real physical channel and designing simulation tools of the link layer and physical layer; performing simulation tests for different channel conditions and video formats. The following stereo video representations are considered: two channel stereo, encoded either independently with H.264 encoder (simulcast) or by multi-view coding (MVC) and video plus depth both encoded by H.264. These encoding approaches are implemented and combined with an IP streaming application. DVB-H specific tools such as MPE-FEC encapsulator and decapsulator and physical layer simulator have been developed and are described in the report. Thus, a complete software environment has been built to run end-to-end video transmission simulations. It accompanies the fully operational DVB-H broadcast system set up at Tampere University of Technology (TUT). The system encompasses the whole broadcast chain including encoders, encapsulator, transmitter and receivers. In the experimental tests, various physical channel models have been used to represent different reception conditions. These are put against different stereo video encoding modes and MPE-FEC rates. While these preliminary results show the feasibility of the DVB-H channel to handle encoded stereo video streams, they also show that optimal MPE-FEC rates and other application layer error protection schemes have to be studied in order to ensure an adequate delivery of 3D video content over the targeted system. 2

4 Table of Contents Executive Summary... 2 Table of Contents Introduction Overview of DVB-H System Broadcasting of 3D Video in Different Formats D Video Characteristics Characteristics of two simulcast video Characteristics of joint coded stereo video Characteristics of 2D video+depth data D Video Coder D Video Streamer/Client D Video Decoder DVB-H Transmission Channel Recommended Transmission Modes DVB-H Broadcasting System for Experiments and Demonstrations DVB-H Channel Simulation Physical Channel simulator Physical layer experiments Simulation of 3D Coded Video Transmission over DVB-H Simulation Environment Experimental Results Conclusion References

5 1. Introduction Currently, there are a few candidate formats for delivery and storage of 3D video. The first alternative is to transmit a pair of stereoscopic views captured by stereoscopic cameras. These views can be either coded separately, also known as simulcast [5], or they can be coded together using a multi-view codec such as the MVC extension of H.264 [7]. Multi-view codecs achieve higher coding efficiency than simulcast by exploiting the inter-view correlation in compression. Alternatively, the 3D video can be represented as a conventional monoscopic video sequence that is augmented with depth information. In this case, the receiver has to take care of generating the views appropriate for the display device. There are three main ways to encode these representations as shown in Figure 1. In all these case, there is a different dependency between streams compared to the monoscopic video. Figure 1 Different ways to encode 3d data On the other hand, wireless networks are often error prone due to factors such as multipath fading and interferences. In addition, the channel conditions of these networks are often nonstationary, such that the available bandwidth and channel error rates are changing over time with large variations. In order to maintain satisfactory Quality of Service (QoS), a number of technologies have been proposed targeting different layers of the networks. Concentrating more on DVB-H, it uses Forward Error Correction (FEC) for error protection and comes with an optional FEC tool at the link-layer. This FEC uses Reed-Solomon (RS) FEC codes encapsulated into Multi-protocol encapsulated sections (MPE-FEC). The MPE-FEC was also introduced to provide additional robustness required for hand-held mobile terminals. MPE-FEC improves the carrier-to-noise (C/N) and Doppler performance in the DVB-H channel while also providing improved tolerance of impulse interference. However, MPE-FEC might fail in the presence of very erroneous conditions. Using a-priori knowledge of the transmitted media and tuning the way MPE-FEC is applied across the media, datagrams can provide better robustness. Since 3D data has a different layered structure and different dependencies between layers, this can be intelligently addressed to provide better robustness. In the following sections, we first give a brief overview of DVB-H system. Then we summarize different 3D representations that can be used for DVB-H transmission, the dependencies in these representations and the effect of error in the overall quality. In sections 3 and 4, we overview the DVB-H transmission channel and the simulation results. Finally, we present the preliminary results that show the effect of MPE-FEC on 3D coded video. We conclude the report by introducing ideas for more robust transmission over DVB-H. 4

6 2. Overview of DVB-H System A conceptual description of using a DVB-H system is shown in Figure 2. In the transmitter, the IP input streams coming from different sources as single elementary streams are multiplexed according to the time slicing method. The MPE-FEC error protection is calculated and added separately for each elementary stream. Afterwards encapsulation of IP packets and embedding into the transport stream follow. The next block is the DVB-T modulator. In addition to the 2K and 8K modes DVB-T provides, DVB-H also uses an intermediate 4K mode with a 4096-point Fast Fourier Transform (FFT) in the OFDM modulation. The objective of the 4K mode is to improve the network planning flexibility by trading off mobility and SFN size. To further improve robustness of the DVB-T 2K and 4K modes in a mobile environment and impulse noise reception conditions, an in-depth symbol interleaver is also standardized. Figure 2 A conceptual description of using a DVB-H system [5]. The objective of time slicing is to reduce the average power consumption of the terminal and enable smooth and seamless service handover. Time slicing consists of sending data in bursts using a significantly higher instantaneous bitrate compared to the bitrate required if the data were transmitted using traditional streaming mechanisms. To indicate to the receiver when to expect the next burst, the time (delta-t) to the beginning of the next burst is indicated within the burst currently being received. Between the bursts, data of the elementary stream is not transmitted, allowing other elementary streams to share the capacity otherwise allocated. Time slicing enables a receiver to stay active for only a fraction of the time, i.e. when receiving bursts of a requested service. Note that the transmitter is constantly on (i.e. the transmission of the transport stream is never interrupted). Time slicing also supports the possibility to use the receiver to 5

7 monitor neighbouring cells during the off-times (between bursts). By accomplishing the switching of the reception from one transport stream to another during an off period it is possible to accomplish a quasi-optimum handover decision as well as seamless service handover. The objective of the MPE-FEC is to improve the C/N and Doppler performance in mobile channels and to improve the tolerance to impulse interference. This is accomplished through the introduction of an additional level of error correction at the MPE layer. By adding parity information calculated from the datagrams and sending this parity data in separate MPE-FEC sections (Figure 1Figure 3), error-free datagrams can be output (after MPE-FEC decoding) even under bad reception conditions. With MPE-FEC, a flexible amount of the transmission capacity is allocated to parity overhead. For a given set of transmission parameters providing 25 % of parity overhead, the receiver with MPE-FEC may require about the same C/N as a receiver with antenna diversity and without MPE-FEC. The MPE-FEC overhead can be fully compensated by choosing a slightly weaker transmission code rate, while still providing far better performance than DVB-T (without MPE-FEC) for the same throughput. Figure 3 The structure of MPE-FEC frame [11]. 3. Broadcasting of 3D Video in Different Formats D Video Characteristics In this section we briefly describe the characteristics of the different 3D representations to show their effects on decoder complexity and transmission errors Characteristics of two simulcast video Simulcast coding is the process of coding each view independent of each other. It only utilizes the correlation between adjacent frames in a single stream. However the correlation between adjacent streams (inter-view correlation) is not utilized. This type of coding is backward compatible, i.e. it is possible to decode both streams using conventional 2-D video decoders. In 6

8 addition, extra decoder complexity is eliminated. The effect of error on the transmitted streams is similar to 2D video streaming, however, the overall quality is related to how the human visual system responds to the artefacts in either views Characteristics of joint coded stereo video Stereo video is a special case of multiview video with the number of views equals to two. In stereo video coding both the temporal and inter-view prediction is utilized. One of the views (usually left view) is coded without reference to the right view using a conventional 2D video codec. This guarantees backward compatibility. The right view is coded using both inter-view prediction (with reference to the left view) and temporal prediction. Joint coding of stereo video utilizes the bandwidth more efficiently with a slight decrease in the Peak Signal-to-Noise Ratio (PSNR). Table 1 shows the PSNR (db) and bitrate values for the original, simulcast and joint coded stereoscopic sequence Botanic [3]. Algorithm Normalized bitrate PSNR overall PSNR left PSNR right Original N/A INF INF INF Simulcast Joint Table 1 Quality and bitrate differences for simulcast and joint coded stereo video. However, since the right view is predicted from the left view, any error on the left view will affect the quality of the right view directly, thus decreasing the overall quality Characteristics of 2D video+depth data 3D video can also be represented by 2D video signal and a per sample depth map. Per sample depth map determines the position of the associated colour data in the 3D space. Using a single 2D video signal and depth map, a stereo view can be rendered at the receiver. The coding of such a representation can be achieved by conventional 2D video encoders since the depth map can be assumed as a monochromatic video signal. In this case, similar to simulcast video, 2D video and depth video can be coded as two independent video streams. The reconstructed quality of the stereo video in the receiver side is heavily dependent on the accuracy of the depth map, so any error introduced on the depth map during transmission will have a severe effect on the reconstruction quality. In Figure 4, an example is given to show the importance of the depth map. 7

9 (a) (b) (c) (d) (e) Figure 4 (a) The 2D video sequence (left view) and associated depth map, (b) A zoomed segment of rendered right view using the original depth map, (c) A zoomed segment of depth map where some blocks are lost during transmission, (d) A zoomed segment of rendered right view using the depth map in (c). Since some depth values are missing, these pixels are filled by the values in the left view. (e)) Difference between (b) and (d). 8

10 3.2. 3D Video Coder As stated before, there are a few candidate formats for delivery and storage of 3D video: Simulcast coding, joint coding (MVC coding), coding as Video-plus-depth (2D+depth). Hence, there are three main ways to encode these representations. Our system supports all of them. a) Encoding both views independently using H.264/AVC. Our system uses H.264/AVC reference encoder version 10.1 [8]. b) Video-plus-depth (2D+depth) (VPD) representation is coded by independent coding of video and depth signal by H.264/AVC with small information about depth data embedded into the high-level syntax. MPEG specified a corresponding container format ISO/IEC Auxiliary Video Data Representations, also known as MPEG-C Part 3 [9]. Similarly same encoder is used with extra syntax. c) Exploiting temporal and inter-view redundancy by interleaving camera views and coding using a hierarchical manner with some Multi-View Video Coding (MVC) specific tools like illumination and color compensation, improved disparity estimation and coding and some high-level syntax changes. MVC is an amendment (Amendment 4) to H.264/AVC [10]. MVC encoder used in our system is JMVM Main prediction structure is quite complex introducing a lot of dependencies between images and views. This is valid even if the number of views to be coded is 2 (stereoscopic video). These dependencies make use of the redundancies present in both spatial and temporal directions to reduce the bitrate, however they also impose many restrictions in decoding and packet loss sensitivity. An alternative simplified structure is presented in [13], and shown to be very close to the main prediction structure in terms of overall coding efficiency. In this simplified prediction structure which is used in this study, the temporal prediction using hierarchical B-pictures remains unchanged when compared to original MVC prediction structure, but spatial references are only limited to anchor frames, such that spatial references are only allowed at the beginning of a group of pictures (GOP) between I and P pictures. This simplified version is shown in Figure 5 when the number of views = 2. Figure 5 Simplified MVC coding scheme in case of two views D Video Streamer/Client 9

11 In the current streaming structure, video coder produces coded frames which are sent to the Network Abstraction Layer (NAL). A NAL Unit (NALU) consists of one byte header followed by a bit-stream that contains macroblocks of a frame or a slice. NALU type specifies which type of data structure is contained in NALU. NALU type for MVC data is different than monoscopic video. The NALUs that are produced are then sent to the RTP packetizer (Figure 6). The streaming of H.264/AVC streams over RTP is standardized by the IETF in RFC 3984 [15], which defines the RTP header usage and necessary packetization rules for H.264/AVC. RFC 3984 is perfectly applicable for the streaming of MVC bit streams as a whole, because they are H.264/AVC compatible. Because of this, for simulcast coded representations (Two independent video streams or 2D+depth data), [15] is used. For MVC data, stream packetization is applied in a similar fashion to [15] with some modifications as proposed in [3] and [16]. Instead of sending the MVC stream as a single H.264/AVC stream over RTP, we break up the stream into several parts. NALUs belonging to different views are streamed over different UDP port pairs as if they were separate H.264/AVC streams. For synchronization, the time stamps corresponding to the left and right frames at the same time should be the same. Contrary to simulcast streaming, only the stream for the independently encoded view will be decodable by itself in the case of MVC streaming. The final step of the streamer is the UDP/IP packetization. IP datagrams corresponding to right and left views are encapsulated in the link layer and put into different elementary streams with different particular program identifiers. RTPheader NALU 11 RTPheader NALU 12 RTPheader NALU n1 RTPheader NALU n2 RTPheader NALU n3 RTP packets corresponding to NALU 1 header coded data header coded data header coded data header coded data NALU 1 NALU 2 NALU n-1 NALU n Figure 6 RTP packetization D Video Decoder We are currently using FFMPEG library [14] for real-time decoding of H.264/AVC streams. In order to decode MVC streams, we modified FFMPEG library with the appropriate syntax changes and interleaving of two views. First of all, DPB (Decoded Picture Buffer) size is increased since MVC prediction requires more pictures to be held in the buffer. There are some modifications in SPS (Sequence Parameter Set) to signal for the prediction structure and to signal the MVC encoding and modifications in NALU header to signal for the view ID and related other information. Since prediction of each frame and memory management depends on view ID of the frame, view ID tag is added to each frame in DPB and related functions such as prediction list generation, buffer management are modified accordingly. Support for new picture reordering commands is also implemented to modify the list more efficiently for inter-view prediction. Since MVC required the base-view to be standard compatible, view ID and related other information cannot be signalled in the NALU 10

12 header of the base-view NALU. It is decided to have another NALU following each NALU of base-view with the required information which is called suffix NALU. Also for compatibility reasons, B-pictures that are marked as non-reference for the base-view can be used for inter-view prediction and needs to be stored in DPB. Finally illumination compensation is added into the motion/disparity compensation module. In order to cope with losses, decoder examines picture order count (POC) of each frame and identifies missing frames. For 2D+depth decoding, again, FFMPEG library is used for real-time decoding of H.264/AVC streams. Once decoded, the stereo view is generated using the depth-based-image renderer [19]. 4. DVB-H Transmission Channel 4.1. Recommended Transmission Modes The physical layer operation of DVB-H is mostly inherited from DVB-T although DVB-H adds a few new features that cannot be used in DVB-T broadcasts. Both of these technologies are specified in ETSI EN standard [20]. The new DVB-H specific transmission modes, 4K mode, 5MHz bandwidth and in-depth interleaver are defined in annexes F and G of the standard. The transmitter operations for creating the DVB-H baseband signal from MPEG Transport Stream (TS) are briefly explained below: 1. Transport stream is randomized and adapted for energy dispersal 2. Outer coder. A shortened Reed-Solomon RS (204,188) code that allows correction of up to 8 erroneous bytes at the receiver is applied to each randomized TS packet 3. Outer (bytewise) interleaver 4. Inner, convolutional coder with code rate of 1/2. The code can be punctured to code rates 2/3, 3/4, 5/6, and 7/8, which provide higher bit rate for application data with the tradeoff of having lower error correction capability. 5. Inner bitwise interleaver and symbol interleaver. The DVB-H specific in-depth interleaving mode can be used instead of the 'native' DVB-T interleaver to gain some additional protection against impulsive interference. 6. The output of symbol interleaver is mapped to the carriers of the OFDM frame according to the constellation in use. If hierarchical modulation is used, it is possible to multiplex two separate transport streams in the high and low priority parts of the broadcast. 7. Pilot and TPS carriers that aid the receiver devices are inserted 8. OFDM 9. Guard interval insertion There are two coders at the physical layer adding redundant error correction information to the transmission. The MPE-FEC mechanism at the DVB-H link-layer acts as a third, additional layer of protection. The constellation, convolutional code rate and MPE-FEC code rate are the parameters having most significant impact on the channel C/N performance. Simpler constellations (QPSK) are more robust against channel errors than the more complex ones (64-11

13 QAM), stronger FEC codes naturally improve the reception performance under difficult conditions. The Doppler performance is affected also by the OFDM mode. [22] There are altogether five tunable physical layer parameters: constellation (QPSK, 16-QAM, 64- QAM), OFDM mode (2K, 4K, 8K), code rate of the convolutional coder (1/2, 2/3, 3/4, 5/6, 7/8), inner interleaver mode (native, in-depth), bandwidth (5-8MHz), and guard interval (1/4, 1/8, 1/16, 1/). Hierarchical transmission mode adds even more alternative ways to construct the broadcast. The large amount of different modulation parameters and their combinations gives flexibility to network planning but makes it more difficult to find the optimal transmission mode. Wing-TV project [23] studied the physical, link and application layer performances of different DVB-H transmission modes, as well as developed new physical channel models and more efficient methods for MPE-FEC decoding. The project used channel simulations and field experiments for comparison of C/N and Doppler performance of the different constellations and various combinations of MPE-FEC and convolutional code rates. In the study, the following transmission modes, which provided the best performance for the corresponding usable bit rate, were recommended: Constellation Convolutional code rate MPE-FEC code rate QPSK 1/2 3/ QPSK 1/2 5/ QPSK 2/3 5/ QAM 1/2 3/ QAM 1/2 5/6 8. IP bit rate (Mbps) 16-QAM 2/3 5/ Table 2 Recommended DVB-H transmission modes If high-speed reception (Doppler frequency 80Hz) is desired, only the QPSK modes and 16- QAM with convolutional code rate ½ provide adequate performance. Additionally, the study showed that, at low Doppler frequencies, also the 64-QAM mode with code rate (2/3, 5/6) could provide acceptable quality. As a rule of thumb, they suggest that when choosing the code rates for the convolutional coder and MPE-FEC, the best overall performance can be achieved by spending most of the redundant data in the convolutional coder and using relatively weaker RS code for the MPE-FEC. [21][23] The bit rate requirements within the Mobile3DTV project are moderate as transmission of more than a few programs won't be needed. Therefore, one of the more robust QPSK or 16-QAM modes can be selected for use in future experiments and demonstrations. The optimal MPE-FEC code rate as well as various UEP schemes for 3D video transmission will be studied later on in the project. 12

14 4.2. DVB-H Broadcasting System for Experiments and Demonstrations The project members have various DVB-H broadcasting equipment and software tools for performing field and laboratory experiments as well as for building demonstrations of end-to-end transmission systems. There is a full featured transmitter setup at TUT that can be used for field tests [Figure 7]. It consists of proprietary broadcasting equipment. It is equipped with 50W power amplifiers and rooftop antennas, and it is capable of delivering DVB-H broadcasts within the town district of Hervanta. The DVB-H playout server (Cardinal) is used for creating timesliced transport streams of A/V content in MPEG-4 files or live IP streams. File delivery over FLUTE data carousels is also supported. The playout server can create all the PSI/SI tables as well as Electronic Service Guide (ESG) files needed for a complete DVB-H service. The transmitted streams can be recorded for future use, which provides means for using the playout server for creating TS files for simulations. Transmitting complete, pre-generated or recorded transport streams is also possible. RF MPEG-2 Transport Stream Source content (.mp4) RTP Streamer IP Network Playout server Cardinal Compact for DVB-H Exciter Rohde & Schwarz Power amplifier Rohde & Schwarz Figure 7 DVB-H Transmitter setup The DVB-H exciter takes the transport stream as its input and creates an OFDM modulated DVB signal to be broadcasted by the power amplifier and rooftop antenna. Optionally, it is possible to utilize also a diversity unit that prepares the RF signal for two-antenna transmitter diversity broadcasting. For laboratory experiments and demonstrations, TUT, METU, and MMS have all obtained a PC based transmitter setup that uses a DVB-H modulator card (Dektec DTA-115/DTA-110) for creating the DVB signal [Figure 8]. A set of open-source software tools (FATCAPS, JustDVB- IT) is used for creating the transport stream. Currently, this does not offer as full-featured solution as the Cardinal playout server because the ESG data cannot be automatically created. On the other hand, the open-source tools offer more flexibility in experimenting with new source/channel coding or content delivery schemes. In the link layer, the IP datagrams are first read column-wise into the application data table part of the MPE-FEC frame, which consists of a total of 191 columns. The application data table is encoded row-wise using a systematic Reed-Solomon (RS) code; the resulting 64 correction bytes per row are then added to the RS data table part of the MPE-FEC frame. The MPE-FEC code rate (cr) can be adjusted by either zero-padding the application data table, or puncturing the RS data table. After the MPE-FEC frame is constructed, the application data table IP datagrams are encapsulated into MPE sections. In a similar fashion, the RS data table columns are encapsulated into MPE-FEC sections. A stream of MPE and MPE-FEC sections are then put into an 13

15 Elementary Stream (ES), i.e., a stream of MPEG-2 TS packets with a Particular Program Identifier (PID). The ESs are time sliced for transmission. The concept of time slicing is to send ESs in bursts using significantly higher bitrate compared to the bitrate required if the data was transmitted using conventional bandwidth management. Within a burst, the time before the start of the next burst (delta-t) is indicated. Between the bursts, data of the ES is not transmitted, allowing other ESs to use the bandwidth otherwise allocated. This enables a receiver to stay active only for a fragment of the time, while receiving bursts of a requested service, which enables considerable power savings. In order to realize DVB-H link layer, we used FATCAPS: A Free, Linux-Based Open-Source DVB-H IP-Encapsulator [25]. The FATCAPS implementation builds on JustDVB-IT [26], which is another open source project allowing for establishing a low-cost, highly configurable environment for DVB-T playout as well as possessing a set of tools to create, transform and multiplex MPEG transport streams, and adds the DVB-H specific features. The FATCAPS software reads IP packets, encapsulates in MPE sections and then in MPEG-2 transport stream packets. Finally, the timeslicer component of the software collects the different DVB-H transport streams and multiplexes them using a time division mechanism, so that only one DVB-H stream is seen on the channel at any given time. MPE-FEC sections can be optionally generated and inserted in the stream at this stage in order to provide additional error protection. The final output can be forwarded to a hardware modulator to perform real broadcasting. In our implementation, we have also modified the FATCAPS software for offline simulation purposes. Current implementation of the software captures IP datagrams from a streamer and encapsulates for transmission in real-time. In our modified implementation, stored IP datagrams can be fed to the software with a given rate and output TS packets can be stored offline. In this way, TS packet losses simulations can be performed. RF Source content (.mp4) RTP Streamer IP Network IP encapsulator software FATCAPS Modulator card Dektec MPEG-2 Transport Stream Figure 8 DVB-H Transmitter setup The DVB-H terminal devices available on consumer market can be used for viewing the broadcast as long as the TS contains the full ESG data and the broadcasted audio and video streams follow the profile and level restrictions imposed by the terminal device. At TUT, we have Nokia models N92 and N77 which could be used as legacy clients that receive the backward compatible part of the 3D video transmission. In practice this is difficult since the devices do not support video resolutions in the range that is considered for the 3D video. 14

16 For research and development work, a PC based receiver offers greater flexibility. For this, we use Linux computers equipped with DVB-H/T frontends. Linux operating system has built-in support for decapsulating the IP data in DVB-H transport streams (dvbnet). Alternatively, the streams can be decoded using the Decaps software [27] developed within this project. Compared to dvbnet, it adds several features such as MPE-FEC error correction and collection of error statistics. The IP decapsulator performs the reverse operations of IP encapsulator. It extracts MPE and MPE-FEC sections from the TS. Errors or erasures in the TS can be reliably detected by the CRC- code included in each section. The data and RS tables of an MPE-FEC frame are filled with the correctly received sections, and RS decoding is performed to recover the lost data. Integrity of the recovered data finally verified using the UDP checksum. After the error correction, the IP datagrams are fed to the stereo video client, either over a network connection or by saving them to a file. In our system, we implemented the IP decapsulator according to the specifications given in [][]. Video players with RTP support can be used to view the decapsulated broadcasts. The software components of the Linux receiver setup are illustrated in Figure 9. RF DVB-T/H receiver Linux DVB API dvbnet kernel decapsulation TS capture IP Network RTP client IP Decapsulator MPEG-2 Transport Stream Packet data 5. DVB-H Channel Simulation Figure 9 Linux based DVB-H receiver 5.1. Physical Channel simulator For simulating the physical transmission channel, we have a MATLAB/Simulink tool that models the DVB-T/H modulation and demodulation processes and the physical transmission channel as illustrated in Figure 10. The channel is modeled as multipath Rayleigh fading channel with additive white gaussian noise. Various commonly used channel models have been predefined in the simulator (Table 3). This tool can be used to collect reception statistics such as Bit Error Rate (BER), TS-Packet Error Rate (TS-PER), Average Burst Error Length (ABEL) and Variance in Burst Error Length (VBEL) as well as to record packet error traces for video transmission simulations. 15

17 Digital Video Broadcasting 8MHz, 2/8k Mode, M-QAM, guard interval, Nonhierarchical Transmission Display 0 2 Tf :[ ] Probe [10240x1] Random Integer [188x1] [204x1] [188x1] RS Encoder (204,188 ) Shortened RS Encoder Convolutional Interleaver I=12 [204x1] [204x1] Punctured Convolutional Code [2448x1] DVB Inner Interleaver [24192x1] DVB M-QAM Mapper [6048x1] TPS & Pilot Insertion [8192x1] OFDM [8192x1] Guard interval Insertion [10240x1] [10240x1] [188x1] Tx [188x1] [188x1] Rx received _ts _stream [188x1] packet error trace Tx Rx BER 3 0 BER 0 Total Errors e+006 Total Bits [204x1] [204x1] Tx Rx BER e+006 BER Total Errors Total Bits Channel Model for Dynamic multipath fading [10240x1] [188x1] Tx Error Rate received _ts _streamcalculation Rx [188x1] 3 0 SER 0 Total Errors e+005 Total Bytes [204x1] [204x1] Tx Error Rate Calculation Rx e+005 SER Total Errors Total Bytes Delayed Scatter Plot [6048x1] B-FFT Spectrum Scope [10240x1] received _ts _stream [188x1] RS Decoder [204x1] Convolutional Deinterleaver I=12 [204x1] [204x1] Viterbi Decoder [2448x1] DVB Inner [24192x1] Deinterleaver DVB Demapper [6048x1] [6048x1] TPS & Pilot Removal [6817x1] Pilot Processing [8192x1] OFDM Receiver [8192x1] Guard interval Removal [10240x1] (204,188 ) Shortened RS Decoder Demapper TPS &Pilot removal 1 Figure 10 Simulink model of DVB transmission COST 207 o Typical Urban (6 and 12 taps) o Bad Urban (6 and 12 taps) o Rural Area (4 and 6 taps) o Hilly Terrain (6 and 12 taps) JTC o Indoor Commercial (7 taps) o Outdoor Residential (10 taps) Wing-TV o Pedestrian Indoor o Pedestrian Outdoor o Vehicular Urban o Motorway Table 3 Physical channel models 5.2. Physical layer experiments The Simulink model was used to simulate the DVB-H transmission channel behaviour with two physical channel models. A mobile use case is modelled with Typical Urban channel having maximum Doppler frequency of Hz. Residential Outdoor channel with maximum Doppler of 1Hz represents situation closer to conventional reception of terrestrial using rooftop antenna. The 16

18 BER TS-PER aim of this study was to compare the channel performances and to see whether the mobility of receiver devices introduces characteristically different errors also at the application layer. Both channel models were tested using the same DVB-H transmission mode: Constellation 16-QAM Convolutional code rate 2/3 OFDM mode 8K Bandwidth 8MHz Interleaver native The channel was simulated using SNR levels ranging from 11dB to 24 db. The simulation results are illustrated as bit error rates and packet error rates in Figure 11. At low SNR levels both channel models show similar performance. At high SNR levels, Residential Outdoor channel has lower error rates while Typical Urban continues to suffer of errors caused by the Doppler shift tu6 residential 10 0 tu6 residential SNR (db) SNR (db) Figure 11 Bit error rate and transport stream packet error rate for both channels Besides error rates, the temporal distribution of errors plays a crucial role in real performance. Because MPE and MPE-FEC sections are transmitted in adjacent TS packets, dispersed errors may cause more sections to have a few errors, whereas dense error bursts would result in few sections being completely lost. If erroneous sections are marked wholly lost in MPE-FEC decoding as proposed in the standard (known as packet erasure method), the end result is that more dispersed errors cause more severe packet loss at the application layer. Figure 12 shows that although both channel models have similar average length of error bursts the variance of burst length is much higher in the Residential Outdoor channel. This is caused by relatively few very long error bursts. 17

19 Average error burst length (TS packets) Variance of error burst length 25 tu6 residential tu6 residential SNR (db) SNR (db) Figure 12 Transport stream burst error length statistics The application layer performance was approximated by mapping the transport stream error trace to an IP packet error trace, under the assumption that all IP packets are of the same size which is also the number of rows in the MPE-FEC frame. These assumptions simplify simulation of the FEC decoding because both IP packets and RS data sections map directly to columns of the MPE-FEC frame and the so-called column erasure decoding method can be used. MPE-FEC code rate 3/4 was used in the simulation (191 application data columns, 64 RS columns). The IP packet error rates before and after FEC decoding are shown for both of the channel models in Figure 13. In Typical Urban channel, a clear improvement of PER at SNR values from 16 to 22 db can be seen, whereas with the Residential Outdoor channel the impact is negligible. These results show that MPE-FEC is quite effective against the dispersed errors caused by Doppler shift. The MPE-FEC frame size also affects the error correction performance as it is directly related to the interleaving depth. This is, however, not studied within this report. 18

20 IP-PER IP-PER Typical Urban (6 taps) Before MPE-FEC After MPE-FEC Residential Outdoor (10 taps) SNR SNR Figure 13 Application layer performance with and without MPE-FEC 6. Simulation of 3D Coded Video Transmission over DVB-H 6.1. Simulation Environment The transmission simulation environment is based on the same open-source tools, Fatcaps and Decaps, as the Linux based transmitter and receiver toolset. When a measured or simulated packet error pattern is applied to the transport stream, the system forms an end-to-end simulation of the DVB-H transmission chain. The simulator can be used for both online and offline simulations. In the online case, live IP streams are encapsulated in transport streams and sent back to IP network by decapsulator. In offline simulations the different phases can be separated from each other and it is not necessary to use live IP streams as input and output. For instance, one can first generate a transport stream that is then passed through various different channel conditions. The building blocks of our system can be seen in Figure 14. The input videos (Stereo video content with right and left view or 2D video+depth) are first compressed with a 3D encoder (2D video encoder for simulcast coding or stereo encoder). Resulting NALUs are fed to the 3D video streamer. The streamer encapsulates the NALUs into Real Time Transport Protocol (RTP), User Datagram Protocol (UDP) and finally Internet Protocol (IP) datagrams. The resulting IP datagrams are encapsulated in the DVB-H link layer where the Multi Protocol Encapsulation Forward Error Correction (MPE-FEC) and time slicing occurs. The link layer output MPEG-2 Transport Stream (TS) packets are passed to physical layer where the transmission signal is generated with a DVB-T modulator. After the transmission over a wireless channel, the receiver receives distorted signal and possibly erroneous TS packets are generated by the DVB-T demodulator. The errors are tried to be corrected in the link layer by the MPE-FEC functionality 19

21 and the TS packets are decapsulated into IP datagrams. IP datagrams are handled in the 3D video streamer client and resulting NAL units are decoded with the 3D video decoder (2D video decoder, 2D video decoder+depth based image renderer or stereo video decoder) to generate right and left views. Finally, these views are put into an appropriate format to be displayed as 3D in the displayer. Stereo Video / 2D video+ depth data 3D Video Encoder NAL Units 3D Video Streamer Stream 1 (left view) RTP, UDP, IP Stream 2 (independent right view/ dependent right view/depth data) RTP, UDP, IP DVB -H IP Encapsulator TS Packets DVB- T Modulator Wireless Channel Right view Displayer Left view 3D Video Decoder NAL Units Stereo Video Streamer Client IP DVB- H IP Decapsulator TS Packets DVB -T Demodulator Figure 14 The system overview. The display unit of the clients is an autostereoscopic display which allows a 3D vision without any need for eye glasses or special equipment. In an autostereoscopic display, the right and the left views go into an interdigitation process in which an appropriate picture is generated from the pixels of left and right views. In the displayer of our system, as a pair of left and right frames is decoded, the interdigitation process takes places and the generated image is displayed on the autostereoscopic display. An important feature in our system is backward compatibility, i.e. existing mobile users without 3D features are able to receive stereo video broadcast and play the monoscopic video (either right view or left view). The backward compatibility is accomplished in the link layer where the IP datagrams corresponding to right and left views are encapsulated in the link layer and put into different ESs with different PIDs. In this way, the data for left and right view are transmitted in different bursts so that a non-stereo compliant user can play monoscopic video by only receiving corresponding ES. In this approach, the IP decapsulator is modified so that the receiver knows that it needs to filter two PIDs Experimental Results To validate the performance of the simulation setup, we performed several transmission experiments where (a) an MVC coded test sequence is transmitted over a simulated DVB-H channel, (b) simulcast coded test sequences are transmitted over a simulated DVB-H channel, (c) 2D + depth data coded independently are transmitted over a simulated DVB-H channel. We simulated three transmission modes: 1) QPSK constellation, convolutional code rate of 1/2 and MPE-FEC rate of 3/4 over Typical Urban (TU) channel with 6 taps, 2) 16-QAM constellation, convolutional code rate of 2/3 and MPE-FEC rate of 3/4 over TU channel with 6 taps and 3) 16- QAM constellation, convolutional code rate of 2/3 and MPE-FEC rate of 3/4 over residential 20

22 outdoor channel. The test sequence was encapsulated in a transport stream by putting each view or 2D +depth data in its own MPE stream. The erroneous channel conditions were simulated on the physical layer using the previously mentioned Simulink model of the transmission system. Different channel conditions were tested by varying channel SNR values and the simulation results were saved as transport stream level packet error traces [17]. The error traces were then mapped on to the test transport stream in order to introduce errors. We evaluated the impact of MPE-FEC by comparing two scenarios when extracting the H.264/MVC video (or simulcast video) from the transport stream: 1) Client receiving the FEC codes within the MPE-FEC frame, 2) Client receiving only the data part of the MPE-FEC frame where power consumption is reduced. Finally, the quality of received and decoded video was measured by computing Y-PSNR of the views using original, uncompressed video as the reference. This simulation process was repeated 50 times for each channel SNR condition in order to obtain figures corresponding to the average behaviour of the channel. In all the figures, PSNR values are all in db and calculated according to the following formulas where D l and D r represent the distortions in left and right views respectively. 2 PSNR 255 jo int 10log 10 ( D D ) / 2, l r PSNR left log 10, D l PSNR right log 10 Eq. 1 D r For simulations, the same sequences (Hands and Horse) that are prepared by HHI are used [ref]. For 2D+depth coding, the 2D video is assumed to be the left view and both the original depth map and the received depth map are used to render the right view. The PSNRs are compted according to Eq. 1. For joint stereo and multicast coding, the sequences are coded in such a way that, no loss PSNRs are almost the same. Thus bitrate of joint coding is slightly less than that of simulcast coding. The summary of 3D sequences and coding parameters are shown in Table 4-.Table 6 Producer KUK Sequences Hands Horse Length [frames] 1 1 Framerate [frames/second] Resolution [pixel] 480 x 272 Coding Type Simulcast Stereo 2D+Depth Data Format V L + V R V L + V R V L + V D Table 4 3D video data set HANDS 21

23 V L V R V D Coding Type Rate [kbps] PSNR [db] Rate [kbps] PSNR [db] Rate [kbps] PSNR [db] Simulcast N/A N/A Stereo N/A N/A 2D+Depth N/A Table 5 Hands sequence coding parameters HORSE V L V R V D Coding Type Rate [kbps] PSNR [db] Rate [kbps] PSNR [db] Rate [kbps] PSNR [db] Simulcast N/A N/A Stereo N/A N/A 2D+Depth N/A Table 6 Horse sequence coding parameters Simulation set 1 QPSK constellation, convolutional code rate of 1/2 and MPE-FEC rate of 3/4 over TU channel with 6 taps No loss FEC No FEC No loss L FEC L No FEC L No loss R FEC R No FEC R (a) (b) 22

24 Figure 15 Simulation results for the Hands sequence. The videos are jointly coded where the stereo GOP size is 16. (a) joint PSNR, (b) individual PSNRs No loss FEC No FEC No loss L FEC L No FEC L No loss R FEC R No FEC R (a) (b) Figure 16 Simulation results for the Hands sequence. The videos are simulcast coded. (a) joint PSNR, (b) individual PSNRs No loss FEC No FEC No loss L FEC L No FEC L No loss R FEC R No FEC R

25 (a) (b) Figure 17 Simulation results for the Hands sequence. 2D+depth coding is utilized. PSNR is computed using the right views rendered by the original sequence+original depth map and the received sequence+received depth map. No loss FEC No FEC No loss L FEC L No FEC L No loss R FEC R No FEC R (a) (b) Figure 18 Simulation results for the Horse sequence. The videos are jointly coded where the stereo GOP size is 16. (a) joint PSNR, (b) individual PSNRs. 24

26 (a) No loss FEC No FEC (b) No loss L FEC L No FEC L No loss R FEC R No FEC R Figure 19 Simulation results for the Horse sequence. The videos are simulcast coded. (a) joint PSNR, (b) individual PSNRs. No loss FEC No FEC 27 No loss L FEC L No FEC L No loss R FEC R No FEC R (a) (b) 25

27 Figure 20 Simulation results for the Horse sequence. 2D+depth coding is utilized. PSNR is computed using the right views rendered by the original sequence+original depth map and the received sequence+received depth map. Simulation set 2 16-QAM constellation, convolutional code rate of 2/3 and MPE-FEC rate of 3/4 over TU channel with 6 taps No loss FEC No FEC 26 No loss L FEC L No FEC L No loss R FEC R No FEC R (a) 22 (b) Figure 21 Simulation results for the Hands sequence. The videos are jointly coded where the stereo GOP size is 16. (a) joint PSNR, (b) individual PSNRs. 26

28 34 34 No loss FEC No FEC No loss L FEC L No FEC L No loss R FEC R No FEC R (a) 25 (b) Figure 22 Simulation results for the Hands sequence. The videos are simulcast coded. (a) joint PSNR, (b) individual PSNRs No loss FEC No FEC No loss L FEC L No FEC L No loss R FEC R No FEC R 25 (a) 22 (b) Figure 23 Simulation results for the Hands sequence. 2D+depth coding is utilized. PSNR is computed using the right views rendered by the original sequence+original depth map and the received sequence+received depth map. 27

29 No loss FEC No FEC 26 No loss L FEC L No FEC L No loss R FEC R No FEC R (a) 22 (b) Figure 24 Simulation results for the Horse sequence. The videos are jointly coded where the stereo GOP size is 16. (a) joint PSNR, (b) individual PSNRs (a) No loss FEC No FEC (b) No loss L FEC L No FEC L No loss R FEC R No FEC R

30 Figure 25 Simulation results for the Horse sequence. The videos are simulcast coded. (a) joint PSNR, (b) individual PSNRs No loss FEC No FEC No loss L FEC L No FEC L No loss R FEC R No FEC R 25 (a) 22 (b) Figure 26 Simulation results for the Horse sequence. 2D+depth coding is utilized. PSNR is computed using the right views rendered by the original sequence+original depth map and the received sequence+received depth map. Simulation set 3 16-QAM constellation, convolutional code rate of 2/3 and MPE-FEC rate of 3/4 over residential outdoor channel with 6 taps

31 34 34 No loss FEC No FEC 26 No loss L FEC L No FEC L No loss R FEC R No FEC R 24 (a) 22 (b) Figure 27 Simulation results for the Hands sequence. The videos are jointly coded where the stereo GOP size is 16. (a) joint PSNR, (b) individual PSNRs No loss FEC No FEC No loss L FEC L No FEC L No loss R FEC R No FEC R (a) 25 (b) Figure Simulation results for the Hands sequence. The videos are simulcast coded. (a) joint PSNR, (b) individual PSNRs.

32 (a) No loss FEC No FEC 27 (b) No loss L FEC L No FEC L No loss R FEC R No FEC R Figure Simulation results for the Hands sequence. 2D+depth coding is utilized. PSNR is computed using the right views rendered by the original sequence+original depth map and the received sequence+received depth map. No loss FEC No FEC No loss L FEC L No FEC L No loss R FEC R No FEC R (a) (b)

33 Figure Simulation results for the Horse sequence. The videos are jointly coded where the stereo GOP size is 16. (a) joint PSNR, (b) individual PSNRs No loss FEC No FEC.5.5 No loss L FEC L No FEC L No loss R FEC R No FEC R.5.5 (a) (b) Figure Simulation results for the Horse sequence. The videos are simulcast coded. (a) joint PSNR, (b) individual PSNRs No loss FEC No FEC No loss L FEC L No FEC L No loss R FEC R No FEC R.5

34 (a) (b) Figure Simulation results for the Horse sequence. 2D+depth coding is utilized. PSNR is computed using the right views rendered by the original sequence+original depth map and the received sequence+received depth map. In the figures, No loss corresponds to the scenario where there is no loss in the transmission, FEC corresponds to the scenario where MPE-FEC is decoded in the case of lossy transmission and No FEC corresponds to the scenario where the receiver discards the FEC bytes. The results show a clear improvement of quality when the received MPE-FEC data is used especially in low channel SNR cases. Another observation is that using MPE-FEC frame achieves almost no loss performance for high channel SNR values. In the results, it is shown that because of the dependencies between right and left views, joint coding is less robust to errors. However, using Unequal Error Protection (UEP), this robustness can be increased. In terms of complexity at the decoder side, for simulcast decoding and 2D+depth decoding conventional video decoders can be used. However for depth data additional rendering has to be done. For joint coding simplified MVC decoder has to be implemented at the decoder. In the figures, it is seen that all representations give approximately the same average PSNR. However when the bitrates are compared, simulcast coding is the highest and 2D+depth is the lowest. However for 2D+depth coding, we are not using the original right view for computing the PSNR of right view. Instead, we are using the right view which is rendered using the original left view and the depth information. In Figure - Figure 35, we also present the nature of the packet losses for different channel conditions. The results are given for the Horse sequence transmitted over TU 6 Channel with SNR = 18 db and SNR = 23dB. In both cases only frames from the left view are lost. For the case of SNR=18 db, the lost frames are shown in Table 7. For the case of SNR=23 db, only the th frame (I frame) from left view stream is lost. In both cases, the right view streams are received without any losses. But as the anchor frames of right views depend on the left view, the right views are also affected from the losses as well. P O C C O R E C P O C C O R

35 Frame PSNR (db) E C Table 7 Loss distribution of frames for the left view for channelsnr= 18dB. POC: Picture Order Count, the order displayed on the screen, CO: Coding or Transmission Order, in our case, the order obeys the hierarchical structure, REC: 1: received, 0: lost Left Right Aver Lost left frames Frames Figure PSNRs for reconstruction of individual frames for the loss conditions given in Table 7 Fr Left Right 74 34

36 Figure 34 The reconstructed frames (74-79) for the loss conditions given in Table 7 35