Until recently, varispeed acquisition was used for

Transcription

1 Optimal Temporal Sampling Aperture for HDTV Varispeed Acquisition By Emil Borissoff According to the perceptions of the human visual system, optimization of the temporal sampling aperture is required to best convey motion blur during HDTV image capturing. Filtering temporal aliasing artifacts becomes possible with the latest technologies in picture acquisition, and is applicable for advanced tape-based or tapeless recording. This paper presents an analysis of video capturing and recording in HDTV, for the case of variable frame rate (varispeed) acquisition. It proposes an optimization of the exposure duration, so that the duration becomes dependent on the frame rate of the master format. This approach restrains the strobe effect and utilizes motion blur for better rendering of dynamic scenes. The improved overall image quality also benefits the processes of editing, compositing, conversion, and compression. Finally, the research results are integrated into an embedded algorithm to suggest how to implement this strategy. Until recently, varispeed acquisition was used for film cameras only. This mechanism allows film to be transported and picture exposed at a wide range of frame rates. Replaying processed film at the standard rates of 24 frames/sec creates attractive effects of slow motion or accelerated scene object displacement. Motion blur caused by emulsion response and blurring of the background image when tracking a moving object have been noted as a desirable smoothing factor on the time-frame samples, if retained in an optimal range. Today, advances in HDTV varispeed acquisition allow two ways of accomplishing the same task in the video domain, both counting on the camera s optical sensor integration ability: Capturing at variable frame rate and recording at the same rate on hard disk drives or optical media. Playback from these devices is at the standard HDTV rate, and therefore the process fully competes with the film procedure. At present, this method is used in some nonbroadcast applications and requires an adaptive sync generator. Capturing and recording at the highest possible standard frame rate on a VTR. The playback of selected frames from the shot sequence is at the master HDTV rate, thus transforming scene motion proportionally to the ratio of the two frame rates. This method has already been successfully implemented and is well accepted in TV production, post-production, and broadcasting. 1 Furthermore, the improved spatial resolution of HDTV leads to an improvement of the temporal resolution, including uses in the varispeed mode. The use of progressively scanned pictures associated with some

2 HDTV formats, and the possibility for enhanced motion rendering, provide another stimulus for renewed exploration of the principles behind frame sampling. From this perspective, Fig. 1 illustrates the motion blur traces in 24 frame/sec acquisition. If the exposure shutter is open for the duration of the TV frame, maximum motion blur is recorded at 24 discrete moments per second. The human visual system (HVS) perceives it differently from the natural way of viewing, and only the image retention on the retina, together with brain interpolation, provide a realistic scene reproduction. 2 Film cameras have their shutter open for 1/48 of a second in normal shooting, which reduces motion blur, but also acts as a partial low-pass temporal filter that reduces possible strobe effects. Zagier 3 proposed that the HVS needs to receive an optimal amount of motion blur in order to simulate a natural scene reproduction. In the case of varispeed acquisition, the motion blur amount is recorded at one shutter speed and reproduced proportionally at another speed. This method ensures good rendering of scene dynamics in standard-definition television (SDTV), but appears to leave some room for improvement in HDTV, in which striving for better picture means matching the perception of human vision more adequately. Temporal Sampling Aperture Figure 1. Motion blur, fill factor = 1. Along the frame-rate axis, the Nyquist criteria for minimizing aliasing artifacts are more complex than the horizontal and vertical sampling axes, along which, for the most part, conventional sampling theory applies. 4 To reduce the visible artifacts resulting from the spectral overlapping that is inherent to unfiltered image sampling, a few conditions must be met. Nominal Condition 1: Sampling Duration The sampling duration should be less than half of the frame period. Thus the temporal sampling, Ts, would be Ts = ½ Tframe, measured in time units. However, this filtering limitation will ignore unsampled object motion or camera Figure 2. Motion components. zoom that occurs in the rest of the TV frame. The position of the object in the next frame will determine its interframe displacement and may appear as a jump. To constrain a scene element relocation with regard to frame frequency, a second condition applies. Nominal Condition 2: Object Displacement 2.1 The object linear displacement should be less than half of the screen distance during one frame period. 5 Then Ts should be: Ts <= ½ Tobject_pass_Hscreen Ts <= ½ Tobject_pass_Vscreen 2.2 The rotational displacement should make less than half a revolution during one frame period. Then Ts should be: Ts <= ½ Tobject_rotate_cycle. Limiting the object repositioning in the frame sequence according to Conditions 2.1 and 2.2 contributes to keeping the potential strobe effects under control as well. 6 Because there are no practical ways to estimate and control the object displacement for implementing Nominal Condition 2, it is the motion blur that will be analyzed as a constraining factor. Motion blur traces make dynamic elements appear to be expanded, or more continuous, through the sequential time-frame samples. Convoluted object position changes require an advanced approach to the analysis of the elementary shifts. Motion decomposition will reveal the components of a complex scene displacement (Fig. 2):

3 a) circular + vertical + horizontal motion, intra-frame b) circular + diagonal motion c) nonlinear elliptical motion d) retracting motion, total object displacement to be accounted for. Assuming directionally constant motion over the exposure period of a full frame, the length l (h, v) of its blur trace at each pixel k (h, v) is obtained by where a(t) is the aperture open time, H (h, v) and V (h, v) are the sequential displacement vectors in twodimensional space, and T is the frame period. Temporal sampling should not decimate object dynamics. Actually, its optimal amount will highlight the positional transformations. It is the trail of the motion blur that causes an object to be seen in its previous position, and if the strobe effect is restrained, the natural movements in the scene will be preserved. Sampling a continuous image requires distinguishing between a sampling function and an aperture function: where the nominal aperture is given by The temporal sampling aperture is a rect function, corresponding to integration over some period Ts. The temporal fill factor of a frame-sampled image becomes FF t = T exposure /T frame x 100% Assuming constant pixel size (Hp, Vp), the temporal aperture function is where T = 1 24 [sec] for mastering HDTV format of 24 frames/sec. The optimal temporal sampling aperture in a fixed frame rate acquisition is motion dependent. The Figure 3. Frequency response and the fill factor. Gaussian curve according to Young et al. 7 is presented in Fig. 3. For this figure, it is assumed that CCD camera elements charge instantaneously, given sufficient light exposure. Multiple objects and a combination of linear, nonlinear, and rotational object motions would entail the selection of a dominant object in the scene. An adaptive temporal Nyquist criterion in this case will ensure best motion rendering, and should be considered for future manufacturer designs. Reconstruction of continuous (i.e., nonstrobic) temporal images at the receiving end happens only because of the ability of the human visual system to interpolate in between TV frames. 8 This acts as a third dimension equivalent to the well known two-dimensional filtering function sin X/X. 9 Temporal Correction Coefficient The larger the temporal fill factor, the more motion blur will be captured. The smaller the temporal fill factor, the more strobe effects may occur. This paper proposes that the optimal temporal sampling aperture be defined as a function of varispeed timing parameters: Sampling Time = f (Number of Recorded frames/sec, Number of Playback frames/sec, Converted Motion Blur Aperture, Temporal Nonlinear Correction Coefficient) The Converted Motion Blur Aperture is the time duration of the estimated dynamic scene object displacement, captured under variable frame rate, and then linearly converted to the master frame rate. The Temporal Nonlinear Correction Coefficient is introduced to reflect the need for logarithmic correction due to the specificity of the human visual system.

4 Figure 4. Temporal nonlinear correction coefficient. According to studies conducted during the 1990s, 10,11 the eyes perceive the amount of motion blur proportionally to the object speed under a logarithmic law. The same is valid for the temporal sampling rate, or frame rate, at which moving objects leave traces at discrete intervals. It means that at higher frame rates the human visual system needs a smaller amount of motion blur to perceive continuous motion. However, this correlation is not linear, and the capture process in an HDTV camera needs to deal with this phenomenon. Similarly, at lower object speeds or lower frame rates, more motion details are seen than the linear transferred motion blur aperture would suggest. If the temporal sampling aperture is made a function of the variable frame rates, the images captured by HDTV cameras could better take into account the particularities of human vision. The curves of the linear and the corrected sampling aperture duration as a function of frames per second are shown in Fig. 4. Classic film cameras do not have such capabilities. The nonlinear perception of dynamic scenes was not considered in television either. What has been considered so far is the limitation in perceiving spatial elements and temporal details. The image integration time within the video frame affects: accumulative object motion in the picture plane during sampling; strobe effect perception based on variable frame rate; Nyquist criteria for maximum inter-frame motion; Figure 5. Operator control of sampling aperture for effects of slow or fast motion. nonlinear sampling and total amount of intra-frame movement. Figure 5 shows the reaction of the temporal nonlinear correction coefficient when the operator changes the camera exposure time for creative purposes. In this case, the master playback frames per second point remains under the cinematographer s control, and the recorded frames per second suggest a correction aimed at motion artifacts suppression. The nonlinear sampling here is a default mode for optimal shooting in varispeed mode. It will deliver a Gaussian perception at a master frame rate speed with clean slow motion or nonjerky fast motion. Deviations from the optimal temporal sampling are assumed to be effects introduced manually, and involve expressive motion blur or strobe. These effects can be used as tools of the cinematographer s creativity. Multiple Sampling within One Frame Usually the temporal sampling aperture period starts at the beginning of a TV frame and lasts during the opening of the electronic shutter, which is implemented by switching one of the CCD supply voltages. A short sampling duration will mean that only the beginning of the intra-frame motion is captured. 12 In this Figure 6. Multiple samples the fill factor remains unchanged.

5 case, object displacements in the later part of the frame will be recorded in the next frame, as new positions without path traces. One way to retain optimal object traces is to apply multiple samples within the frame. This will distribute captured motion segments throughout the entire frame period. The total exposure time will not be changed. Figure 7. Multiple versus single sampling aperture. Multiple temporal sample-and-hold cycles within one TV frame are possible with CCD-based HDTV cameras for which frame transfer CCD sensors are assumed (Fig. 6). The total sample-and-hold time in one TV frame remains unchanged, and as a sum represents the exposure time. The temporal fill factor preserves its value. Figure 7 compares the prolonged motion path of multiple sampling apertures with the short constant duration of a single sample. The multiple sampling method encompasses more of the intra-frame object path. Blur intensity is best analyzed by Bessel functions, but with an approximation of the first order, it is accepted that the blur distribution is linear during the frame period. 13 Algorithm OPTIMAL TEMPORAL SAMPLING APERTURE FOR HDTV VARISPEED ACQUISITION The temporal fill factor can be sufficiently presented by one byte, or 256 possible time duration values per frame. The source code sequence for the optimal temporal sampling aperture during varispeed acquisition is presented below. It is written in the Visual Studio programming language, with an embedded microcontroller Assembler module. Dim R-fps As Variant ;Recorded Frames Per Seconds Dim P-fps As Byte ;Played Frames Per Second, standard Dim D As Integer ;Ratio P-fps / R-fps Dim T-rec-frame As Integer ;Time period of the recorded frames Dim A As Variant ;Aperture function [time] Dim FF As Integer ;Fill Factor Dim TCC As Variant ;Temporal Correction Coefficient Dim Q As Variant ;Offset for TCC value in Assembler Dim M As Integer ;Multiple samplings in a frame, default M=1 Dim E As Variant ;Manual sampling correction for effects Dim N As Integer ;Temporal Nyquist criteria value Sub Function( ) D = P-fps / R-fps FF = A / T-rec-frame If E=1 And R-fps > 6 Then If FF < 1 And M=1 Then If A > And N = 1 Then TCC = ln D End If End If End If End Sub ;Assembler module for the Look Up Table. Load correction value in TCCREGISTER. MAIN BTFSC IDREGISTER, 1 GOTO LINEAR

6 BTFSC DREGISTER, 1 GOTO LOGUP GOTO LOGDOWN LINEAR INCF TCCREGISTER, 1 GOTO CORRECTION LOGUP ADDLW K_UP, 0 MOVWF TCCREGISTER GOTO CORRECTION LOGDOWN ADDLW K_DOWN, 0 MOVWF TCCREGISTER GOTO CORRECTION CORRECTION MOVF LUT, 0 ADDWF TCCREGISTER, 0 BTFSS IDREGISTER, 2 GOTO MAIN END Conclusion The ability to accurately convey motion is a useful TV image acquisition technique. It addresses the issue of temporal aliasing and stimulates optimal camera electronic shutter action. So far, television techniques have not yet sufficiently used the sampling aperture along the time axis for simulating the integrating characteristics of the human visual system. High-quality HDTV pictures are worthy of temporal improvement because their frame sequence contains high image acceleration, low motion predictability, and complex scene movement all factors that can be optimized. There seems to be conflicting data regarding processing of multidirectional and multispeed object displacement in image scenes. 14 Presently, there are no reliable methods for intra-frame motion analysis during acquisition. Human eyes repositioning (or tracking) on different parts of the display screen also contribute to the final scene perception. The proposed Gaussian method appears to be the best approach for sampling object dynamics. It opens the door for further research, in an attempt to facilitate the ability of the human observer to detect motion disturbance under a variety of conditions. Presented at the 37th Advanced Motion Imaging Conference, Seattle, WA, February 27-March 1, Copyright 2004 by. References 1. J. Merritt, An Introduction to Editing with Variable Frame Rates, Panasonic. 2. D. Tull and A. Kastaggelos, Iterative Restoration of Fast- Moving Objects in Dynamic Image Sequences, Optical Engineering, Vol. 35, May E. Zagier, A Human s Eye View: Motion Blur and Frameless Rendering, Association for Computer Machinery Crossroads, Vol. 3, Mar P. Wilson, Navigating the Rapids of Standard Conversion, J., 109:720, Sept B. P. Lathi, Linear Systems and Signals, Oxford University Press: Oxford, L. Rudin, Geometrical Constraints in Video Photogrammetry, Proceedings of the international Society of Optical Engineering, Vol. 41, Oct I. T. Young and L. J. Van Vliet, Recursive Implementation of the Guassian Filter, IEEE Transactions on Signal Processing, Vol. 44, Feb C. Poynton, A Technical Introduction to Digital Video, Wiley: New York, NY, A. I. Zayed, Advances in Shannon s Sampling Theory, CRC Press: Boca Raton, F. Huck, C. Fales, and Z. Rahman, Visual Communication: An Information Theory Approach, Kluwer Academic Publishers: Dordrecht, The Netherlands, Y. Z. Wang, Filtering, Sampling, and Aliasing in Peripheral Vision, Ph.D. Thesis, Indiana University, K. Castleman, Digital Image Processing, Prentice Hall: NJ, I. Rekleitis, Visual Motion Estimation Based on Motion Blur Interpretation, M.S. Thesis, McGill University, Montreal, S. Mann, Intelligent Image Processing, Wiley: New York, NY, THE AUTHOR Emil Borissoff is chief engineer of Toybox Toronto, a division of Command Post & Transfer Corp., which he joined 13 years ago. He has been working in the area of digital video systems and motion imaging. His research interests include 3-D sampling and conversion, compression optimization, dynamic scene artifacts, and HDTV standard limitations. Borissoff received an M.S. and a Ph.D. degree in electronic engineering and image processing from the Polytechnic University of Sofia, Bulgaria, in 1977 and 1987, respectively. He started his career as a broadcast engineer with a TV station in this city, engaged from the beginning with R&D. He moved to Canada in 1991 as a senior post-production engineer. Borissoff has four international patents in the field of video algorithms and has been a member of since 1994.