Real-time Computational Fluid Dynamics for Flight Simulation

Real-time Computational Fluid Dynamics for Flight Simulation James Kenny, Kenji Takeda & Glyn Thomas School of Engineering Sciences University of Southampton, Southampton SO17 1BJ, UK james.kenny, ktakeda, tgt@southampton.ac.uk ABSTRACT A service oriented architecture is described that enables computational fluid dynamics (CFD) simulations to run alongside a human-in-the-loop flight simulator; thereby informing the behavior of a simulated aircraft whilst it is being piloted. The scenario of a helicopter landing on a moving ship at sea, is used as an example application. A generic service-oriented architecture is presented that allows coupling of a real-time flight simulator, flight dynamics model and CFD simulation running on a high performance computer. The case study used is performing unsteady CFD calculations used to model the aerodynamic development of, and interaction between, the ship and helicopter wakes; The CFD code resides on a cluster computer and is exposed to a PC-based flight simulator as a service, enabling two-way data exchange between the CFD and flight model whilst the simulation is running. Realtime analysis of the CFD results and control inputs allows prediction of the forces acting on the helicopter rotor, this is fed into a full six degree of freedom flight model. Performance results for the full end-to-end architecture are presented to demonstrate the capability, and limitations, of this approach. The paper concludes with a short discussion regarding the potential for this architecture to provide a generic representation of aircraft-environment interactions, and their influence on performance and handling. Implementing a more accurate representation of these phenomena in flight simulators could improve the ability to prepare pilots for challenging tasks such as: landing on ships, flying in urban environments, dealing with brown-out conditions, and encountering the wakes of other aircraft. ABOUT THE AUTHORS James Kenny is a research student in the School of Engineering Sciences at the University of Southampton; he holds a degree in Aerospace Engineering, and has eight years of experience working in the defence and aerospace sectors. James area of research includes helicopter simulation, computational fluid dynamics, and high performance computing. Dr Kenji Takeda is a senior lecturer in the School of Engineering Sciences at the University of Southampton, and co-director of the Microsoft Institute for High Performance Computing. Dr Takeda s research covers unsteady aerodynamics and aeroacoustics, race car aerodynamics, flight simulation, and advanced engineering computation. His work has been applied in projects with companies such as Airbus, BAE Systems, Microsoft, and Formula One teams. Dr T Glyn Thomas is a lecturer in the School of Engineering Sciences at the University of Southampton. His research interests include computational fluid mechanics and turbulence, flight simulation, wind engineering, and parallel computing. 2008 Paper No. nnnn Page 1 of 8

Real-time Computational Fluid Dynamics for Flight Simulation James Kenny, Kenji Takeda & Glyn Thomas School of Engineering Sciences University of Southampton, Southampton SO17 1BJ, UK james.kenny, ktakeda, tgt@southampton.ac.uk INTRODUCTION The modeling of aerodynamic effects in real-time flight simulators is a continuously evolving field. It is becoming increasingly relevant for flight simulators to more accurately describe the local unsteady environmental flow around the vehicle, in a timeaccurate way for applications such as wake vortex and helicopter-ship airwake interaction scenarios. This is important from both development and training perspectives. This is an extremely challenging problem that is being tackled using a variety of techniques. Current approaches typically involve using measured wind tunnel or flight trial data, or pre-computed computational fluid dynamics (CFD) results, as inputs to a real-time flight dynamics model. This can provide representations of the local environment that significantly enhance the flight dynamics behavior. For example, Spence et al (2007) demonstrate how highresolution large eddy simulation computations of an aircraft wake vortex can be used as inputs to a real-time flight simulator. Zan et al (2005) reviews progress in the helicopter ship airwake interaction problem, and how several groups are attempting to model this. Most recently Keller et al (2007) have demonstrated how unsteady CFD of a ship airwake can be integrated into a real-time helicopter simulation. These approaches are, however one-way in nature, calculating the effect of the local environment on the vehicle, but not including interaction of the vehicle aerodynamics on the environment. This is particularly critical in, for instance, the helicopter ship airwake interaction problem. The inclusion of other dynamic effects, such as ship motion on the airwake, is currently beyond state-of-the-art. The development of offline, two-way models using computational fluid dynamics coupled to flight dynamics solvers has been successfully demonstrated for both fixed and rotary wing problems. Allan et al (2005) coupled a CFD solver to a three degree of freedom model to investigate the effect of elevator control on longitudinal stability. CFD ship airwake computations by Lee et al (2003) and Sezer-Uzol et al (2005) use an automatic control system to simulate the pilot inputs required to land a helicopter on the ship. The coupling of a flight simulator to HPC CFD code is an example of real-time computational steering (Modi et al, 2005). With the improvements in price/performance of high performance computers, it is becoming possible to consider real-time CFD calculation being used in human-in-the-loop flight simulators. Computation of a free-wake model in free space in near-real time has been demonstrated by Horn et al (2006), and simplified airwake and multi-vehicle interaction (McKilip et al, 2006). In this paper we describe a generic proof-ofconcept framework for the use of unsteady Navier- Stokes computational fluid dynamics calculations in a human-in-the-loop flight simulator to compute the twoway interaction between the vehicle and the local environment. Notably, this architecture is designed to exploit HPC from the outset. There are, of course, a number of significant issues related to this goal, which is discussed here. By using a service-oriented software approach, we describe how it can also be extended to other applications. The following section describes the overall methodology for our approach, including service-based software architectures, high performance computing, and how this can be coupled to a flight simulator in real-time. A case study of a helicopter interacting with a ship airwake is then described, including performance results for the round-trip messaging used in the framework. This is followed by a discussion of how this framework can be extended and used for other high-fidelity physics modeling tasks. Some concluding remarks are then included at the end of the paper. METHODOLOGY In order to achieve real-time CFD inputs for a flight simulator, a distributed computing architecture is 2008 Paper No. nnnn Page 2 of 8

required. Here we aim to exploit high performance computing, notably commodity personal computerbased cluster systems (Takeda et al, 2001). The software framework uses the concept of a serviceoriented architecture (SOA). This type of architecture relies on a number of services that can interact. Services differ from components and objects, in that they comprise contract, interface and implementation. In this way the precise implementation details are not needed in order to interact with a service. These are described in more detail, specifically as applied to flight simulation, in Takeda & Kenny (2008). By using an SOA, it is possible to build up functionality using a set of services. These can be coordinated over an Enterprise Service Bus, which is a logical construct over which messages can be managed. While such systems are now becoming more commonplace for large-scale, multi-player distributed simulation scenarios, here we use the idea to create a coupled system between the flight simulator and HPC resource. The advantage of using an SOA is that the framework is extensible, which is a feature that will be discussed later in the paper. The key feature of our generic framework is exposing the HPC resource as a service. This allows a running compute job, in this case a CFD simulation, to interact with processes over the network in a seamless way. By using a bus architecture it is also possible to extend this to other services. SOAs are often based on open web standards, such as XML, SOAP and WSDL. This is appropriate for internet-based applications where interoperability is paramount, and latency is not critical. For higher performance applications, other protocols can be used. One example of this is the Microsoft Windows Communication Foundation, which is an integrated framework for distributed computing. It can use webbased HTTP protocols, but also supports higher performance TCP/IP communications. The next section describes how WCF is used to implement service-based communications between the Matlab flight model server and HPC system. In order to perform CFD at update rates required to support human-in-the-loop simulation, multi-processor computing is usually required. A flow solver that can be run at appropriate speed and necessary level of physics fidelity is required. This code should be scalable, so that as physics fidelity is increased, so more computing resource can be used to maintain the rate of results delivery. Using a service-based interface means that the details of the CFD implementation are hidden from the rest of the system, so as long as the service contract is maintained, then the CFD solver can be changed as necessary. One of the main constraints of using real-time CFD for flight simulation is the update rate required to drive the flight dynamics model. Typically a visual system update rate of 50-60Hz is mandatory, so a flight model and control update rate of at least that is required. The update rate required by the CFD inputs must be on a time scale related to the physics of the interaction. i.e. how the local aerodynamics affect the dynamics of the vehicle. This rate may be significantly lower than the visual rate, as it is dependent on the flow physics and vehicle dynamics response. Recent work by Hodge and Forrest (2008) showed that an update rate of around 20Hz was sufficient to describe helicopter ship airwake interactions when using a flight dynamics model flying through a pre-computed CFD solution. In the architecture described here, the update rate can scale with the available HPC resource. Our aim is to demonstrate the feasibility of using real-time CFD for flight simulation, given sufficient HPC resource is made available. In the next section we describe a specific implementation of the proposed SOA framework. The aim is to show how the system is able to sustain the required flight model update rates, and that this performance is decoupled from the CFD solver in a way that allows scalable HPC resources to be integrated at the appropriate level. CASE STUDY In this section we describe an implementation of the SOA-based architecture for real-time CFD. The scenario of interest is that of a helicopter landing on a ship. Two-way coupling between helicopter aerodynamics and the ship airwake is required to provide a more representative dynamic simulation than is currently possible. Performance results that demonstrate achievable update rates using this approach are presented. The software architecture is based around the Microsoft ESP flight simulation engine (Zyskowski, 2008), with the flight model equations being solved using Matlab. This is coupled to a CFD solver running on an HPC cluster running Windows HPC Server (WHPCS) 2008. This particular technology is necessary to provide the service-based interface between the flight model and CFD simulation. 2008 Paper No. nnnn Page 3 of 8

Figure 1. Real-time CFD service-based architecture Figure 2. ESP visualisation of helicopter and ship The logical architecture for this system is shown in Figure 1. ESP is used as the front-end of the system, capturing pilot inputs through the flight controls and providing audio-visual feedback through the image generation and audio system. This is interfaced to Matlab using the SimConnect Application Programming Interface that provides two-way communications using a variety of transport mechanisms (Aces Studio, 2008). In this case, C# managed code is used to transfer input information from ESP to Matlab, and also aircraft position and orientation back to ESP for visualization by the image generator system (Figure 2). The flight model is computed in Matlab, with CFD being used to provide force inputs to the rotor dynamics. The CFD code runs on the HPC cluster and communicates with Matlab using a service, implemented using Windows Communication Framework (WCF), known here as the CFD service. This service allows input and output data to be sent to and from the HPC cluster in a clean fashion, using a service contract. An advantage of this approach is that the details of the CFD implementation are hidden from the Matlab flight model. In this way the CFD code can be changed without disrupting other parts of the architecture directly. The sample CFD service XML registration code is shown in Figure 3. 2008 Paper No. nnnn Page 4 of 8

CFDSvc.config <configuration> <Microsoft.Hpc.Session.ServiceRegistration> <service assembly= c:\services\cfdsvclib.dll contract= CFDSvcLib.ICFDSvc type= CFDSvcLib.CFDSvc /> </Microsoft.Hpc.Session.ServiceRegistration> </configuration> Figure 3. Sample XML service registration. CFDSvc.cs using System.ServiceModel; namespace CFDSvcLib [ServiceBehaviour(IncludeExceptionDetailInFaults=true)] public class CFDSvc : ICFDSvc #region ICFDSvc Members public double Update(double rhubx, double rhuby, ) return dpx, dpy, ; #endregion ICFDSvc.cs using System.ServiceModel; namespace CFDSvcLib [ServiceContract] public interface ICFDSvc [OperationContract] Double Update(double rhubx, double rhuby, ); Figure 4. Sample C# CFD service code. The CFD computation is performed using a research code that solves the unsteady incompressible Navier- Stokes equations on a staggered, multiblock grid. A second-order finite central finite difference scheme is used in space, with a second-order explicit Adams- Bashforth scheme in time. Continuity is imposed using pressure-correction methods, making use of a parallel multigrid Poisson solver. It can be used for both direct numerical simulation (DNS) and large-eddy simulation (LES), depending on grid density. This code has been extensively validated for flows with solid boundaries (Yao et al, 2001, Shi et al, 2001) and for vortex ring calculations (Archer et al, 2008). It was originally developed to take advantage of parallel, distributed architecture computers, and shows excellent speedup on a variety of different parallel computers on over 1000 processors. A typical CFD visualization from this code is shown in Figure 7 (Archer et al, 2008). In this case a rotor model is embedded within the CFD solver that is moved inside the mesh via the pilot control inputs and Matlab. The resultant rotor forces are then computed in the CFD solver and feedback to the flight model using the CFD service. CFDSvcClient.cs using System.ServiceModel; using Microsoft.Hpc.Scheduler.Properties; using Microsoft.Hpc.Scheduler.Session; namespace CFDClient class Program static void Main(string[] args) // Specify info needed to create session SessionStartInfo info = new SessionStartInfo( headnode, servicename); // Create session using (Session session = Session.CreateSession(info)) // Bind session to client proxy CFDSvcClient client = new CFDSvcClient( new NetTcpBinding( SecurityMode.Transport, false), session.endpointreference); client.beginupdate(rhubx, rhuby,, UpdateCallback, new RequestState(client)); class RequestState public double GetResult(IAsyncResult result) return client.endupdate(result); // Define callback static double UpdateCallback(IAsyncResult result) // Send CFD results to flight model acresponse = flightmodel(state.getresult(result)); Figure 5. Sample C# CFD client code. Figure 6. SOA architecture to HPC cluster 2008 Paper No. nnnn Page 5 of 8

Figure 7. Typical visualization showing CFD results for vortex ring calculation The main aim of this paper is to demonstrate how this end-to-end SOA can provide an extensible way of coupling real-time flight simulation with HPC-based physics calculations. Here we present performance results to show the relevant latencies and update rates achievable using this proof-of-concept architecture. The results were obtained using basic, off-the-shelf computer hardware. Six PCs with a guide specification of: 2.4GHz dual core processor, 2GB+ of RAM, Gigabit Ethernet network interface cards; communicating over a pair of Gigabit network switches. The time taken to complete the roundtrip task of interest was measured; this was done for several different tasks, and averaged over 1,000,000 calls. Table 1 presents some useful findings about the three key communication routes for the proposed architecture. For each route some time is taken preparing the communication: this ranges from user authentication when establishing access to the HPC CFD service, to loading the Matlab flight model into the Matlab Component Runtime. Additionally it can be seen that the first few calls to a service, or an API method, can occupy significantly more time that subsequent calls. The average time to complete a call, once communications have been established, is clearly seen to be of the order of milliseconds. This is important because it shows that the communications routes will not be a performance barrier for the proposed application. Therefore by increasing the computational resources available to each part of the application, it should be possible to obtain the update rates required for simulation. Based on the timing and latency results discussed, Table 2 shows some conservative estimates for the update rates which the various communication routes can support. Taking the bottom entry in the table as an example, it tells us: if we code our flight model in C#, collect CFD results from an HPC cluster using the SOA interface outlined in this paper, and use Microsoft ESP to carry out the human-in-the-loop simulation we could achieve update rates above 100Hz. The update rate achieved will be dependent upon the computational resources available, if the CFD service running on the HPC can only run at say 30Hz, then the whole system update rate would be limited to approximately 25Hz. This illustrates the challenge involved in providing coupled CFD solutions to inform flight simulation in real-time, flight dynamics data might normally be expected to have update rates in excess of 60Hz. For now this demands advanced filtering, extrapolation, and smoothing functions; but over the next few years as computer speeds increase that demand will reduce, eventually reaching the position where every simulated frame is driven directly by the CFD service. One final result to note is the time costs associated with using Matlab. In this work managed C# code interacts with Matlab functions that have been compiled as.net assemblies. It seems that the overheads involved are prohibitive; however further testing is required to prove whether this is indeed the case, or if perhaps it is the computational efficiency of the script itself that needs improvement. Duration Table 1. Time cost and latency results C# - HPC Service (Gigabit Ethernet) C# - ESP (Gigabit Ethernet) C# - Matlab (local machine) Communication ~ 20s ~ 2s ~ 25s startup Initial calls ~ 10s ~ 4s (per call) (1 st call only) (1 st -5 th / calls) Subsequent calls ~1.1ms < 5ms 20ms Note: s = seconds, ms = milliseconds Table 2. Update rates using tested configuration Roundtrip ESP C# - Matlab C# - ESP ESP C# - Matlab C# - HPC C# - Matlab C# - ESP ESP C# - HPC C# - ESP Update rate (s) 40Hz 20Hz > 100Hz 2008 Paper No. nnnn Page 6 of 8

DISCUSSION The case study presented here demonstrates how a service-oriented architecture can be used to provide a two-way coupling mechanism between a real-time flight simulator and a CFD solver running on a high performance computer. This proof-of-concept provides a generalized framework that opens up a variety of applications. Traditionally vehicle simulation has been limited by image generation and network bandwidth restrictions. As high performance computing has reduced in cost and network performance has improved, the possibility of using HPC to provide highlevel physics capability to real-time simulation is becoming realizable. While this is extremely challenging, the availability of a generic architecture that can be implemented as technology continues to improve to meet the requirements of simulation end-users, will allow developers to focus on services that can deliver the required fidelity. Use of offline, or pre-computed results, is the current state-of-the-art. While the proofof-concept developed here currently has performance limitations, the approach has several advantages in the medium- to long-term. Performance limitations of the SOA architecture that uses HPC to perform real-time computations are based around CPU, memory and network capabilities. The consumer and high performance computing markets are driving these components to keep improving at a rapid pace. Disk-based solutions, however, suffer from fundamental limitations in improvements in performance that will not be as scalable. Performance limits are different, and arguably more scalable in the long-term for real-time computational approaches. The architecture proposed here can also be easily extended to include disruptive technologies, such as generalpurpose graphics processing units (GPGPUs) that offer an order of magnitude improvement in computational speed over general-purpose processors. demonstrate that even with this layer, real-time update rates are possible. This is particularly relevant for research and development projects in which a rapid prototyping environment such as Matlab is desirable. While this paper describes an HPC proof-of-concept demonstrator for CFD, we believe that its extension to other areas, such as engine modeling, avionics, stealth, and artificial intelligence, is easily possible. CONCLUSIONS In this paper we have demonstrated a service-oriented architecture for the two-way coupling of a real-time flight simulator with an HPC-driven CFD solver. Results for the end-to-end message roundtrip time are given that demonstrate the ability of this architecture to be used for practical research and training applications. The generic SOA framework described here can be extended to include any high-fidelity simulation components that are beyond the scope of a single machine. This opens up the possibility of extending vehicle simulation fidelity that is only limited by the achievable update rate. The additional value of this framework is that it is flexible and extensible, with developers able to add functionality in a systematic way using a service-based approach. ACKNOWLEDGEMENTS The authors would like to thank Microsoft for their ongoing support of this work. The use of a service-based approach also allows other features to be easily incorporated. For example, the new High-Level Architecture (HLA) update, HLA Evolved, includes extensions to allow HLA content and processes to be integrated into SOAs (Möller et al, 2007). In order to improve update rates, using an in-process flight model will remove the Matlab layer and hence provide significant performance increases. Here we 2008 Paper No. nnnn Page 7 of 8

REFERENCES Aces Studio, Microsoft ESP Software Development Kit, Published by Microsoft Corporation, 2008. Allan, M.R., Badcock, K.J. and Richards, B.E. (2005) CFD based simulation of Longitudinal Flight Mechanics with Control. proceedings of Aerospace Sciences Meeting. AIAA-2005-0045. Archer, P.J., Thomas, T.G. and Coleman, G.N. (2008) Direct numerical simulation of vortex ring evolution from the laminar to the early transition regime. Journal of Fluid Mechanics, 598, 201-226. (doi:10.1017/s0022112007009883) Hodge, S. and Forrest, J, (2008), Fidelity Standards for Maritime Rotorcraft Simulation. In Proc. Royal Aeronautical Society June 2008 Maritime Operations of Rotorcraft conference, London, England. Horn, J. F., Bridges, D. O., Wachspress, D.A., Rani, S.L., (2006), Implementation of a free-vortex wake model in real-time simulation of rotorcraft, Journal of Aerospace Computing, Information and Communication, vol. 3., pp. 93 114. Spence, G.T. Le Moigne, A., Allerton, D.J., and Qin, N., (2007), Wake vortex model for real-time flight simulation based on Large Eddy Simulation,, AIAA Journal of Aircraft, Vol. 44, No. 2, pp. 467-475. Keller, J.D., Whitehouse, G.R, Boschitsch, A.H., Nadal, J., Jeffords, J., and Quire, M. (2007), Computational fluid dynamics for flight simulator ship airwake modeling, The Interservice/Industry Training, Simulation & Education Conference (I/ITSEC). Lee, D., Horn, J. F., Sezer-Uzol, N. and Long, L. N., "Simulation of Pilot Control Activity during Helicopter Shipboard Operation," AIAA Paper 2003-5306, AIAA Atmospheric Flight Mechanics Conference and Exhibit, Austin, Texas, August 11-14, 2003. McKillip, R. M. Wachspress, D. A. Keller, J. D. Whitehouse, G. R. Quackenbush, T. R. Boschitsch, A. H. (2004), Distributed Multi-Vehicle Simulation Including High-Order Airwake Representation, Annual Forum Proceedings- American Helicopter Society, Vol 60 (1), pp 286-314. Modi, A., Sezer-Uzol, N., Long, L.N., and Plassmann, P.E., Scalable Computational Steering System for Visualization of Large-Scale Fluid Dynamics Simulations, Journal of Aircraft, Vol. 42, No. 4, July- August, 2005. Möller, B., Dahlin, C.,and Karlsson, M, An Overview of the HLA Evolved Modular FOMs, Proceedings of 2007 Spring Simulation Interoperability Workshop, 07S-SIW-108, Simulation Interoperability Standards Organization, March 2007. Sezer-Uzol, N., Sharma, A., and Long, L. N. (2005) Computational Fluid Dynamics Simulations of Ship Airwakes, Vol. 219, Part G, Journal of Aerospace Engineering. Shi, J., Thomas, T.G. and Williams, J.J.R. (2001) Coarse resolution large-eddy simulation of turbulent channel flows. International Journal of Numerical Methods for Heat and Fluid Flow, 11, (1), 20-35. (doi:10.1108/09615530110364060). Shi, J., Thomas, T.G. and Williams, J.J.R. (2001) Coarse resolution large-eddy simulation of turbulent channel flows. International Journal of Numerical Methods for Heat and Fluid Flow, 11, (1), 20-35. (doi:10.1108/09615530110364060) Takeda, K., Allsopp, N. K., Hardwick, J. C., Macey, P. C., Nicole, D. A., Cox, S. J., and Lancaster, D. J. 2001. An Assessment of MPI Environments for Windows NT. J. Supercomput. 19, 3 (Jul. 2001), 315-323. DOI= http://dx.doi.org/10.1023/a:1011138816820 Takeda, K., and Kenny, J. (2008), Service-oriented architectures for simulation: opportunities and challenges, In Proc. Royal Aeronautical Society Spring 2008 flight simulation conference, London, England. Yao, Y.F., Thomas, T.G., Sandham, N.D. and Williams, J.J.R. (2001) Direct numerical simulation of turbulent flow over a rectangular trailing edge. Theoretical and Computational Fluid Dynamics, 14, (5), 337-358. (doi: 10.1007/s001620050144) Zan, S. (2005), On Aerodynamic Modelling and Simulation of the Dynamic Interface, Proceedings of the Institution of Mechanical Engineers, Part G: Journal of Aerospace Engineering, Volume 219, Number 5 / 2005, pp. 393-410. Zyskowski, (2008), Building 6-Degree of Freedom Physics Models with Microsoft ESP. In Proc. Royal Aeronautical Society Spring 2008 flight simulation conference, London, England. 2008 Paper No. nnnn Page 8 of 8