The Development Of Selection Criteria For Game Engines In The Development Of Simulation Training Systems

Transcription

1 The Development Of Selection Criteria For Game Engines In The Development Of Simulation Training Systems Gary Eves, Practice Lead, Simulation and Training Systems; Pete Meehan, Senior Systems Engineer QinetiQ Consulting Pty Ltd Abstract. The use of Game Engine technology for the construction of training simulations for defense applications is becoming common practice and will soon be employed in commercial applications. The choice of engine to be used has been largely driven by Defence Department strategies and evolution of games that looked like a visualisation of what soldiers do in real training. In the commercial sector few engines have been used and for new developers the choice is not only a technically complicated one, but one compounded by the ever growing number of engines to choose from. This paper presents the results of a project to create a set of selection criteria that would enable commercial users to make a balanced decision over cost, realism, customisation and licensing among others. The criteria developed and applied will illustrate three key areas of consideration;1) technical, the ability to construct, review and control the world and scenario building process. 2) the ability to create levels of user interaction that evolve from simple dependant rule abeyance, through to interdependent behavior and consequence awareness, 3) the ease and cost of distribution along with intellectual property management. The result of this study was a process applied to an industrial project where the criteria was used to make a commercial decision over how an engine was chosen and used to develop a training package. 1. INTRODUCTION The use of game technology for simulation development is becoming common place and well documented. To date most of the reviews of this technology application have focused on the success and proficiency of user interaction [1], and similar studies have in part reported on the use of pre-existing game developments mildly adapted (content changes only) for serious use [2], and the are many more being presented all the time, too many to cite here. There some reviews of the underlying technology presented with a view to suitability in the simulation market[3] and some reviews of actual methodologies for the assessment of finished product applications, some of which have been developed using game technology[4],[5]. This paper addresses the problem facing a potential developer who wishes to produce a novel application, not just one adapted with additional content and wishes to consider the use of the raw development tools as well the packaged game development environments such as Torque( This paper will report on a process developed specifically to assist in the first phase of an industrial serious game engine based development project. The project required a methodology to support the decision making process of engine selection and the details of that process are described here. It should be noted that the process was desk based activity, relying on what information could be obtained form freely available, public and vendor sites and in producing the final recommendations did not address the use of plug-in technology and tool extensions. Previous works [5] have produced helpful, but simple assessment criteria, the which we could call Phase 1 selection, this study aimed to improve on previous studies producing Phase 1 Plus, a new level Phase 2 where weighted and scored assessment could be made before a Phase 3 of actual bench testing of code is possible. The process does not yet include code and application testing, these tasks are suggested for a further adaptation of the process and will be discussed at the end of this report. This report covers the following: a. criteria and conduct of the first pass assessment; b. results of the first pass assessment; c. criteria and conduct of the detailed assessment; d. results of the detailed assessment; and e. conclusion and recommendations 1.1 Initial Selection Criteria If a potential developer is to search the internet and commercial developer sites, it is easy to come up with a list which could include over 250 tools and products that claim to enable game style development activity. In order to cull this list to a more manageable sub set to which more detailed assessment criteria could be applied, a first pass selection test should be applied. Similar first pass selection criteria have been presented before[5], with tick box style rating. In this study the following broad criteria were selected for this first pass cull: a. Current Technology. Was the game engine built on recent technology? b. Cost to Procure. The game engine is relatively inexpensive to procure. c. On Going Costs. The ongoing costs of using the games engine, or to distribute products developed from it, are low. SimTecT 2008 Refereed

2 d. Code versus Runtime. The game engine is supplied with full source code versus being supplied as a run time library only. e. Documentation. The game engine is supplied with detailed documentation describing its design and use. f. Support Group. The game engine has an established online support group of developers who are using the engine. g. Sound Support. The game engine supports the playback of sounds and music. h. Network Play Support. The game engine can support multiple players in a networked configuration. i. Animated Model Support. The game engine supports the rendering and animation of animated character and vehicle models. j. Built in Physics. The game engine has built in physics to detect collisions between in game objects. k. Editor Tools Provided. The game is shipped with editing tools for the creation of content and design of levels. Each of the above criteria can be further classified as described in Table 1 to apply a weighting between desirable features or just nice to have and not critical to the applications success. Importance Essential Desirable Table 1: Criteria Definition This must be satisfied. Its absence would inhibit the use of the game engine to achieve the required This is highly desirable. Its absence would diminish the extent and ease of using the game engine to achieve the required This is desirable. Its absence would slightly diminish the extent and ease of using the game engine to achieve the required By the application of the criteria described above it was found that the sub set of development tools could be reduced to a more manageable total of not more than 1 of the initial internet survey. In this particular project it was possible to reduce the number down to 10 development tools in total, from Detailed Assessment Criteria The following assessment criteria were defined to assist in determining which, of the short listed game engines, were best suited to the task of developing an industrial simulation training game. Each of the assessment criteria have been defined in the following format: AC-XXX The text of the criteria. Importance Where: AC defines the paragraph as an assessment XXX assigns a unique number to the assessment for ease of reference Importance defines the criticality of the as described by the classification in Table 1 The categories of selection criteria are group as follows, within each a series of detailed classifications. The top level assessment criteria are suggested to be: 1) Cost, 2) Rendering 3) Sound Effects 4) Avatars 5) 6) Physics 7) User Interface 8) Networking 9) Editing Tools 10) Level Design 11) Content 12) Recording 13) Documentation and 14) Support An example of how each category was expanded is shown by the example for, resulting in 70 criteria in total,: AC-024 Essential The game engine shall support the direct scripted actions of NPCs. AC-025 The game engine shall support the scripted actions of NPCs based on AI commands, e.g. walk to here, run away from here. AC-026

3 The game engine shall support the definition of new AI behaviour for NPCs. 2. ASSESSMENT METHODOLOGY 2.1 Assessment Methodology The methodology for assessing each short listed game engine against the criteria was to perform a desk top review each against the assessment criteria, which evolved up to 70 specific items derived from the 14 top level assessment criteria listed above in Section 1.2.and score the engines for each based on the scoring scheme shown in Table 3. The example for Cost. Table 3: Assessment Scoring Evaluation Scale: General Guidance Cost Related Criteria Guidance Completely fails > $ to meet per license Barely satisfies Mostly satisfies Satisfies Exceeds Substantially Exceeds $5000< $10000 $1000< $5000 $500< $1000 Free The scores for each engine are then tallied based on the criticality of each. For each engine the combined score for each criticality level were then combined again, using the weighting shown in Table 4 to produce and overall score for each engine. Table 4: Assessment Weighting Criticality Weighting Essential 3 2 Desirable 1 A spreadsheet was designed to capture all the assessment results, group them by major category, and reduce them to a percentage score based on the actual raw score and maximum possible score. Reduction of scores to a percentage enables easier comparison between results for any engine or criteria category. 2.2 Representation Of Assessment Criteria The matrix of assessment criteria can quickly become unwieldy to interpret, with as in the pilot project s case over 70 criteria were evolved and applied to a large number of engines, showing; Criterion, Criticality, Weighting, Assessment, Weighted Score, Possible Maximum Score, % Score and a Comments field., as illustrated below: Criticality Weight Assessment Weighted score Max Score Essential % score It is therefore advisable to use a simple pictorial representation for each engine and then for each assessment criteria by engine, examples are shown in Figures 1 and 2: Documentation Recording Content Level Design Support Editing Tools Torque Results 2 1 Cost Networking Rendering User Interface Sound Effects Physics Avatars Figure 1: Detailed Assessment Criteria. G3D Delta 3D Apocalyx Crystal Space Avatars Results Torque 2 1 OGRE DX Studio Unreal Figure 2: Criterion Assessment. Virtools Pow er Render

4 3. OBSERVATIONS G3D Results 3.1 Survey Method Validity The report illustrates an extensive set of assessment criteria, scoring and weighting system to review selected commercial and open source development engines subject to the first and second pass evaluations described. The process has been and is a subjective assessment and scoring process and it is therefore difficult to replicate and provide substantive absolute support for each result. This was further complicated by the varying levels of information available for each of the engines being assessed. Quite often, particular capabilities had to be inferred from discussions on support forums rather than from official technical specifications. Support Documentation Recording Content Level Design Editing Tools Cost Networking Rendering Sound Effects Avatars Physics User Interface However, in support of the methodology, in testing several people were chosen to conduct the review and cross checks were made between reviewers and a final internal review of the scores and between tool comparison was made by a fourth person. As a result of this process, it is felt that without actually testing each tool with data for each variable in the selection process, it has been a fair and balanced survey. A final check using game developers was made of the final tool ratings to ensure concurrency with market developments. 3.2 Testing Considerations At the onset of the project it was decided to explore the use of Commercial Of The Shelf and Open Source tools. Therefore, additional comment should be made regarding the Open Source tools reviewed. Open Source tools by their very nature are constantly evolving and supported by an enthusiastic and often very competent community. The tools reviewed in this category have been done so in their standard form, that is without inclusion of the multiple plug-ins available that address issues such as sound, physics, avatars etc. The available plug-ins are constantly evolving and could have greatly affected the performance and rating of a tool if they were included for use and assessment. This can be illustrated by the rating of G3D below in it s standard form which shows comparatively poor capability, as shown in Figure 3, below, but on closer inspection and used with additional tools G3D can clearly work with Avatars, Physics, Level Design and Sound ( Figure 3: G3D Assessment. However, if the assessment were made to include such plug-ins, it may not have been the fairest assessment since extensive searching and testing would have been required beyond the initial scope of this report. The plug-ins are constantly evolving and are often themselves open source, meaning that the view taken would only be a snap shot in time and not take account for any application specific development underway in the development community. It has been observed that the plug-in capability is extensive and evolving for most of the open source tools reviewed and the additional capability arrived at by their use should not be ignored in the choice between commercial and open source tools. This represents a dilemma for the would be developer. Another key consideration for assessing the engine you wish to develop your application with, is that not all the features you wish to exploit will be available in one engine, fairly obvious, but it does mean the potential developer needs to consider a hybrid development environment. Such a hybrid, could use different tools, or engines, selected for attributes with different modules for Physics as a common example. The preparation of a set of evaluation criteria becomes increasingly more complex when you move away from a desk based survey to a standard package code based evaluation to a hybrid application evaluation. It is the consideration of hybrid applications at which point the construction and use of a set of selection criteria ceases to be of value. The consideration of the use of such hybrid tools was clearly articulated [6], by the need to consider this evolution of plug-ins to ensure that at any one time your application uses the best of breed and by nature of the open architecture you are not locked in and restricted to a capability as is often the case with COTS tools.

5 4. CONCLUSION This paper has presented the framework of a selection methodology suitable for the application of a desk based survey of game engine development systems. The methodology was applied to a commercial project and successfully used to cull the vast array of available tools down to a manageable number and the then produce a recommendation for further code/application testing based evaluation. For the evaluation conducted the results indicated that Torque Game Engine Advanced represented the most complete development environment and best value for money for the commercial project. Torque satisfied the majority of the functional requirements and provided a capable suite of tools to aid in the development of content. However, this was based purely on assessment of the engines as-is. Subsequent experimentation showed that a low scoring engine such as Ogre, was readily extended for relatively little effort to incorporate, physics, sound. Similarly, there is an extensive selection of capable open source tools for content development. Consequently, the results of the desktop analysis methodology can be challenged. Certainly, given the substantial investment a commercial game engine can represent, further software experimentation is recommended before finalising any decision. Furthermore, consideration needs to be given to establishing a complete development environment which includes not only the games engine SDK but compatible content creation tools. REFERENCES 1. Morrison, P, etal. Proficient Soldier to Skilled Gamer: Training for COTS Success, Simtect 2005,, pp Lewis, E, Dr and Barlow, M, Dr The use of Games to Investigate Tactical Decision Making, Simtect 2005, pp Fernando, S and Stojkovski, D. Adaptation of game software development practices to simulation and operational software production. Simtect 2005, pp Loughran, J. Eenie Meenie Minie Mo Using a DSS to Select Simulations and Games. Simtect 2005, pp McShaffry, M.L Capabilities of Today s Game Engines, I/ITSEC2005 Paper McDowell, P et al Delta3D; A Complete Open Source Game and Simulation Engine for Building Military Training Systems. I/ITSEC 2005 Paper 2347