SOLVING SEQUENTIAL DECISION-MAKING PROBLEMS UNDER VIRTUAL REALITY SIMULATION SYSTEM. Wang Fei

Transcription

1 Proceedings of he 2001 Winer Simulaion Conference B. A. Peers, J. S. Smih, D. J. Medeiros, and M. W. Rohrer, eds. SOLVING SEQUENTIAL DECISION-MAKING PROBLEMS UNDER VIRTUAL REALITY SIMULATION SYSTEM Yang Xianglong Feng Yuncheng Li Tao Sysem Simulaion Laboraory School of Economics & Managemen Beijing Universiy of Aeronauics & Asronauics Beijing, , P.R. CHINA Wang Fei Insiue of Inernaional Economy Sae Developmen Planning Commission People s Republic of China Rm. B-912, Guohong Building Beijing, , P.R. CHINA ABSTRACT A large class of problems of sequenial decision-making can be modeled as Markov or Semi-Markov Decision Problems, which can be solved by classical mehods of dynamic programming. However, he compuaional complexiy of he classical MDP algorihms, such as value ieraion and policy ieraion, is prohibiive and will grow inracably wih he size of problems. Furhermore, hey require for each acion he one sep ransiion probabiliy and reward marices, which is ofen unrealisic o obain for large and complex sysems. Here, we provide he decision-maker a sequenial decision-making enviromen by esablishing a virual realiy simulaion sysem, where he uncerainy propery of sysem can also be shown. In order o obain he opimal or near opimal policy of sequenial decision problem, simulaion opimizaion algorihms as infiniesimal perurbaion analysis are applied o complex queuing sysems. We presen a deailed sudy of his mehod on he sequenial decision-making problem in Boeing-737 assembling process. 1 INTRODUCTION Sequenial decision-making means ha he decision-maker makes a series of acions according o he sysem saus as well as his preference o form a decision policy. Many problems of sequenial decision-making under uncerainy ranging from manufacuring o compuer communicaion, of which he underlying probabiliy srucure is a Markov process, can be modeled as Markov or Semi-Markov Decision Problems (MDPs or SMDPs). Such problems can be solved by classical mehods of sochasic dynamic programming. The framework of dynamic programming and MDPs developed by Bellmen (1957) and exended by Karlin (1955), Howard (1960), Blackwell (1965) is quie exensive and rigorous. Well known algorihms, such as value ieraion, policy ieraion, and linear programming can find opimal soluion of MDPs. However, hey require compuaion of he corresponding one sep ransiion probabiliy marix and he one sep ransiion reward marix using he disribuions of he random variables ha govern he sochasic processes underlying he sysem. For complex sysems wih large sae spaces, he burden of developing he expressions for ransiion probabiliies and rewards could be enormous (Das e al. 1999). Also, dynamic programming suffers from he curse of dimensionaliy and from he curse of modeling, which is why i is of lile use in solving problems wih a large sae space and complex probabiliy srucures. In he absence of beer approaches, problem-specific heurisic algorihms are ofen used o reach accepable near-opimal soluions. Recenly, compuer simulaion-based reinforcemen learning (RL) mehods of sochasic approximaion have been proposed as viable alernaives for obaining nearopimal policies for large scale MDPs wih considerably less compuaional effor han wha is required for dynamic programming (DP) algorihm. RL has wo disinc advanages over DP. Firsly, i avoids he need for compuing he ransiion probabiliy and he reward marices. The reason being ha i uses discree even simulaion (Law and Kelon 1991) as is modeling ool, which requires only he probabiliy disribuion of he process random variables bu he one-sep ransiion probabiliies. Secondly, RL mehods can handle problems wih very large sae spaces since is compuaional burden is relaed only o value funcion esimaion, ha is jus he advanage of compuer simulaion (Das e al. 1999). However, he decision-making rules are predefined in RL mehod ha means he fixed acions should be made according o corresponding saes. So i can no reflec he prejudice of decision-maker and limi his freedom in he 905

2 sequenial decision-making process. In his paper, a new decision mode (3W+N) of simulaion-based sequenial decision is proposed and a virual realiy (VR) simulaion environmen is designed for solving such problems. This new Ineracive VR simulaion sysem is especially derived from research of sequenial decision-making, and i is also buil and esed in local nework (Wang 2000). In his paper, we firs discuss he new decision mode (3W+N) of simulaion-based sequenial decision, and hen he simulaion opimizaion algorihm on sequenial decision-making under VR simulaion enviromen is proposed. How o realize he VR enviromen and simulaion opimizaion in such enviromen is also discussed. A las, he experimen for solving sequenial decision problem involving Boeing-737 secion-48 assembling manufacuring process is presened, which is one of he join venures beween China and U.S in he area of aircraf manufacuring. 2 3W+N DECISION MODE Decision-maker can ake a serial of acions in accordance wih he sysem saes and his preference in sequenial decision-making process. By his way, he risk in sequenial decision-making process should be less han ha making decisions before he implemenaion of he sysem. Many sequenial decision-making problems in engineering can be modeled as Markov or Semi-Markov Decision Problems, which can be solved by classical mehods of dynamic programming in heory. Bu dynamic programming suffers from he curse of dimensionaliy and from he curse of modeling, which is why i is of lile use in solving problems wih a large sae space and complex probabiliy srucures in real-life. So a new decision mode (3W+N) based on experimen for such research is proposed. I is designed for solving Markov Decision Problems under VR simulaion environmen. Virual Realiy (VR) is a rapidly developed inegraed echnology in recen years. I is a way for decisionmakers o visualize, manipulae and inerac wih compuers and exremely complex daa. By his way, he virual space is creaed o provide he ineracion beween human and virual real sysem displayed on compuer. In his virual world, users can experience he ineracive behavior such as looking, hearing, moving and so on. Moreover, he ineracive dynamic hree-dimensional (3D) simulaion under VR environmen is consruced by combining compuer simulaion and VR echnique ha is called VR simulaion mechanism. In his virual real world, decision-maker can obain he dynamic saes of sysem running and ener i o make series of decisions according o he sysem saes and his preference, he decision epochs is no longer deermined by predefined crierion. Therefore, decision-maker can obain more useful informaion from his near real-world sysem o make he sequenial decision process more inuiive and reliable. In his way, we propose he 3W+N decision mode ha means Where, Wha, When and Number of decisions. All of which depend on he curren saes of he sysem and he preference of he decision-makers. Where is o choose he place where he decisionmaker o make an acion. A properly chosen decision-making place can grealy represen he decision-maker s ase or preference. Wha is o decide o ake which kind of acions and he inensiy of acions, all he acions will consruc a decision se, from which he sequenial policy will be given. When and Number is o decide he ime o make decision and how many imes in he decision process. Hence, his kind of decision model is differen from he radiional one, because i can really represen he rue life. Such more agile and inuiive decision mode can reflec he random and nonperiodic propery of sequenial decision in real-life. I is obvious ha he decision place, ime, numbers and acions inensiy are all sochasic in 3W+N decision mode. Then such decision process can be viewed as he combinaion of wo processes. Firs one is formal Markov Process, i can be shown as: 1 2 m { s, s, s } S =, (1) where s denoes he sysem sae a he decisionmaking epoch. A decision making epoch, where S = i, i S and he decision aken is X = x, x X, X denoes he se of possible acions a ime. In he second process, suppose ha x is he decision aken a epoch. Then he nex decision is aken a epochs + τ where τ is a random variable beween he decision period. Suppose ha X and S be finie se, he ransiion probabiliy from s i j a epoch o s + τ a decision epoch + τ is independen o he pas saes, i.e. j i i k k j i i ( s + τ s, x, s v, x v,,) = P( s + τ s x ) P, (2) where S = Sn provided ha = T. The implicaion is ha such sochasic decision process is a Markov chain oo. Howard (1960) has proposed he proof ha he Bellman Theory (1957) is also applicable o sochasic decision process. To consruc a whole decision mode, a performance crieria is necessary, which can sum over he ime horizon and represen he resul of sequenial decisions. For in- 906

3 sance, he value of WIP (work-in-process) is he mos imporan performance for manufacuring sysem. Indeed only he crieria represened he effec of 3W+N can be chosen as he sysem performance crieria. 3 SEQUENTIAL DECISION OPTIMIZATION ALGORITHM Perurbaion analysis (PA) is a kind of gradien-based esimaion mehod in opimizaion, he mos famous one is infiniesimal perurbaion analysis (IPA) by which all parial gradiens of an objecive funcion are esimaed from a single simulaion run. The idea is ha if an inpu variable ino a sysem is perurbed by an infiniesimal amoun, he sensiiviy of he oupu variable o he parameer can be esimaed by racing is paern of propagaion. This will be a funcion of he fracion of propagaions ha diminished before having a significan effec on he response of ineres. IPA assumes ha an infiniesimal perurbaion in an inpu variable does no affec he sequence of evens bu only makes heir occurrence imes slide smoohly. The fac ha all derivaives can be derived from a single simulaion run, which represens a significan advanage in erms of compuaional efficiency. On he oher hand, he esimaors derived using IPA are ofen biased and inconsisen (Yolanda and Anu 1997). IPA esimaion will be unbiased provided ha he srucured condiion is saisfied. The essenial of such condiion is ha if he evens occurred in succession exchange he occurrence sequence because of perurbaion, sysem saes will keep immovabiliy (Huang 1997). Such four basic problems is involved in IPA algorihm: rules of perurbaion generaion, rules of perurbaion propagaion, compuaion of sample gradien and he saisics properies of sample gradien relaed o performance gradien. In simulaion enviromen, he perurbaion generaion process can be synchronously reaed wih random variables (RV) sampling process. Assumed ha he ineresed parameer is θ and F ( x, θ) is he cumulaive disribued funcion (CDF) of RV X, hen he sampling process can be realized by using inverse-ransform mehod 1 he sampling of X is x = F ( u, θ), where u is he sampling of uniform disribued RV U in inerval (0,1) (Law and Kelon 1991). If he parameer perurbaion is θ, hen he sampling corresponding o θ + θ is 1 x + x = F ( u, θ + θ), i.e. he sampling perurbaion of RV X is x. When he parameer perurbaion is infiniesimal, he raio of RV perurbaion on he parameer perurbaion is: 1 D x = D F ( u, θ) = D F θ θ hen he perurbaion of parameer is ransformed o he sampling gradien of RV. θ D x F (3) In our research, IPA is applied o assembling queuing nework. Boeing secion consiss of 9 subassemblies, which are composed of hundreds of componens or pars. Those subassemblies, componens and pars are assembled on he special jigs and equipmen named as FAJ and FME and each subassembly possesses is own FAJ and FME. Then an assembling queuing nework is consruced ha subassemblies, componens and pars can be viewed as cusomers and FAJ and FME as servers. The produc srucure ree is also provided as Figure 1. Obviously i is a very complex queuing nework, we choose he Texas Sar assembling queuing sysem as he 48-secion Major Assembling 1156 Bulkhead 1088 Bulkhead Lower Deck Module Upper Deck Module Lower Panel Upper Panel Texas Sar APU Door 1016 Bulkhead Figure 1: B secion Assembling Produc Srucure Tree research objec on sequenial decision o expaiae he algorihm for simpliciy. Texas sar assembling process is shown as Figure 2: FAJ9801L FAJ9801R 2FAJ9801L 2FAJ9801R FAJ9800 Figure 2: Assembling Process of Texas Sar FBOF9805 The manufacuring equipmen named wih heir codes used in assembling process as servers are shown in Figure 2. Lef X-beam and Righ X-beam are all composed of wo beams, so he firs beams of hem are processed on FAJ9801L and FAJ9801R separaely. Then Lef X-beam and Righ X-beam are assembled wih wo beams on 2FAJ9801L and 2FAJ9801R. The maching process of Lef and Righ X-beam and he Back Beam o form Texas sar is in succession. Afer refined processing on FBOF9805 and inspecing, he whole process ends. There are wo ses of equipmens o assembling Lef and Righ X-beam and Texas sar, one of hem are sandby. The deail assembling process can be viewed in Figure 5. According o 3W+N decision mode, he key working procedure and equipmen can be chosen as decision making place (Where), as well saring or sopping sandby equipmen and assembling 907

4 j process and changing he arrival rae or service rae can be chosen as acions or decisions (Wha) in his assembling model. Decision-maker can make acions a he epochs of equipmen failure or deficiency of assembling abiliy (When) and decide he decision numbers (Number) according o he sysem saes and his preference. Here, he performance measure includes he value of WIP C(W) ha can be expressed as v i N i, where N i and v i are he i number and cos of he i h par or componen in sysem. We can use he average sojourn ime o compue WIP in simulaion. A he same ime, i also should include he variable cos derived from decision C(D), such as he running cos of sandby equipmen and he salary of workers ha can be expressed as v j M j, where M j and v j are he number of j h equipmen and he corresponding cos on his equipmen. Usually, decreasing he value of WIP will cause he increasing of variable cos. So our objecive funcion is: { C C C( W ) C( D) } min = +. (4) Here, define he sysem before he perurbaion is added as Nominal Sysem (NS) and he sample pah of NS as Nominal Pah (NP). The relevan sysem and sample pah o perurbaion are PS and PP. Define he Generalized Busy Period (GBP) provided ha Q n, where Q is he number of cusomers in sysem and n is he number of servers. Then in decision model, he acions made as changing he arrival rae or service rae and saring or sopping he sandby equipmen a cerain working procedure will affec he sysem in 3 aspecs in his decision model. The firs aspec is perurbaion generaion o he number of pars a currenly working procedure, he second is direc perurbaion propagaion o he number of pars a working procedure jus afer he former one and he hird is indirec perurbaion propagaion o he number a oher working procedure. Such 3 ype perurbaions have heir algorihm separaely. Here, he second algorihm is presen o explain he PA mehod on he direc perurbaion propagaed working procedure: Sep 1.Define he variables in sysem: arrival ime of cusomer (ARRT) and is perurbaion (PARR), he reference ime of PA (NTIM), he arrival ime incremen of he firs cusomer in PP (NARR), he ime incremen while service begins (NSER), he adjus of dispach inerval (BTDP), he number of served cusomer (M) and he perurbaion of waiing ime (PAWT) and is oal ime (SPAWT). Sep 2. Mark he value of INDEX of arriving cusomers ype: (1)Q<n or (2)Q n. Sep 3. According o he INDEX o decide he operaing ype when he cusomer is served, hen go o sep INDEX+3. Sep4. NTIM=ARRT, NARR=PARR, NSER=PARR, PAWT=0, BTDP= he service ime of cusomer, hen go o sep 7. Sep 5. If his cusomer is served in PP in advance, i.e. ARRT+PARR<NTIM+NARR, hen {if his cusomer is he firsly arriving in GBP, hen {NSER=ARRT+PARR-NTIM} else {NSER=max{NSER, ARRT+PARR-NTIM}, he waiing ime of cusomer in PP is NSER+NTIM-ARRT-PARR, PAWT= waiing ime in PP - waiing ime in NP} } else {if he immediaely predecessor cusomer is served in PP in advance, hen {NSER=NSER+BTDP} else { NSER=NSER+ he dispach inerval of he cusomer in NP, NSER=max{NSER, ARRT+PARR-NTIM} PAWT = NSER-NARR- waiing ime in NP} } Sep 6. If his cusomer is no served in PP in advance, hen { if n>1, hen {BTDP=2*service ime of cusomer/(n-1)} else { BTDP= he dispach inerval of he cusomer in NP} } Sep 7. SPAWT=SPAWT+PAWT when service ends, M=M+1, If simulaion ends, he expeced perurbaion of sojourn ime a his server is SPAWT/M; else, go o sep 2. Then we can obain he gradien sampling from he parameer perurbaion o performance measure of sysem. In he sep n of ieraion, assume ha he ieraion sep lengh is α n and esimaion of ieraed direcion d n from he perurbaion process, he ieraed parameer θ n + 1 in ieraion sep n+1 can be received as: θ +1 = θ α d. (5) n n Usually, ieraion is processed once afer he predefined number of eniies have been served in simulaion and he erminaed condiion is ofen expressed as: n n θ n+ 1 θ n < ε (6) 908

5 where ε is a predefined small posiive number. Then he laes esimaion of parameer θ n + 1 is used o esimae he performance measure as he opimal parameer in simulaion. Here, he gradien sampling or esimaion of ieraed direcion in each sep is obained by PA. In sequenial decision process, decision-maker wans o receive reasonable suppor from simulaion opimizaion echnique such as IPA. For example, if decision-maker is no saisfied wih he processing capabiliy according o he VR simulaion sysem saes and his preference and hen he will make a decision o change he sysem parameer such as service rae. IPA can provide he improved sysem parameers a such decision epochs. Beween wo successive decision epochs, IPA can ge an ieraion resul o modify he sysem parameers a presen epoch according o he opimizaion regulaion. In he formal opimizaion regulaion, ieraion occurs when a predefined number of eniies have been served. Because of he randomness of sequenial decision, he ieraion number is no a consan and hen i will be deermined according o he decision epochs. If he ime is oo shor beween wo epochs, hen he sample size is no enough o esimae he gradien and modify he parameers. So his is a noiceable problem in solving sequenial decision problem using IPA. Bu as a Single Running Opimizaion (SRO) mehod, we ake advanage from is execuing efficiency. 4 VR SIMULATION-BASED DECISIONS AND ITS OPTIMIZATION ENVIROMENT AND EXPERIMENT RESULT In order o realize such decision-making sysem under virual realiy simulaion, a new ineracive VR simulaion sysem is esablished. This sysem is simulaion based wih he focus on he represenaion and paricipaion. Now many VR sysems have been presened, bu mos of hem need expensive equipmen, group of expers, los of ime o consruc a perfec complex VR sysem. However, he mos imporan role is simulaion in our research. We need simulaion o provide decision-maker he saisical informaion of he sysem being modeled and give he sysem saes auomaically. The major funcion of he VR sysem is o provide he decision-maker a near real world 3D animaion o observe he running of sysem. Basing on his opinion, VRML (Virual Realiy Modeling Language) is seleced o esablish such virual realiy environmen, which is an easy, economic and powerful ool o consruc he VR world. Then we can focus on he simulaion aspec and spend lile ime o code i and realize he Virual Realiy Simulaion. Because of being designed for Web, VRML has no sric requiremen for hardware. Bu i needs a web explorer plug-in o explain he VRML file and show he VR scene, his plug-in is called VRML player such as Cosmo Player an so on. Wih he explanaion o VRML file, users can observe he virual world from he monior and conrol he moving objecs and making decisions in he VR scene by using keyboard or mouse. Moreover, VRML provides he exernal inerface o realize he applicaion of simulaion jus as our demand. Because of he web propery of VRML, i provides he JAVA class as Exernal Applicaion Inerface (EAI). By using is exernal classes, users can obain he handle of sysem objec Browser. Some mehods of browser objec are very useful, such as: gebrowser(), creaevrmlfrom- Sring(), genode(), addroue() and deleeroue(). From he browser, we can ge he handle of objec in VRML scene by mehods gebrowser() and conrol is behavior, and display he simulaion process by VRML. Combining hese mehod and field addchildren and removechildren in VRML nodes, eniies can be added and removed dynamically in he virual world. Using he correc combinaion of TimeSensor, PosiionInerpolaor and Orienaion- Inerpolaor Node, he Roue() mehod can drive he objec move as he proper logic a he suiable ime. (Wang 2000 and Marrin 1997). Moreover, simulaion programming has he naural inerface o JAVA, and he laer is an objecoriened programming language. Many JAVA-based simulaion programs have appeared, such as JavaSim(Lile, M.C, 1996 ), SimJava (McNab, R., 1996), ec. They provide he chance o combine he Java-based simulaion program and VRML file, he laer wo can be inegraed ino one web page o display he simulaion animaion process. The VRML browser reads and explains he VRML file and builds a 3D virual world. The VRML file jus ells he browser how o build he predefined virual world, he simulaor accomplishes he simulaion work and drives he virual world o run dynamically. Then simulaion programming can combine he VRML by imporing VRML exernal inerface and JAVA-based simulaions class o consruc VR simulaion sysem easily (The VRML Consorium Incorporaed 1997). In our research, such VR simulaion sysem is consruced o provide he funcion o making sequenial decision and opimizing he parameers wih above algorihm. All hese funcions can be realized hrough he Java Apple in HTML pages and JDBC driver suppor o access daabase used o sore he simulaion daa and display he improved parameers. The sysem framework can be viewed as Figure 3 and he VR simulaion-based sysem is shown in Figure

6 HTML VRML Scene VRML Plug-In EAI Java Apple JDBC DaaBase Figure 3: Mechanism of VR Simulaion-Based Decision and Opimizaion Sysem In he VR simulaion enviromen shown in Figure 4, he separae 3D objecs in his scene are esablished wih 3DSMax, he famous 3D animaion sofware, and can be conver ino he VRML file forma by using he funcion Expor o. Some oher sofware as Inerne 3D Space Builder, AC-3D ec., can also provide such funcion. By his way, we can build complex 3D models more easily bu he size of hese files is ofen bigger han ha direcly buil in VRML. There is a ip o sele his problem parially: in some 3D building sofware, a funcion is provided Opimize o remove some duplicae verices and surfaces from objecs and do no influence heir 3D effec. Then he convered file size will be reduced. In VRML file, he world coordinae space for all objecs is defined, and all objecs included in he file by he Inline node. And he animaion according o simulaion resul is realized wih he mehod of combining VRML EAI and Java-based simulaor as menioned above. In Figure 4, he lef scene in Inerne Explorer is a VR scene and he righ one is Java Apple where he inpu parameer can be changed for decision-making and he simulaion oupu is displaying. The buons in Apple are used o sar animaed or numeric simulaion, pause simulaion, resar simulaion and run opimizaion o obain improved parameers. By using such simulaion sysem a model was developed for pracical assembling sysem Texas Sar as menioned above, which saves los of cos and ime for he cooperaed facory. In his assembling process, he supervisor should ake accoun of he resricions of he loading capaciies, and decide wheher he backup machine should be sar or no when he main assembling machine is failed or he producion capaciy is no enough. The supervisor will Figure 4: VR Simulaion-based Sysem of Boeing-737 Texas Sar Assembling decide he nex acion based on presen working saus and final objecives. In his assembling model, we have se 5 decision-making place (where), 10 kinds of decisions (wha), and he decision epochs and imes can be decide by he supervisor o make he minimal performance measure C in formula (4). Then an experimen of such sequenial decision and opimizaion under VR simulaion enviromen is presened. In his experimen, acions include changing he arrival rae of Lef X-beam, Righ X-beam and Back Beam, saring he sandby equipmen of X-beam and suspending cerain assembling service and so on. The iniial experimen condiion of average arrival and service rae is shown in Table 1. Table 1: Arrival and Service Rae of Pars and Equipmen Lef X- beam Righ X- beam Back Beam FAJ9801L/ R Sandby FAJ9801L/ FAJ9800 Sandby FAJ9800 FBOF9805 R The numbers in Table 1 marked wih Lef X-beam, Righ X-beam and Back Beam are heir average arrival rae and he numbers marked wih he name of equipmen are heir average service rae. The arrival of cusomers is Poisson process and service ime is subjec o exponenial disribuion. By running simulaion in VR enviromen, we can ge he informaion from VR scene and simulaion oupu in Apple ha he assembling capaciy of Lef, Righ X-beam and Back Beam is absence o mach ha of refined processing on Texas Sar. The lengh of queues increases rapidly o make he oversock of WIP, which is shown in Figure

7 Table 3: The comparing experimen resuls Iniial Experimen Modified Experimen Toal number of producs Simulaion lengh Number of WIP Queue lengh of Lef X-beam 12 8 Queue lengh of Righ X-beam 8 5 Queue lengh of Back Beam 8 5 Queue lengh of Refined Sar 3 1 Figure 5: Number of WIP in Iniial Simulaion The verical axis shows he number of WIP and he horizonal one shows he number of producs compleed assembling in simulaion. In order o solve such problem, sequenial decision is made as following sequence shown in Table 2: Table 2: Decision Sequence in Experimen Sequence Place Acion Time 1 Sandby Saring sandby FAJ9801L/R equipmen FAJ9800 Decreasing arrival rae of Back Beam Sandby Saring sandby FAJ9801L/R equipmen FBOF9805 Suspending service FAJ9801L/R Decreasing arrival rae of Lef X- Beam 5734 From sequenial decision as above and run opimizaion a decision epochs wih IPA, we can obain new resul of simulaion running as Table 3 and Figure 6: Figure 6: Number of WIP in Modified Simulaion The numbers of WIP, Queue lengh of Lef X-beam, Righ X-beam, Back Beam and Refined Sar in Table 3 are all heir average in he simulaion. I can be viewed ha he simulaion resul processed wih sequenial decision and opimizaion is beer ha iniial one. 5 CONCLUSION This paper proposes he opimizaion algorihm on sequenial decision problem based on VR simulaion sysem. By using such simulaion sysem, a model abou real sysem was developed o provide he decision-maker a virual world o making decisions wih he suppor of opimizaion algorihm. The VR Boeing secion assembling simulaion sysem makes our cooperaion facory save los of cos and ime in esablishing he assembling line. The properies of man-compuer ineracion in VR and describing he uncerainy in simulaion provide he feasible mehod and convenien experimen space o analyze he man-in-he-loop decision problem. This simulaion sysem can be no only used o rain decision-makers: eiher novice decision-makers or poenially esablished decisionmakers by comparing he decision resul o opimized one, bu also o operae he real-life faciliy. Then we can see his special simulaion environmen is he new rends of oday s simulaion sofware, Visual Simulaion and VR Simulaion. As for furher developmen, he inegraed developmen environmen is under esablishing. In such VR simulaion sysem, every simulaion user can easily code and compile he model and consruc he VR scene according o his demand. ACKNOWLEDGMENT This work is suppored by Naional Science Foundaion of China Gran o Beijing Universiy of Aeronauics and Asronauics. 911

8 REFERENCES Bellman, R. E Dynamic Programming. Princeon Universiy Press, Princeon, NJ Blackwell, S. J Discree dynamic programming. Ann. Mah. Sa., Cinlar, E Inroducion o Sochasic Process. Prenice-Hall, Inc., Englewood Cliffs, New Jersey Das, T., Gosavi, A., Mahadevan, S. and Marchallek, N Solving semi-markov decision problems using average reward reinforcemen learning. Managemen Science, Vol. 45, No. 4 Howard, R Dynamic Programming and Markov Processes. MIT Press, Cambridge, MA. Huang Hongxuan Perurbaion Theory of Discree Even Sysem and Single Run Opimizaion via Simulaion. Docor Degree Disseraion of Beijing Universiy of Aeronauics and Asronauics, Beijing Karlin, S The srucure of dynamic programming models, Naval Res. Logis., Quar Law, A. M., W. D. Kelon Simulaion Modeling and Analysis. McGraw Hill, New York Lile, M.C JavaSim Homepage, <hp:// marlish.ncl.ac.uk:8080/javasim> Marrin, C Proposal for a VRML 2.0 Informaive Annex. hp://realiy.sgi.com/cmarrin/ vrml/exernalapi.hml McNab, R A Guide o he SimJava Package, Deparmen of Compuer Science, Universiy of Edinburg, UK,<hp:// e/simjava/simjava-1.0> The VRML Consorium Incorporaed The Virual Realiy Modeling Language Inernaional Sandard ISO/IEC :1997 Wang Fei Research on Theory and Applicaion for Simulaion Decision under Virual Realiy. Docor Degree Disseraion of Beijing Universiy of Aeronauics and Asronauics, Beijing Wang Fei, Feng Yuncheng, Wei Youshuang Virual Realiy Simulaion Mechanism on WWW. In Proceedings of AreoScience 2000 of SPIE Yolanda Carson, Anu Maria Simulaion Opimizaion: Mehods and Applicaion, Proceedings of he 1997 Winer Simulaion Conference, ed. S. Andradóir, K. J. Healy, D. H. Wihers, and B. L. Nelson FENG YUNCHENG is a Professor a he School of Economics & Managemen, Beijing Universiy of Aeronauics & Asronauics. He has served as honorary Presiden of Compuer Simulaion Associaion of China. His curren research ineress include simulaion opimizaion, simulaion oupu analysis, manufacuring sysem simulaion, Virual Realiy echnology and Web based simulaion. WANG FEI had obained his Ph.D. degree a Beijing Universiy of Aeronauics & Asronauics in His curren research ineress include Macroeconomics analysis and WTO sraegy of China. LI TAO is a Ph.D. suden a he School of Economics & Managemen, Beijing Universiy of Aeronauics & Asronauics. His curren research ineress include mulimedia simulaion and Web-based simulaion. AUTHOR BIOGRAPHIES YANG XIANGLONG is a Ph.D. suden a he School of Economics & Managemen, Beijing Universiy of Aeronauics & Asronauics. His curren research area is Virual Realiy echnology and simulaion opimizaion. 912