A Dynamic Networking Substrate for Distributed MMOGs

Transcription

1 Receved 31 Octobe 2013; evsed 9 Apl 2014; accepted 9 June Date of publcaton 00 xxxx 0000; date of cuent veson 00 xxxx Dgtal Object Identfe /TETC A Dynamc Netwokng Substate fo Dstbuted MMOGs MOHSEN GHAFFARI, BEHNOOSH HARIRI, SHERVIN SHIRMOHAMMADI, Seno Membe, IEEE), AND DEWAN TANVIR AHMED Dstbuted and Collaboatve Vtual Envonment Reseach Laboatoy, Unvesty of Ottawa, Ottawa, ON K1N 6N5, Canada CORRESPONDING AUTHOR: S. SHIRMOHAMMADI dshevn@gmal.com) ABSTRACT Ths pape poposes a dstbuted and dynamc netwokng achtectue fo massvely multplaye onlne games MMOGs). The MMOG netwoks deal wth the challenge of update message exchange among a lage numbe of playes that ae subject to both moblty constant change of vtual locaton) and chun jonng and leavng the physcal netwok at wll). Ideally, a playe s update messages should be multcasted to the playe s aea of effect, whch s a neghbohood aound the playe. But, ths eques the system to have a centalzed ndexng sevce that keeps ecod of playes esdng n each egon, makng the system less scalable as the numbe of playes nceases. The use of geometc outng helps allevate ths equement by explotng locaton addessng and thus elmnatng the need fo IP-seach quees. Howeve, geometc outng comes wth a numbe of convegence and pefomance ssues that eque solutons fo educng hop-count and mnmzng oveall delay whle povdng guaanteed message delvey. In ths pape, we popose a geometc outng ovelay fo message exchange among a lage numbe of MMOG playes that not only povdes educed delay and guaanteed delvey, but also suppots playe moblty and chun. In addton, we enhance ou geedy outng method to moe effcently suppot long dstance messages. We demonstate the effectveness of ou poposed scheme usng both theoy and smulatons. INDEX TERMS MMOG, Delaunay tangulaton, geometc outng, P2P netwoks. I. INTRODUCTION Massvely Multuse Onlne Games MMOG) ae vtual wolds whee a lage numbe of uses dstbuted all ove the Intenet can nteact wth each othe n eal-tme. Such envonments ae vey popula wth tens of mllons of playes [1] spendng sgnfcant tme and money nteactng wth one anothe n games. Among many challenges, MMOGs must deal wth eal-tme exchange of update messages among a lage numbe of playes n ode to povde them wth a common sense of tme, place, and game state. Thus MMOG netwok desgn nvolves dstbuton of update messages n ode to meet stct end-to-end delay constants, between 100 to 1000 msec [2] among a vey lage goup of uses, n the mllons. Real-tme scalable suppot fo such magntude of playes hnts at the usage of ethe fully dstbuted achtectues whee thee s no cental swtchng pont and thus no potental scalablty bottleneck, o hybd achtectues whee a centalzed system such as a cloud s futhe enhanced wth a dstbuted delvey mechansm such as P2P netwokng fo even hghe scalablty [3]. In ethe achtectue, thee wll be a dstbuted netwokng component, whch entals solvng the followng man poblem. In a dstbuted achtectue, outng of the update messages becomes challengng because MMOGs eque each playe a node n the netwok) to send updates to a specfc aea n the game known as that node s aea of effect, n a eal-tme manne such that playes n that aea of effect ae awae of the node s latest state. The use of IP outng n ths scenao s neffcent as t would eque a node to send fequent locaton quees to dscove who s located n ts aea of effect. On the othe hand, the use of geometc outng s one way to allevate the need fo such fequent locaton quees. Geometc outng s a type of dstbuted outng whee updates can be sent to specfc coodnates nstead of specfc IP addesses. Ths allows an MMOG node to send update messages to a specfc coodnate n the vtual space ts aea of effect) wthout the need fo queyng the IP addess of the nodes n that aea o havng any global knowledge of the topology [4], [5], VOLUME X, NO. X, MARCH XXXX IEEE. Tanslatons and content mnng ae pemtted fo academc eseach only. Pesonal use s also pemtted, but epublcaton/edstbuton eques IEEE pemsson. See fo moe nfomaton. 1

2 sgnfcantly educng complexty and ovehead. As such, geometc outng s an attactve soluton fo MMOGs n dstbuted o hybd modes. Among many geometc outng potocols, geedy outng s the most commonly used due to ts speed, smplcty, and convegence ate. Howeve, geedy outng has the shotcomng that message delvey cannot be guaanteed when outng s pefomed ove a genec ovelay. Even though seveal modfcatons have been poposed to fx ths poblem, guaanteed convegence has been ganed only at the pce of much longe convegence tme and hence hghe end-to-end delay that s not desable n MMOGs. In ths atcle, we pesent a soluton to acheve guaanteed delvey n geedy outng wthout sacfcng latency. We do so though caeful desgn of the outng ovelay and by keepng the outng scheme smple and shftng most of the complexty to the ovelay tself. We also solve the challenge of playe chun and moblty by poposng a dstbuted and dynamcally updatng ovelay system whch emoves the need fo a complex outng scheme that would slow down the game. FIGURE 1. Voono dagam sold) and delaunay tangulaton dashed). We have pevously poven [4] that geedy outng can only povde guaanteed delvey ove a outng substate ff that substate satsfes the Delaunay Tangulaton constant DT, see Fgue 1). DT s defned as follows: fo a set P of ponts n the plane Delaunay Tangulaton, DT P) s a specfc tangulaton of the ponts such that no pont n P s nsde the ccumccle of any tangle n DT P). Howeve the geneaton of a Delaunay gaph eques global knowledge of the netwok topology and nvolves a hgh computatonal ovehead fo lage numbe of nodes, whch s the case n MMOGs. In ths atcle, we also addess ths poblem and desgn a dstbuted Delaunay suppot gaph fo MMOGs such that t satsfes two condtons: fst, we eque a totally dstbuted soluton whee thee s no cental pont of falue, and second, as netwok dynamcs constantly change, the ovelay gaph should be epeatedly updated aganst any changes that affect the Delaunay suppot constants. Pehaps the most mpotant obstacle to acheve the above s the lack of any efeence pont fo all geometc opeatons. Ou poposed soluton s to patton the ovelay nodes nto two goups efeed to as Red and Black goups. Each of these goups foms a gaph that can be updated usng the othe gaph as a efeence. We wll also show how the oveall topology wll be constucted based on the two sub-gaphs. Futhemoe, even though the Delaunay ovelay helps geedy outng to acheve ts best pefomance, anothe majo shotcomng stll pessts: geedy geometc outng, pmaly ntoduced n the context of weless outng, s not ntended to pefom heachcal outng due to the lmted tansmsson ange n weless netwoks. Howeve, such lmtaton does not exst n MMOGs. Theefoe, falue to use heachcal outng may esult n a lage numbe of hops n outng long-dstance messages. We wll late addess ths concen by ntoducng dstant outng. The wok n ths atcle s the contnuaton of ou wok n [6] and has the followng contbutons: 1) Poposng a new scheme fo topology contol and mantenance n geedy suppotng dynamc ovelays fo MMOGs whee uses fequently jon, leave, and change the locatons n the vtual wold. 2) Intoducng a dstbuted state management scheme fo shaed objects n the MMOG, whch was unaddessed n [6]. 3) Enhancng the outng nfastuctue n [6] by addng a Dstant Routng mechansm that educes the numbe of hops to oute messages between nodes that ae long dstances away fom each othe n the ovelay. 4) Pefomng extensve new smulatons whee the newly-ntoduced dstant outng scheme s smulated, analyzed, and shown to be hghly effectve wth a lage numbe of nodes 12,000) ove a aea. Ou poposed soluton can be appled to both fully dstbuted and hybd cloud + P2P) achtectues, though n ths pape we concentate only on dstbuted netwokng. It should be noted that pats of ths pape s sectons III to VI have some ovelap wth ou pevous pape [6]. The eason s to make ths pape self-contaned and to fee the eades fom the nconvenence of havng to constantly efe to [6] fo mathematcal and techncal detals. Afte dscussng the elated wok n Secton II, Secton III coves the geneal poblem model and equed pelmnaes to the poposed soluton. Secton IV dscusses the geneal achtectue, whle Secton V pesents the ovelay updatng theoy and futhe detals of the achtectue. Secton VI addesses consstency ssues, whle Secton VII descbes the nfastuctue fo state management. Secton VIII coves the poposed dstant outng achtectue, whle Secton IX pesents the smulaton esults. Fnally, the last secton ncludes the summay and concluson. II. RELATED WORK Many dstbuted MMOG achtectues have been poposed n the past decade. At a hgh level, [7] categozes these achtectues nto 6 man goups: wold pattonng, DHT, multcast, fully connected, neghbo-lst exchange, and mutual notfcaton appoaches, whle [8] dvdes them nto stuctued, unstuctued, and hybd, the latte 2 VOLUME X, NO. X, MARCH XXXX

3 tself dvded nto seve+ip multcast, seve+unstuctued, and seve+stuctued. Ou wok n ths pape focuses on mutual notfcaton accodng to [7], and stuctued fully-dstbuted MMOGs) o hybd: seve+stuctued cloud+p2p MMOGs) accodng to [8]. Mutual notfcaton achtectue eles on the geometcal popetes of ts undelyng P2P ovelay to mantan pee connectvty, whle stuctued achtectue uses a detemnstc potocol to fom a specfc P2P gaph stuctue that ensues any node can oute a message to any othe node. In both of these achtectues, the pees ae esponsble to collaboatvely buld and mantan a gaph stuctue and the connectvty among the playes s defned based on the locatons n the vtual envonment and the aeas of effect. But befoe descbng how these achtectues ae used to buld ou poposed system, let us take a look at some of the exstng elated wok n ths aea. SCORE s a multcast communcaton potocol fo Lage-Scale Vtual Envonments [9]. It suppots multple multcast goups and multple agents to a handle lage numbe of concuent patcpants. SCORE dynamcally pattons the vtual wold nto spatal aeas and assocates the aeas wth multcast goups. Fo ths pupose, t apples a plana pont pocess to detemne the pope cell sze. Thus t ensues that taffc at the eceve sde emans below a theshold wth a gven pobablty. On the othe hand, Tsa et al pesent a outng potocol usng vtual coodnates fo weless senso netwoks named ABVCap that establshes an equato and a numbe of medans, and assgns multple vtual coodnates to senso nodes n a fou-phase pocess [10]. ABVCap outng guaantees packet delvey wth no computaton and stoage of global topologcal nfomaton. But to be opeatonal, nodes must be statc n ABVCap. Thus t s nappopate fo MMOG whee nodes ae dynamc and move n the vtual space. Pobably the most staghtfowad appoach to geogaphc outng s geedy fowadng. Ths s benefcal especally n densely populated aveage-case netwoks. Also n wostcase netwoks, the cost mposed by geedy outng cannot become abtaly hgh. Howeve, geedy fowadng s not guaanteed to always each the destnaton, and so a numbe of wokaounds have been poposed to fx that. Face outng, fo example, offes guaanteed message delvey wth a cost of On), whee n s the numbe of netwok nodes [11]. Howeve, ths s unsatsfactoy as a smple floodng algothm wll each the destnaton wth On) messages. Kuhn et al. pesent GOAFR + [12] fo weless ad hoc netwoks combnng the featues of geedy fowadng and Face outng, guaanteeng that the destnaton wll be eached, but leadng to a fa less effcent pefomance. Bose et al. popose anothe algothm whch s compettve wth the shotest path between souce and destnaton on Delaunay tangulatons [13]. The latte authos also demonstate Delaunay Tangulaton DT) as one of the stuctues that suppots geedy outng;.e. geedy outng on DT guaantees message delvey. Due to ths popety, many eseaches have ted to come up wth a dstbuted DT constucton algothm, whch we dscuss next. Lebehe et al. popose a potocol to buld a dstbuted DT substate [14] by explotng the locally equangula popety of DT n the 2D space. The basc dea s that each node has a pe-assgned logcal coodnate n the plane and checks whethe the equangula popety holds among tself and ts neghbo nodes. Wheneve a volaton s detected, the node flps tangles to mantan a coect DT. Flppng tangles s a basc step that s used n a numbe of Delaunay Tangulaton algothms, and woks as follows: consde a tangle wth nodes at a, b, and c. Now suppose that the ccumccle of the tangle abc contans a node p. Note that ths should not happen n a Delaunay Tangulaton. To fx ths ssue, the flppng step eplaces tangles abc and bcp wth tangles abp and apc. As DT s estctve n the context of dynamc vtual envonments, an altenatve ovelay stuctue, named Relaxed Tangulaton RT), s poposed by Elya et al. compomsng DT s equangula popety constant [15]. The same authos also pesent a message outng potocol ove the RT usng local knowledge of pees [16] by fomng a connectvty achtectue and povdng guaanteed message delvey ove the ovelay stuctue. Whle the esults ae geneally good, expements show hghe latency n some patcula cases. Stene et al. suggest anothe dstbuted appoach to constuct a dstbuted DT n the 3D space [17]. Upon jonng of a new node, the tetahedon that contans the jonng node wll be splt n two. The new tetaheda wll then be checked and flpped f they nclude any nodes n the ccum-sphees. Lee et al. pove that a dstbuted DT algothm s functonal f and only f t meets a specfc equement [18]. They popose two jon and leave algothms and povde the functonalty poof fo sequental jons and leaves usng the pecedng cteon. But the poposed algothm assumes that only one node jons o leaves at a tme whle a mantenance scheme has been used to handle concuent jons and leaves, as well as node falues. The same authos have suggested a newe potocol fo jon, leave and mantenance n [19]. Howeve, consdeng the fast and fequent moton of nodes n an MMOG, these algothms fal to mantan the stuctue n the pesence of multple nodes motons o depatues. It s well-known that unless an algothm povdes a geat degee of edundancy and acts quckly) n dealng wth falues, dsconnectons, and chun, t wll not wok well n open dstbuted systems such as MMOGs [7]. Smla to the pevous woks, Hu et al. popose VON: a set of pocedues n the case of jon, leave and move n ode to etan the dstbuted DT [20]. The jon and leave pocedues ae smla to [18], although VON handles the moton wth a sepaate pocedue. VON ntoduces the new concepts of bounday and enclosng neghbos. Upon the movement of a node, poston updates ae sent to all of ts cuently connected neghbos. Then, the bounday neghbos wll help the movng node to fnd new neghbos to connect to. Wth the moton of one node, pevous lnks to the bounday VOLUME X, NO. X, MARCH XXXX 3

4 nodes would be checked and deleted f the bounday nodes ae outsde of the movng node s aea of effect. In ths pocedue, t s assumed that all of the othe elevant nodes do not change the locatons and they have elable lnks between themselves. Howeve, as dscussed befoe, the majo dffculty of handlng dynamc stuctues n MMOGs s due to the fequent and concuent moton of nodes. As such, the assumpton of sngle node movement at a tme s not ealstc. Ou poposed appoach s compaable to [18] [20]. Howeve, unlke these woks, we do not need sepaate pocedues fo jon, leave, and move pocedues. Instead, we popose a new dea whee the achtectue peodcally updates tself to take such changes nto account. It should also be mentoned that, n all of the above appoaches ncludng ous, MMOG pees collaboate n the message exchange pocess. Theefoe, the poblem of cheatng and denal of collaboaton cannot be avoded. Whle ths poblem s beyond the scope of ths pape, we efe the eades to Baughman et al. who popose an asynchonous synchonzaton AS) potocol povdng cheat-poof and fa play-out of both centalzed and dstbuted netwok games [21]. The AS allows optmstc executon of events wthout the possblty of conflctng states due to packet loss o cheatng, and wthout the need fo the well-known ollback technque. Nowadays avatas, equpment, and vtual cuency ae taded fo eal money. So, thee mght be ll motvaton and cmnal enegy fo cheatng. Cheatng s always a dange fo all MMORPGs. Sebastan pesents a cheat handlng concept whch states that neve update the state of a node based on the states of othe nodes but each node updates ts state based on ts own old state and the actons of playes [22]. It s also shown that event-based smulaton nstead of the standad update loop solves both cheatng and tme synchonzaton poblems. Wth the above nfomaton n mnd, let us now descbe the detals of ou poposed system. We stat ths by explanng the theoy and poblem model n the next secton. III. PROBLEM MODEL AND THEORETICAL BACKGROUND In ths secton, we pesent a detaled explanaton of the poblem and all the equements and objectves. We wll then descbe the foundaton of ou poposed dynamc achtectue as a soluton to ths poblem n the next secton. Befoe mentonng the poblem specfcaton, we stat by pesentng defntons that wll late be used n the poblem descpton. A. DEFINITIONS Voono Cell Consde a set of ponts V, n the Eucldean plane whee each pont n ths set epesents the poston of an avata n the vtual envonment. Fo each pont v V, Voono Cell of v, VCv ), s defned as the set of all ponts n the plane that ae close to v than any othe pont n V. It should be noted that accodng to ths defnton, Voono Cells of the membes of V patton the Eucldean plane. Delaunay Tangulaton Pesumng a set of ponts V, Delaunay Tangulaton of V s a gaph G = V, E) wth the edge set E that satsfes the followng popety e = v, v j ) E, ff VCv ) and VCv j ) have a sde o at least a pont) n common. If VC v ) and VCv j ) shae a sngle pont, the edge e = v, v j ) s called unusual edge. An example of Voono Dagam and Delaunay Tangulaton s pesented n Fgue 1. Vetex Regon Let G = V, E) be a gaph on a set of ponts V. Fo each v V, let Nv ) denote the set of neghbos v. We also defne Vetex Regon of v n gaph G, VR G v ), as the set of all ponts n the plane that ae close to v than to any othe pont n Nv ). Fgue 2 llustates VR G v ) wth a ed tangle, whee the nodes n the ed tangle ae close to v than to any of ts neghbos. Remak 1: Snce fo each v V, we have Nv ) V, thus, VCv ) VR G v ). FIGURE 2. Vetex egon of v, ndcated by the ed tangle. As Voono Dagam pattons the plane accodng to poxmty constants the closest node to each pont s the node that les n ts petnent Voono Cell), thee has ecently been a tendency towads the use of Voono Dagam o Delaunay Tangulaton as ts dual gaph) as the base fo state management. Tumbde et al. have poposed a Voono based pattonng appoach to suppot state management n MMOGs whee the nodes n each Voono patton communcate though the coodnato of that patton [23]. Theefoe, the gaph of connectons acts as a cluste of sta gaphs whee neghbong stas ae connected va the cente nodes. Howeve, the authos state that due to the complexty of updatng the Voono dagam due to node movements, an effcent dstbuted soluton s equed. Chen et al. pesent Delaunay State Management DSM) whee Delaunay Tangulaton s used to patton the vtual wold nto egons and each egon s managed by thee supe pees [24]. These supe pees ae handled by seves and theefoe ae not subject to hgh moblty. Hu et al. popose Voono State Management VSM) that pattons the Vtual Envonment by consdeng use nodes as ponts of the Voono Dagam [25]. VSM uses the pevously dscussed VON as ts ovelay netwok. Howeve, as dscussed aleady, VON does not suppot concuent movement of nodes. As mentoned n [26], the man dffculty wth these schemes s that they ae not dynamc enough to mantan a consstent topology n the pesence of smultaneous movement of nodes. As ou fst contbuton, we solve ths 4 VOLUME X, NO. X, MARCH XXXX

5 poblem usng ou poposed Red-Black achtectue. But fst, we need to defne the theoetcal aspects of the poblem, shown next. B. PROBLEM DEFINITION As mentoned befoe n Secton I, the use of geometc outng allevates the need fo IP-quees and lets nodes dectly multcast the updates to the aea of effect whee the nteested enttes ae located. Geedy outng, whch s among the smplest and most common onlne geometcal outng algothms, tes to bng a message close to the destnaton n each step usng only local nfomaton. Thus, each node fowads the message to the neghbo that s most sutable fom a local pont of vew. The most sutable neghbo can be the one who mnmzes the dstance to the destnaton n each step. Gaph G = V, E) suppots geedy outng ff geedy outng on G delves each and evey packet to the closest node n V to the packet s destnaton. It should be noted that f the destnaton of a packet exsts n V, geedy outng should delve the packet to the exact destnaton. Theefoe, n ode to assue guaanteed delvey of updates whle usng geedy outng, the ovelay topology should suppot geedy outng. The challenge would then tun nto the dynamc constucton of an ovelay topology that suppots geedy outng. We shall theefoe popose a dynamc topology contol scheme that can handle hgh chun and node moblty n MMOGs. In the followng sub-secton, we wll show the analytcal descpton of geedy outng suppot ove a gaph. We wll then pesent ou appoach fo the geneaton of geedy suppotng gaph n secton IV. C. THEORETICAL FOUNDATION OF THE PROPOSED APPROACH The poposed soluton towads the suppot of a Delaunay gaph eles on the followng two theoems: Theoem 1: Gaph G = V, E) on the node set V n the Eucldean plane suppots geedy outng ff fo evey node v V, VR G v ) = VCv ). Theoem 2: Gaph G = V, E) ove the set V n the Eucldean plane suppots geedy outng ff fo all v, v j V, VR G v ) VR G vj ) =. The poof of these theoems can be found n [4]. Theoem 2 povdes an nteestng nsght to the gadual update of a andom gaph towads a geedy suppotng stuctue. Based on theoem 2, we can modfy any abtay gaph to tansfom t nto a geedy suppotng gaph. In ode to effcently esultng n mnmum numbe of connecton) tansfom a gaph to a geedy suppotng one, t suffces to connect each pa of nodes whose VR G s have a nonempty ntesecton and delete the edundant edges aftewads. In the followng sectons, we wll dscuss ths soluton n detal. IV. RED-BLACK ARCHITECTURE Ou poposed achtectue s genec and composed of a geedy suppotng gaph coveng all MMOG nodes. Ths ovelay wll be used fo any knd of message exchange, except fo postonal update and topology contol messages. We mentoned that nodes ae usually n movement and they use locaton update messages fo announcng the new locatons to the othe nodes. A geneal method fo synchonzng nodes s to send the locaton update messages peodcally wth a specfc ate. Theefoe, evey T seconds, nodes ae supposed to geneate locaton update messages and send t to the aeas of effect. T could be anywhee fom 10 to 280 msec, dependng on the game gene [27]. The challenge of sendng these updates n a dstbuted topology s the lack of a stuctue that can be used fo tansfeng the update messages to the elated aeas. In ode to deal wth ths poblem, we have devsed a dual phase topology update mechansm whee the nodes ae pattoned nto two non-ovelappng sets efeed to as Red and Black sets, whee each s modfed n one of the update phases. The whole topology gaph s composed fom ed and black sub-gaphs and these two subgaphs ae kept up-to-date to suppot the geedy outng;.e., they ae DT. As a esult, the man gaph wll also suppot geedy outng. An example of the smultaneous ed and black gaphs s shown n Fgue 3. Each ed and black set s able to use the othe set as ts landmak n ode to update tself. As such, we colo ou nodes wth Red and Black. Each goup geneates and dstbutes ts update messages wth a T /2 tme shft wth espect to the othe goup. FIGURE 3. The ed-black achtectue. Assume that ed nodes send the locaton updates n tmes 0, T, 2T,... and black nodes send the updates n tmes T /2, 3T /2, 5T /2,... At the tme of updates fo ed nodes, the ed gaph has pobably changed due to the movement of ed nodes and t mght not suppot geedy outng anymoe, causng ou geneal gaph to fal to suppot geedy outng. Howeve, the good news s that the black stuctue, whch has just been updated T /2 seconds ago), stll suppots geedy outng. Of couse, dung ths T /2 seconds, some of the black nodes mght have also moved fom the pevous locatons; howeve, the change of the postons has not yet been announced. Consdeng the postons n the MMOG as vtual concepts, the poston change of the black nodes wll take effect upon the own update phase and can be gnoed fo the update of the ed gaph. The same method wll be used dung the update of black nodes, except that ths tme the ed stuctue s used as the elable stuctue. VOLUME X, NO. X, MARCH XXXX 5

6 These two sets can be selected abtaly, although unfom dstbuton of ed and black nodes n the plane can help mpove the convegence ate. A smple and pactcal method can be the use of IP-addesses n the goupng pocess whee even and odd IP addesses ae assgned to ed and black goups, espectvely. The othe ssue that affects the pefomance s that t eques the nodes to stay synchonzed such that the update pocedue can un n a common tme nteval. Fo MMOGs, we can use two synchonzaton schemes: dstbuted synchonzaton va message exchange, and the contnuaton of a global tme. The fome can ntoduce ovehead n tems of message complexty and tme delay, thus affectng playes nteactvty and game qualty. Developes can use the mechansm of a global tme lke Netwok Tme Potocol NTP) whch fts well n ths envonment, as t s a potocol fo synchonzng the clocks of compute systems ove vaable-latency data netwoks. In the followng secton, we wll dscuss the updatng mechansm of ou algothm, and mathematcally pove ts functonalty. V. RED-BLACK TOPOLOGY UPDATE Though the est of ths secton, wthout loss of genealty, we wll explan the ed goup update phase, knowng that the update phase fo black nodes s smla. The algothm used n ths update phase wll allow the ed nodes to exchange the vtual locaton updates and then change the elated ed subgaph so that t esumes suppotng geedy outng. We efe to ths algothm as the esumpton algothm. Next, we descbe ths algothm. A. UPDATING RED STRUCTURE USING THE BLACK GRAPH At the begnnng of the update phase, the ed nodes announce the new locatons to the pevous ed neghbos. Theefoe, each node gets to know the new locaton of ts old neghbos n the ed gaph and uses ths nfomaton to calculate ts Vetex Regon. It should be noted that thee s no sngle pont whee all of the equed nfomaton fo checkng the vetex egons ntesectons s gatheed. Instead, a dstbuted method s exploted whee the nodes n the black goup collaboate n ode to calculate the ntesectons between the ed vetex egons. The specfcaton of each VR R s sent to the black nodes whose VR B have at least a pont n common wth the VR R. Assume the case when a black node named v b j fnds a common pont such as xbetween the two ed VR R s whee x VR B v b j ). The vb j wll then notfy one of the nvolved ed nodes to make a lnk wth the othe one. An example of ths scenao s pesented n Fgue 4. Late we wll efe to the desed set of black nodes fo evey ed node v, the Black Paents BP) of v, whee BPv ) = {vb j VR Bv b j ) VR Rv ) = }. The poblem s how to delve the VR R v ) specfcatons to the membes of BPv ). Ths can be solved by fst delveng the message to one of nodes n BPv ) and then popagatng t to the othe nodes of ths set. We wll dscuss each of these two steps sepaately. FIGURE 4. The dstbuted method to checkng ntesectons. 1) MESSAGE DELIVERY TO A MEMBER OF BLACK PARENT SET Stojmenovc et al. have nvestgated the soluton to a poblem smla to ths step [28]. The poblem was how to delve a message to a pont whose locaton s not pecsely specfed; nstead t only needs to be located wthn a specfed polygon. Consde the fst phase of ou poblem;.e, the delvey of VR R v ) to a membe n BPv ). If v VR B v b j ), t wll be clea that v b j BPv ). Theefoe, the fst step would be the delvey of ths message to v b j. In ode to do ths, t s enough fo each ed node to know the neaest black node to tself so called ts black owne). Dung the ed update peod, each ed node encapsulates the specfcatons of VR R n a message and sends ths message to ts pevous black owne and asks the black owne to etun the message to t usng the black gaph. As the black stuctue suppots geedy outng, the packet wll each the new black owne of the ed node that s pobably the same as the cuent black owne. 2) BROADCAST OF THE MESSAGE AMONG THE MEMBERS OF THE BLACK PARENT SET When the message eaches the desed paent set, t should be boadcasted to the othe membes of BPv ). It should be noted that each black node only knows about ts own VR B and the locaton of ts black neghbos. Upon the ecepton of evey new VR R specfcaton by a black node, that black node popagates the eceved VR R specfcaton to evey black neghbo n the black gaph whose VR B s common sde o pont) has a nonempty ntesecton wth the VR R encapsulated n the message except the node that the message has just been eceved fom). Theoem 3: Usng the above method, the message encapsulatng VR R v ) wll be eceved by a black node vb j, ff v b j BPv ). Poof: The necessty condton s obvous efeng to the algothm descpton. The suffcency pat can be poved usng contadcton. Suppose that thee exsts a black node v b j BPv ) that does not eceve the message. Fgue 5 hghlghts the message n ed and the black node n yellow. Assume that S s the bggest connected subset of BPv ) that have not eceved the message and v b j S. Obvously, 1 S. Suppose that v b k s the black owne of the v. As each VR s convex, f we daw a lne fom a pont n VR B v b j ) 6 VOLUME X, NO. X, MARCH XXXX

7 FIGURE 5. Poof of Theoem 3: the message hghlghted n ed wll be eceved by the black node hghlghted n yellow because t satsfes the condton specfed n theoem 3. ) VR R v to a pont VRB v b k ) VR Rv ), all of ponts belongng to ths lne should belong to VR R v ), as well. If S has at least one sde n common wth VR B s of the nodes that have eceved the message, that causes a conflct wth the assumpton of S beng the bggest possble subset, because we can add anothe black node to S: the node fom the sde of v b j such that the lne passes though ts black VR B to S. It should be noted that that the locatons of nodes n a vtual envonment s a vtual concept. As such, we keep usng the olde ed-black gaph whle one of the two subgaph ed, n the case dscussed n ths secton) s beng updated. Afte the update, the new vtual locatons of the ed nodes wll be consdeed n the new ed-black gaph. Theefoe, at all tmes, even when the ed nodes wee usng the old locatons, the ed-black gaph s a Delaunay Tangulaton and thus povdes guaanteed delvey of the outng messages. B. TOPOLOGY REFINEMENT Pevously, we descbed ou appoach to delveng the specfcaton of VR R v )s to the membes of BPv ) n ode fo them to check the pobable nonempty ntesectons between VR R s. We have also mentoned that each black node s esponsble fo fndng VR R ntesectons happenng nsde ts VR B, and f t fnds an ovelap of two nodes between VR R s, t wll send a message to one of these pa equestng t to consde the othe ed node as a neghbohood canddate. Afte ecevng ths message, the ed node wll dscove the new set of ed nodes and send a Neghbohood Request message to all those futue neghbos. Upon the ecepton of a new neghbohood equest message at a ed node v, t daws the othogonal bsecto of lne passng though t and the canddate. The ed node wll then add the canddate to ts neghbo set, ff ths othogonal bsecto has an ntenal ntesecton wth VR R v );.e., thee s a subset of ths VR R v ) that s close to the canddate than to v. If ths condton s satsfed, the neghbohood equest s ganted and VR R v ) wll change to add the new VRv ) membe to the pevous subset. Theefoe, some neghbos mght be consdeed as edundant n the new VR R v ). The edundant neghbos ae ecognzed accodng to the followng ctea: the othogonal bsecto of edges between the man node and the edundant neghbos ae located fully outsde VR R v ) of the man node. Theefoe, deletng the edges between the man node and the edundant neghbos wll not affect VR R v ) of the man node. The good pont s that t s not necessay to nfom the othe end of an edge when the edge has been deleted. In ode to pove ths clam, we efe to the defnton of neghbohood n the Delaunay Tangulaton and also any othe undected gaph) that s a mutual concept. Theefoe, the neghbohood possblty s enough to be checked fom the vewpont of only one of the ends of an edge. Smlaly, a edundant edge s enough to be edundant fom the vewpont of both of ts ends. The poposed algothm needs to check the old membes of the neghbo set fo possble edundancy upon the jonng of each new neghbo. Theefoe, the complexty of ths algothm s of On 2 ). As the degee of the nodes n a stuctued gaph s usually small, the algothm executon won t consume a consdeable amount of pocessng powe. Howeve, f the numbe of neghbos nceases beyond a cetan theshold, t wll be pobably moe atonal to take advantage of Fotune s algothm [29], desgned fo fndng the Voono Dagam fo a set of ponts. The complexty of Fotune s algothm s of Onlogn)), whee n s the cadnal of set of ponts. In ths case, t s pefeed to mantan all of the canddates n a set and un the update algothm only once afte the ecepton of all of these messages. The algothm wll detemne the Voono Cell of the man node, and the neghbos set fo the man node wll then be defned as n the Delaunay Tangulaton case. C. TOPOLOGY GRAPH CONSTRUCTION BASED ON RED AND BLACK SUB-GRAPHS We pevously dscussed the soluton to the poblem of updatng the ed gaph usng the geedy suppotng black gaph as the efeence. In ths secton, we wll adapt ths method fo use n the update pocedue of the oveall topology gaph. Ths can be acheved by a slght modfcaton as descbed n the followng paagaph. Afte ecevng a new VR R v ), each black node vb j checks whethe the othogonal bsecto of the lne dawn fom v b j to v has an ntenal ntesecton wth the specfed VR R v ). If so, v b j makes tself canddate as a neghbo of that ed node n the oveall topology gaph. Theefoe, t sends neghbohood equest message to v n whch t ncludes ts own locaton. v b j e-computes ts oveall gaph n the pesence of v as a new neghbo. All the othe detals of the algothm ae smla to the pevously explaned algothm fo ed sub-gaph update. It should be noted that the enty of a new node n the netwok o depatue of an exstng node may tempoaly make a small pat of the gaph aound the enteng/extng nodes unable to suppot geedy outng. Howeve, these new VOLUME X, NO. X, MARCH XXXX 7

8 entes o exts wll be taken nto account n the next update cycle and fom thee t esumes suppotng geedy outng. D. FURTHER IMPROVEMENTS OF THE ALGORITHM We poposed a method to check the ntesectons of nodes n both ed and oveall gaphs, and adapted these gaphs connect new lnks and maybe delete some old lnks) to suppot geedy outng. As explaned befoe, a message contanng the specfcaton of VR of node v, VR Rv ) should be sent to the black owne of v whee t wll be sequentally boadcasted to the othe Black Paents of v. Ths algothm can be futhe enhanced as follows: afte the ecepton of VR R v ), the black node vb j checks the ovelap of VR R s n VR B v b j ). Howeve, befoe popagatng the message to the othe black ownes of v, vb j updates the eceved VR R v ) and boadcasts ths new VR ) R v followng the same scheme. As the updated VR R v ) s elayed, the black paents can wok on the latest vetex egon. The update n VR R v ) assumes that all new ed canddates found n v b j ) wll fnally become neghbos of v. E. FUNCTIONALITY PROOF ) Accodng to theoem 1, VR R v = VCv ) fo evey v n the esultng stuctue. ) Wth ths new method, n each step, ) the new VR R v wll be a subset of the pevous VRR v that stll contans VCv ) that s supposed to be the fnal VR ) R v. Theefoe, the new method checks fo any pobable nonempty ovelap of egons that s smla to fnal VR R s. At the fnal stage of the algothm, VR R s of the constucted ed gaph wll be sepaate sets. Theefoe, the ed gaph wll suppot geedy outng accodng to theoem 2. VI. CONSISTENCY ISSUES In ode to povde a cteon fo consstency measuement, we ntoduce a new tem named locaton age. At each abtay nstance of tme, locaton age of a node s defned as the tme passed snce the node has eceved the last locaton update message fom the othe nodes. Accodng to ths defnton, an updatng method s consdeed to be moe consstent ff t mantans a lowe value on locaton age. To evaluate ths paamete, let us fst consde a smple case whee all nodes send the nfomaton at tmes T fo ntege values of. In each of these ntevals [T -1), T ], at each moment, the age of the nfomaton s a paamete t between 0 and T whch gows as we move fowad n tme, untl we each the next update tme. Thus, the aveage s T 0 N 1) t/t dt = N 1)T /2, whee N s the total numbe of nodes. The aveage fo ou poposed ed-black algothm s calculated smlaly but takng nto account the fact that ed and black nodes send the updates wth a T/2 tme shft. Hee, the locaton age can be found as follows. If the total numbe of ed nodes s m, ths paamete fo the ed nodes would be: T 2 0 T + T 2 m 1) t +N m) 0 T + T 2 m 1) t +N m) t + T )) /T dt 2 t T 2 and fo the black nodes, t would be: T ) 2 m) +N m 1) t t + T 2 m) t T 2 )) /T dt = N 1) T /2 ) /T dt 1) ) ) +N m 1) t /T dt =N 1) T /2 Theefoe, the expected locaton age fo ths method wll stll be the same as that of the smple case. Hence, ou change n the method of sendng locaton update messages does not affect the consstency of the achtectue. VII. INFRASTRUCTURE FOR STATE MANAGEMENT In the pevous sectons we descbed ou poposed soluton and mathematcally poved that t woks;.e., t can guaantee to delve update message between dstbuted playes n a tmely fashon whle dynamcally updatng ts substate to suppot moblty and chun. Howeve, n addton to playes, thee ae othe objects n the scene that eque update messages. Fo example, a table that les n a oom mght change poston due to beng kcked o moved. But unlke a playe, a table s not a node n the netwok. In ths secton, we dscuss how we pefom state management fo such objects. A. CONNECTION TO NODES RESPONSIBLE FOR OBJECTS IN AREA OF EFFECT Non-playe objects n the game eque an owne to take cae of the state management whle the owne s not necessaly assgned n a fxed tem bass, because the owne could move fa fom the object due to moblty. In the appoaches poposed n [23], [25], and [30], the state of evey object s managed by the closest node and n case of a change n ths state, ths node s esponsble fo sendng the petnent update message to all nodes nteested n the state of ths object. Fo example, [31] pesents a two-fold appoach fo state management n P2P vtual envonments whee a stuctued ovelay s used fo object state management and a tangulaton ovelay s used fo avata state and goup membeshp management. [32] pesents a state management and pesstency achtectue fo P2P vtual envonments called Pthos, whch s a goup-based dstance-oented method that coves both esponsve stoage system and the secuty aspects of scalable dstbuted stoage. Whle the above appoaches lead to good esults, they ae not optmzed fo ou specfc case of Delaunay-based P2P ovelays. The only excepton s [33] whch also uses Delaunay-based P2P ovelays and foms a spannng tee to suppot event notfcaton to pees postoned 2) 8 VOLUME X, NO. X, MARCH XXXX

9 n the object s aea of effect. The spannng tee s constucted based on evese compass outng ognally poposed fo geometc netwoks. But t s had to pedct the behavo of a spannng tee n tems of hops and latency. On the othe hand, ou appoach takes advantage of geedy outng whch s a smple and fast outng pocess. In addton, we ntoduce new technques to educe hop count by usng a hybd stuctue fo ou state management. If the pees ae assumed to be the centes of Voono egons n the Voono dagam, the closest node to each object s the owne of the Voono egon n whch the object s located. Theefoe, the object can be managed by the node ownng the Voono egon n whch the object les. In ou poposed appoach, we buld a DT substate and use t fo state geedy outng, as show n the pevous sectons. Geedy outng ove a DT s guaanteed to delve a message to the closest node to the destnaton. Theefoe, f node v deses to change the state of object O whch les n the Voono cell of node u, node v wll send ts equest towad O usng geedy outng. Beng outed ove the DT substate, the equest wll eventually each the closest node to O,.e., node u. Theefoe, the fst pat of a state change s easly taken cae of. Node u wll then pocess ths equest consdeng appopate game logc and ths would pobably esult n a change n the state of object O. If so, node u has to announce ths state change to all nodes that ae nteested n the state of object O;.e., nodes that ae n O s aea of effect. Such announcements can be done usng geedy outng ove the DT substate. Aea of effect of a node s usually consdeed to be ccula [25]. Howeve, the moe ealstc vew s to consde t a geneal aea. Moeove, the effect of pobable obstacles n the vtual envonment should be taken nto account. Geneally, we can choose any abtaly shape fo the aea of effect of each node, but we can always assume that aea of effect s a connected aea. B. STATE MANAGER RENEWAL Anothe poblem n state management s the possble change n the state manage of an object due to the movement of nodes. As the nodes move, state manages wll also change. A new manage should become awae of the exstence of an object and get ts complete state. Ths poblem can also be solved usng the geedy outng algothm suppoted by ou poposed stuctue. At the end of each topology update phase, each node checks to see f all of ts pevously elevant objects ae stll n ts Voono egon o not. If an object has moved out of ts Voono egon, t s no longe esponsble fo state management of ths object and the complete state of ths object should be sent to ts new manage. Ths can be accomplshed by sendng a message contanng the state towads the locaton of the object usng geedy outng. Then the message would get to the closest node to the object;.e., the new state manage of ths object. VIII. DISTANT ROUTING In envonments such as MMOGs, messages ae usually desed to be sent to specfc coodnates n space. But snce FIGURE 6. Routng fom node s to node d. geedy geometc outng takes only geogaphc nfomaton nto account, t can be subject to sgnfcant pefomance degadaton n the case of long dstance connectvty n the game. To explan ths poblem, consde Fgue 6 and assume that node s wants to send a message to node d Fgue 6 epesents the locaton of nodes n the vtual envonment). Snce the vtual dstance between two nodes s not elated to the eal delay of tansfeng a message between them the physcal delay), t s moe easonable to use hop count as a measue of delvey delay. Usng geometc outng, the message needs to go though a oute lke the exhbted path n Fgue 6 whch conssts of 16 hops. On the othe hand, f s knew the IP-addess of d o a node n the vcnty of d, the message could be delveed by fewe hops. Ths s due to the fact that, f IP addess s known, the message can be tansmtted dectly nstead of gong though the ovelay, facng shote hop count. To acheve ths, we popose to add some shotcuts to dffeent egons n the MMOG though the use of supe pees o cloud seves assgned to each of the MMOG egons. In patcula, the patcpaton of supe pees as vtual nodes that povde a sot of shotcut to the dffeent egons of the MMOG can be effectve. Stene et al. pesent a Delaunay based ovelay by nsetng a small and bounded numbe of addtonal lnks called shotcuts [34]. A pee can use ts shotcuts among the nodes that ae physcally close. Lke ou poposed method, [34] can educe the numbe of hops n the ovelay by augmentng the ovelay wth addtonal lnks. Howeve, [34] assumes pees ae capable of nsetng exta lnks. Futhemoe, ths algothm eques nfomaton lke pees physcal locaton that could make ts mplementaton costly. Fo ths pupose, we dvde the envonment nto a numbe of small pattons such as S 1, S 2, etc, and assgn a node v to evey S as the epesentatve of nodes n S. If a message s to be sent fom node s to a dstant node d, the outng potocol can send the message to the epesentatve of the aea whee d les. Geedy outng wll then be used wthn the patton to gude the message to d, as dscussed next. A. PARTITIONING AND REPRESENTATIVE PLACEMENT As just mentoned, we am at potonng the vtual envonment nto a numbe of egons whee each egon has a epesentatve. Geneally speakng, aea epesentatves can be potentally chosen ethe fom the pees n each patton, efeed to as supe pees n P2P temnology, o a pool of MMOG nodes ncludng cloud seves. The actual selecton pocess of the supe pee o the cloud seve s beyond the VOLUME X, NO. X, MARCH XXXX 9

10 scope of ths atcle, and many technques can be used fo ths pupose [35], [36]. Fo smplcty, n ths pape we efe to both supe pees and cloud seves as supe pees. To balance the load on supe pees, t s desed to patton the envonment such that evey dvson contans almost the same numbe of nodes. Ths poblem s known to be ntactable fom theoetcal pont of vew [37]. On the othe hand, snce an exactly balanced patton s not so mpotant n ths case, ths can be accomplshed oughly by pattonng the envonment wth a small gd patten, and then assgnng a numbe of adjacent boxes to each supe pee such that the numbe of assgned nodes s appoxmately balanced. The gd patten can be changed ove tme; we just need t to be fxed fo the duaton n whch we ty to oute some messages va the supe-pees n ths gd. Snce node locatons ae just a vtual concept, we can do the computatons of a new gd wthout swtchng to t, and then, afte the computaton s done, at an ageed upon tme, all supe-pees swtch to the new vtual locatons of the epesentatves. Along the pocess, the IP-addess of supe pees and assgned pattons ae shaed among the supe pees. In ths dstant outng pocess, evey supe pee needs to be assgned a vtual poston as the epesentatve of ts aea. If the supe pee s not a cloud seve, ths vtual poston s dffeent fom the vtual locaton of the avata petnent to ths pee and theefoe, the vtual poston of evey supe pee would not be changed by the movement of ts elated avata. In fact, the avata can even be out the patton assgned to a supe pee. In the fully-dstbuted achtectue, due to the natual movement of nodes n the game, a specfc pont cannot be selected to be the best poston fo the epesentatve, as the nodes n the patton may move away fom ths pont and gathe aound anothe pont fa fom t. To mtgate ths poblem, evey supe pee can use m epesentatves nstead of a sngle one and unfomly dstbute these epesentatves n the assgned patton. Ths would ncease the numbe of connectons to the supe pee. Howeve, as the typcal node degee n Delaunay Tangulaton gaph s aound 5 to 7, supe pees ae expected to have enough pocessng powe to handle these connectons. Evey supe pee can adjust the elevant m numbe n ts aea consdeng communcaton capabltes and numbe of nodes n the aea. It s clea that nceasng m would decease the outng delay but ncease the load on the supe pee. The epesentatves ae andomly coloed, ed o black, and the exstence s announced n the next petnent stuctue update peod. B. MODIFIED ROUTING PROTOCOL In ode to explan the modfed outng potocol fo long dstance messages, we descbe the case whee node s wants to send a message to poston d. Node s efes to a geedy adus theshold n ode to decde about the next outng hop. Ths geedy adus can ethe be selected by evey node accodng to ts ange of nteest, o t can be selected and advetsed by the supe pees of each patton. The pope selecton of geedy adus depends on the applcaton. If the dstance fom s to d s less than the geedy adus of node s, the message wll be outed though smple geedy outng. Othewse, node s fowads the message to ts own aea epesentatve, whch n tun fowads the message to the supe pee belongng to the patton n whch d les. Upon ecepton of ths message, the supe pee uses nomal geedy outng to delve the message to d. It can be obseved that nceasng geedy adus causes the outng algothm to convege to smple geedy outng whle deceasng ths paamete wll foce the algothm to oute most packets though supe pees and would lead to a lage load on the supe pees. It should also be noted that the poposed method does not have any effect on the communcaton o computaton load of the nodes as they only need to know the own supe pee. The supe pees would then deal wth othe pattons and the epesentatves. In ode to avod the seach quey fo fndng the supe pee, anothe conventon s to assgn the supe pee of a egon to be the node that s closest to the cente of that egon. When a supe pee moves away fom the cente o leaves the vtual envonment, the supe pee task s handed ove to the next most elgble canddate. Moe detals egadng supe pee selecton can be found n [38]. IX. SIMULATIONS AND RESULTS In ths secton, we pesent the expemental esults obtaned fom MATLAB smulatons of the poposed achtectue. The smulatons ae pesented n two dffeent subsectons. The fst subsecton pesents the esult of the expements wth dynamcs of the Red-Black Achtectue, and the second subsecton demonstates the esults of the smulatons of the poposed dstant outng stuctue pesented n Secton VIII. FIGURE 7. Temnaton tme of esumpton algothm vesus numbe of nodes. A. RED-BLACK ARCHITECTURE In the pevous sectons, we have agued and poved that afte completon of the esumpton algothm, the esultng gaph wll suppot geedy outng. But, the queston s how fast does t come to completon? We have pefomed smulatons fo vaous numbes of nodes that spead andomly n a specfc vtual aea of [50 50] unts n the game. Round Tp Tmes RTT) ae selected to be unfomly dstbuted n [100, 200] ms nteval. We also assumed that each node can move at most one unt n an abtay decton n each smulaton step. We have plotted the aveage temnaton tme of ou 10 VOLUME X, NO. X, MARCH XXXX

11 algothm vesus the numbe of nodes n the game n Fgue 7. Temnaton tme s defned as the tme that the potocol takes untl all the messages n one update phase, say that of the ed nodes, have eached the espectve destnatons. That s, afte ths tme, thee s no moe communcaton happenng fo the esumpton algothm of ths phase. Natually, ths tme dectly depends on the numbe of hops a message needs to take befoe eachng ts destnaton. As can be seen n the fgue, the numbe of nodes n the aea has a slght effect on the temnaton tme of the poposed method. Ths could be explaned as follows. FIGURE 9. The numbe of tansfeed messages vesus numbe of nodes. sent n the esumpton algothm has an appoxmately lnea elaton wth the numbe of nodes;.e., the aveage numbe of messages tansmtted by evey node s almost constant. Ths ssue s smla to the pevous one seen n algothm temnaton tme. Hence, t can be concluded that the poposed method s fully scalable, and nceasng the numbe of nodes does not affect ts complexty. Theefoe, t can be an altenatve to the centalzed-only clent-seve achtectue. FIGURE 8. Black nodes that eceve VR R v ) the specfcaton message, the yellow ponts ae pevous poston of ed nodes, and ed and black ponts epesent ed and black nodes, espectvely. As pevously mentoned n Secton V, evey ed node sends specfcatons of VR R to ts black owne. Ths message wll be then boadcasted to othe black paents of ths node. In the mpovement of the poposed method, we mentoned that evey black node wll update the eceved VR R specfcaton befoe popagatng the message to othe neghbong black paents. Theefoe, when thee exsts a nonempty ovelap between ths VR R and some othe VR R -s, we can oughly say that the VR R wll be cut fom ths sde and thus, the message wll not be sent futhe n ths decton. Assume that we ae consdeng the black nodes that eceve VR R v ) specfcaton Fgue 8). Theefoe, t can be sad that ths VR R specfcaton wll be delveed only to those black nodes that ae black ) paents of v coespondng to the fnal veson of VR R v = VCv ). If ed and black nodes ae unfomly dstbuted n the Eucldean plane, VCv ) wll ovelap wth aound 4 to 8 black VR B s and thus the VR R v ) specfcaton wll be delveed to nealy 4 to 8 black nodes. It can be agued that the aveage degee of vetces n a DT gaph s usually n the ange of 3 to 7, and wth the ncease n the numbe of nodes n a specfc aea, ths paamete does not change. Howeve, wth ths change, the aveage dstance of neghbo nodes;.e., the length of edges wll decease. But, the length of edges n gaphs of MMOGs s just a vtual concept and s not elevant to the eal tme of message tansfe. We have also demonstated the oveall numbe of messages sent fo the esumpton algothm vesus the numbe of nodes, n Fgue 9. As can be seen, the numbe of messages B. DISTANT ROUTING In ths subsecton, we demonstate the effcency of ou poposed dstant outng stuctue descbed n Secton VIII. We used vtual nodes pees) unfomly spead ove a [1000unts) 1000unts)] aea n the vtual space and smulated the poposed outng fo vaous combnatons of numbe of supe pees, numbe of epesentatves of each supe pee and geedy adus thesholds. Fo the sake of smplcty, we assumed that all supe pees use the same numbe of epesentatves and all vtual nodes explot an equal geedy adus theshold. Howeve, t should be noted that all of the pevous assumptons ncludng the unfom dstbuton of nodes do not affect the functonalty of the poposed algothm. In the smulaton, the nodes wll constantly change the locatons by movng one step n a andomly selected decton dung each smulaton step whee the decton s deved accodng to a unfom dstbuton. As agued befoe, we use aveage hop count as a easonable ctea fo assessng the pefomance. Theefoe, we gnoe the conveson of the hop count to eal delvey delay tme and hence we do not consde the possble effect of taffc and congeston n the outng potocol delay. Vtual nodes geneate packets wth an abtay ate towad destnatons selected andomly n the aea wth a specfc dstbuton ntended to esemble the eal stuaton n vtual envonments. As mentoned befoe, the nfomaton exchange and communcatons of each pee s usually elated to the peceptons of ts avata. Theefoe, the nodes have moe tendency towads connectng wth close neghbos and to send messages to neae ponts. To smulate ths condton, nodes select destnatons of the geneated packets wth a dstance dstbuted accodng to gamma dstbuton, whch s a geneal fom of pobablty dstbuton that has the flexblty to pesent a wde ange of VOLUME X, NO. X, MARCH XXXX 11

12 FIGURE 10. The aveage message delvey hop count vesus geedy adus theshold. behavos of pobablty dstbutons by changng ts paametes. The pobablty densty functon of ths dstbuton s of the fom: k 1 e x/θ f X x; k, θ) = x θ k Ɣk) fo x 0 and k, θ > 0 3) whee k and θ ae two paametes named shape paamete and scale paamete, espectvely. We choose k = 3 and θ = 50 as a pagmatc choce epesentng the condtons of the game, snce that choce means a gven node wll send messages to close nodes wth hghe pobablty and to fathe nodes wth lowe pobablty, whch makes sense n a gamng envonment. Theefoe, n ou smulaton, nodes choose the destnaton of the packets wth a dstance selected accodng to gamma dstbuton wth k = 3 and θ = 50 and a unfomly dstbuted angle. Usng the above pobablty functon, the aveage hop count of message delvey vesus geedy adus theshold was smulated fo vaous numbes of supe pees, as shown n Fgue 10 and assumng that evey supe pee uses only one epesentatve. Consdeng ou poposed outng algothm, t s clea that the delvey delay of smple geedy outng can be obtaned by settng the geedy adus theshold to nfnty. In ths smulaton, a adus of 1000 s equvalent to nfnty gven the total sze of the expement. Howeve, as can be seen fom Fgue 10, wth a adus as small as 500, the hop count value conveges aleady. Theefoe, the aveage message delvey hop count fo smple geedy outng n ths expement s about hops. As can be seen n Fgue 10, usng vey small numbes of supe pees 9 supe pees compaed to nodes) would not decease the message delvey delay. Ths s due to the fact that wth such a small numbe of supe pees, the sze of evey patton would be vey lage and hence, the aveage dstance fom even the closest supe pee to destnaton would be lage than the aveage dstance fom sende to destnaton. Howeve, ths stuaton mpoves wth nceasng the numbe of supe pees. Usng the poposed stuctue wth 25 o moe supe pees would lead to a fa bette pefomance compaed to smple geedy outng n tems of delay metc. Anothe mpotant pont n Fgue 10 s the elaton between message delvey delay and geedy adus theshold. It can be seen that statng fom a small close to zeo) geedy adus theshold and nceasng ths adus, the aveage delvey delay deceases untl eachng a maxmal pont. Futhe ncease n the adus fom that maxmal pont on) slghtly nceases the aveage delvey delay towads convegng to the aveage message delvey delay of smple geedy outng potocol. Ths behavo can be easly explaned. In the lowe pat of the hozontal axs small ad), we should ncease the geedy adus theshold to decease the message delvey delay. Ths s because of the lage dstance fom the sende to the destnaton compaed to the dstance fom the closest supe pee to the destnaton, and t means that t s bette to use the oute though supe pees fo wde ange of destnatons. On the othe hand, sendng moe messages usng smple geedy outng becomes moe benefcal n the hghe pat of the hozontal axs whee the local tendency of nodes domnates the pesented effect of the fst pat. FIGURE 11. The aveage message delvey hop count vesus the numbe of epesentatves. We have also smulated the aveage message delvey delay vesus the numbe of epesentatves, used by a gven supe pee, fo dffeent numbe of supe pees, as shown n Fgue 11. In the fgue, the begnnng of each cuve exhbts the case wth no epesentatve, whch s equvalent to the smple geedy outng potocol. Theefoe ths pont s ndependent of the numbe of supe pees and s common among all cuves. Fo each smulaton each pont on the cuves), the geedy adus theshold s set to the optmal value obtaned by expemental esults,.e., the value whch leads to the lowest possble message delvey delay. As can be seen fom the fgue, nceasng the numbe of the epesentatves of supe pees wll decease the aveage message delvey delay n an exponental manne. The convegence pont of all cuves s obseved to be aound 3 hop counts. Ths s due to the fact that wth a vey lage numbe of epesentatves, t would be atonal to always use the oute though the supe pees and ths oute conssts of 3 hops, one fom the sende to the supe pee of the sende s patton, one fom thee to the supe pee of the destnaton s patton, and one fom ths supe pee to ts closest neghbo to the destnaton note that the mes- 12 VOLUME X, NO. X, MARCH XXXX

13 sage should be delveed to the closest odnay node to the destnaton). The fgue also shows the effcency of the poposed outng stuctue compaed to the smple geedy outng potocol. As can be seen, even n the pesence of a small numbe of supe pees;.e., 25 supe pees whch s about 0.2% of pees), and a modeate numbe of epesentatves fo each supe pee 9 epesentatves), the obtaned aveage message delvey delay s about 60% less than the smple geedy outng potocol. Ths can be computed as follows: the fst data pont s wth 25 supe pees and 0 epesentatves, whch means wth just geedy outng, and thee the hop count s oughly 13 ead fom the leftmost pont on the lowest cuve). The second data pont s wth 25 supe pees and 9 epesentatves pe supe pee, and thee the hop count s oughly 5. The educton fom 13 to 5 s 13-5)/13 = % X. CONCLUSIONS AND FUTURE WORK Handlng uses moblty and chun s one of the most mpotant ssues n the desgn of eslent dstbuted o hybd MMOGs. In ths atcle, we poposed a soluton fo that whch benefts fom geedy outng s smple and fast outng pocess, and we ntoduced a new dstbuted algothm fo mantanng a geedy suppot gaph among MMOG nodes, whch pattons the topology nto two dffeent pats whee each pat s altenatvely updated n one phase usng the othe pat as a efeence. We also ntoduced new technques to educe hop count by usng a hybd stuctue and also poposed an nfastuctue fo state management. Smulaton esults demonstated the effcent pefomance of ou solutons. Fo futue wok, the poposed outng scheme can be futhe optmzed to take load balancng and path cost nto account. Moeove, the optmal selecton of supe pees and pattonng the vtual wold fo best outng and load balancng n the cloud ae ssues that waant futhe studes. Fnally, evaluatng ou desgn usng eal wold mplementatons o actual netwok taces s an mpotant futue wok that can help us undestand both the postve aspects and the shotcomngs of the poposal. That evaluaton can take nto account many pactcal ssues that ae not consdeed n a smulaton envonment, such as netwok congeston and actual lnk delays. REFERENCES [1] S. Shmohammad and M. Claypool, Massvely multplaye onlne gamng systems and applcatons, Multmeda Tools Appl., vol. 45, no. 1, pp. 1 5, [2] M. Claypool, The effect of latency on use pefomance n eal-tme stategy games, Int. J. Comput. Telecommun. Netw., vol. 49, no. 1, pp , Sep [3] D. T. Ahmed, S. Shmohammad, and J. C. de Olvea, A hybd P2P communcatons achtectue fo zonal MMOGs, Multmeda Tools Appl., vol. 45, nos. 1 3, pp , Oct [4] M. Ghaffa, B. Ha, and S. Shmohammad, On the necessty of usng Delaunay tangulaton substate n geedy outng based netwoks, IEEE Commun. Lett., vol. 14, no. 3, pp , Ma [5] B. Ha, M. R. Pakavan, S. Shmohammad, and M. H. Alav, Usng geometcal outng fo ovelay netwokng n MMOGs, Multmeda Tools Appl., vol. 45, no. 1 3, pp , [6] M. Ghaffa, B. Ha, and S. Shmohammad, A Delaunay tangulaton achtectue suppotng chun and use moblty n MMVEs, n Poc. ACM Wokshop Netw. Ope. Syst. Suppot Dgt. Audo Vdeo NOSSDAV), Wllamsbug, VA, USA, Jun. 2009, pp [7] E. Buyukkaya, M. Abdallah, and G. Smon, A suvey of pee-to-pee ovelay appoaches fo netwoked vtual envonments, n Pee-to-Pee Netwokng and Applcatons. New Yok, NY, USA: Spnge-Velag, Sep [8] A. Yahyav and B. Kemme, Pee-to-pee achtectues fo massvely multplaye onlne games: A suvey, ACM Comput. Suv., vol. 46, no. 1, p. 9, Oct [9] E. Léty, T. Tulett, and F. Baccell, SCORE: A scalable communcaton potocol fo lage-scale vtual envonments, IEEE/ACM Tans. Netw., vol. 12, no. 2, pp , Ap [10] M.-J. Tsa, H.-Y. Yang, B.-H. Lu, and W.-Q. Huang, Vtual-coodnatebased delvey-guaanteed outng potocol n weless senso netwoks, IEEE/ACM Tans. Netw., vol. 17, no. 4, pp , Aug [11] E. Kanaks, H. Sngh, and J. Uuta, Compass outng on geometc netwoks, n Poc. 11th Can. Conf. Comput. Geomety, Aug. 1999, pp [12] F. Kuhn, R. Wattenhofe, and A. Zollnge, An algothmc appoach to geogaphc outng n ad hoc and senso netwoks, IEEE/ACM Tans. Netw., vol. 16, no. 1, pp , Feb [13] P. Bose and P. Mon, Onlne outng n tangulatons, n Poc. Int. Symp. Algothms Comput., Chenna, Inda, Dec. 1999, pp [14] J. Lebehe, M. Nahas, and W. S, Applcaton-laye multcastng wth delaunay tangulaton ovelays, IEEE J. Sel. Aeas Commun., vol. 20, no. 8, pp , Oct [15] E. Buyukkaya and M. Abdallah, A flexble connectvty achtectue fo avata management n P2P vtual envonments, n Poc. 10th ACM/IEEE Int. Wokshop Netw. Syst. Suppot Games, Ottawa, ON, Canada, Oct. 2011, pp [16] E. Buyukkaya and M. Abdallah, Routng ove elaxed tangulaton stuctues fo P2P-based vtual envonments, n Poc. Int. Conf. Adv. Infom. Netw. Appl. Wokshops, Bacelona, Span, Ma. 2013, pp [17] M. Stene and E. Besack, A fully dstbuted pee to pee stuctue based on 3D Delaunay tangulaton, n Poc. Rencontes Fancophones Aspects Algothmques Télécommun., May 2005, Pesqu île de Gens, Fance, pp [18] D.-Y. Lee and S. S. Lam, Potocol desgn fo dynamc Delaunay tangulaton, n Poc. Int. Conf. Dstb. Comput. Syst., Toonto, ON, Jun. 2007, Canada, p. 26. [19] D.-Y. Lee and S. S. Lam, Effcent and accuate potocols fo dstbuted Delaunay tangulaton unde chun, n Poc. IEEE Int. Conf. Netw. Potocols, Olando, FL, USA, Oct. 2008, pp [20] S.-Y. Hu, J.-F. Chen, and T.-H. Chen, VON: A scalable pee-to-pee netwok fo vtual envonments, IEEE Netw., vol. 20, no. 4, pp , Jul./Aug [21] N. E. Baughman, M. Lbeatoe, and B. N. Levne, Cheat-poof playout fo centalzed and pee-to-pee gamng, IEEE/ACM Tans. Netw., vol. 15, no. 1, pp. 1 13, Feb [22] S. Schuste, Cheatng peventon n pee-to-pee-based massvely multuse vtual envonments, Ph.D. dssetaton, Faculty Eng. Sc., Comput. Sc. Appl. Cognt. Sc., Unv. Dusbug-Essen, Essen, Gemany, [23] A. Tumbde and S. Venugopalan, A voono pattonng appoach to suppot massvely multplaye onlne games, Dept. Comput. Sc., Unv. Wsconsn, Madson, WI, USA, Tech. Rep. CS 740, [24] C.-H. Chen, S.-Y. Hu, and J.-R. Jang, Delaunay state management fo lage-scale netwoked vtual envonments, n Poc. IEEE Int. Conf. Paallel Dstb. Syst., Melboune, Austala, Dec. 2008, pp [25] S.-Y. Hu, S.-C. Chang, and J.-R. Jang, Voono state management fo pee-to-pee massvely multplaye onlne games, n Poc. IEEE Consum. Commun. Netw. Conf., Las Vegas, NV, USA, Jan. 2008, pp [26] A. P. Yu and S. T. Vuong, MOPAR: A moble pee-to-pee ovelay achtectue fo nteest management of massvely multplaye onlne games, n Poc. ACM Wokshop Netw. Ope. Syst. Suppot Dg. Audo Vdeo NOSSDAV), Washngton, DC, USA, Jun. 2005, pp [27] S. Ratt, B. Ha, and S. Shmohammad, A suvey of fst-peson shoote gamng taffc on the ntenet, IEEE Intenet Comput., vol. 14, no. 5, pp , Sep./Oct [28] I. Stojmenovc, A. P. Ruhl, and D. K. Lobyal, Voono dagam and convex hull based geocastng and outng n weless netwoks, n Poc. 8th IEEE Int. Symp. Comput. Commun., Jun. 2003, pp VOLUME X, NO. X, MARCH XXXX 13

14 Ghaffa et al.: Dynamc Netwokng Substate fo Dstbuted MMOGs [29] S. Fotune, A sweeplne algothm fo voono dagams, n Poc. 2nd Annu. Symp. Comput. Geomety, Yoktown Heghts, NY, USA, Jun. 1986, pp [30] E. Buyukkaya and M. Abdallah, Data management n voono-based P2P gamng, n Poc. 5th IEEE Int. Wokshop Dg. Entetanment, Netw. Vtual Envon., Ceatve Technol., Las Vegas, NV, USA, Jan. 2008, pp [31] E. Buyukkaya and M. Abdallah, A twofold appoach to object and avata data management n P2P-based vtual envonments, n Poc. IEEE Int. Conf. Multmeda Expo Wokshops, San Jose, CA, USA, Jul. 2013, pp [32] J. Glmoe, State management and pesstency achtectue fo pee-topee massvely mult-use vtual envonments, Ph.D. dssetaton, Dept. Elect. Electon. Eng., Stellenbosch Unv., Stellenbosch, South Afca, [33] L. Rcc, L. Genoval, E. Caln, and M. Coppola, AOI-cast n dstbuted vtual envonments: An appoach based on delay toleant evese compass outng, n Concuency and Computaton: Pactce and Expeence, New Yok, NY, USA: Wley, Dec [34] M. Stene and E. W. Besack, Shotcuts n a vtual wold, n Poc. ACM CoNEXT, Lsbon, Potugal, Dec. 2006, p. 17. [35] S.-H. Mn, J. Hollday, and D.-S. Cho, Optmal supe-pee selecton fo lage-scale P2P system, n Poc. Int. Conf. Hybd Infom. Technol., Cheju Island, Koea, Nov. 2006, pp [36] V. Lo, D. Zhou, Y. Lu, C. GautheDckey, and J. L, Scalable supenode selecton n pee-to-pee ovelay netwoks, n Poc. Int. Wokshop Hot Topcs Pee-to-Pee Syst., Jul. 2005, pp [37] M. H. Alsuwayel, A paallel algothm fo pattonng a pont set to mnmze the maxmum of dametes, J. Paallel Dstb. Comput., vol. 61, no. 5, pp , [38] B. Ha, S. Shmohammad, and M. Pakavan, A dstbuted nteest management scheme fo massvely mult-use vtual envonments, n Poc. IEEE Conf. Vtual Envon., Human-Comput. Inte., Meas. Syst., Istanbul, Tukey, Jul. 2008, pp MOHSEN GHAFFARI s cuently pusung the Ph.D. degee wth the Depatment of Electcal Engneeng and Compute Scence, Massachusetts Insttute of Technology MIT), Cambdge, MA, USA. He eceved the M.S. degee fom the Depatment of Electcal Engneeng and Compute Scence, Massachusetts Insttute of Technology, n 2013, and the dual B.Sc. degee n electcal engneeng communcatons) and compute scence fom the Shaf Unvesty of Technology, Tehan, Ian, n Hs cuent eseach nteest evolves aound theoetcal poblems n netwokng and dstbuted computng. Dung hs undegaduate pogam, he was nvolved n the eseach subject of ths pape wth the Dstbuted and Collaboatve Vtual Envonment Reseach DISCOVER) Laboatoy, Unvesty of Ottawa, Ottawa, ON, 1276 Canada. He was a ecpent of the 2014 Smons Awad fo Gaduate Students n Theoetcal Compute Scence and the pestgous MIT Pesdental Fellowshp. 14 BEHNOOSH HARIRI eceved the Ph.D. degee n electcal engneeng fom the Shaf Unvesty of Technology, Tehan, Ian, n 2009, and was a MITACS Elevate Post-Doctoal Fellow wth the DISCOVER Laboatoy, Unvesty of Ottawa, Ottawa, ON, Canada, fom 2009 to He eseach was focused on gamng systems and netwoks, n patcula, massvely multuse vtual envonments, and vson-based multmeda systems. She s cuently wth Google Inc., New Yok, NY, USA. SHERVIN SHIRMOHAMMADI M 04 SM 04) eceved the Ph.D. degee n electcal engneeng fom the Unvesty of Ottawa, Ottawa, ON, Canada, whee he s cuently a Full Pofesso wth the School of Electcal Engneeng and Compute Scence. He s the Co-Decto of the DISCOVER Laboatoy and the Multmeda Communcatons Reseach Laboatoy, Unvesty of Ottawa, conductng eseach n multmeda systems and netwokng, n patcula, gamng systems and vtual envonments, vdeo systems, and multmeda-asssted bomedcal engneeng. The esults of hs eseach have led to moe than 250 publcatons, ove 20 patents and technology tansfes to the pvate secto, and a numbe of awads and pzes. He s an Assocate Edto-n-Chef of the IEEE INstumentaton And MEasuement Magazne, a Seno Assocate Edto of ACM Tansactons on Multmeda Computng, Communcatons, and Applcatons, an Assocate Edto of the IEEE TRansactons On INstumentaton And MEasuement, and was an Assocate Edto of Jounal of Multmeda Tools and Applcatons Spnge) fom 2004 to 2012, and chas o seves on the pogam commttee of a numbe of confeences n multmeda, vtual envonments, and games. He s a Unvesty of Ottawa Gold Medalst, a lcensed Pofessonal Engnee n Ontao, and a Pofessonal Membe of the Assocaton fo Computng Machney. DEWAN TANVIR AHMED eceved the Ph.D. degee n compute scence fom the Unvesty of Ottawa, Ottawa, ON, Canada, n 2009, whee he was a Post-Doctoal Fellow wth the DISCOVER Laboatoy fom 2009 to He s cuently an Assstant Pofesso wth the Depatment of Compute Scence, Unvesty of Noth Caolna at Chalotte, Chalotte, NC, USA, whee he s conductng eseach n dstbuted systems and algothms, smulaton and modelng, multmeda communcaton, and compute netwoks, wth applcatons n eal-tme systems, suvellance systems, and seous games. He was a ecpent of the IEEE ICME Qualty Revewe Awad n 2011, the Industal Reseach and Development Fellowshp fom the Natual Scences and Engneeng Reseach Councl of Canada n 2010, the Best Student Reseache Awad fom the Ottawa Cente fo Reseach and Innovaton n 2009, the Best Poste Awad n Compute Scence fom Unvesty of Ottawa s Faculty of Engneeng n 2008, and the Best Pape Awad fom the IEEE Collaboatve Pee-to-Pee Infomaton Systems Wokshop n VOLUME X, NO. X, MARCH XXXX