A Game Theoretic Approach for Distributed Resource Allocation and Orchestration of Softwarized Networks

Transcription

1 A Game Theoretc Approach for Dstrbuted Resource Allocaton and Orchestraton of Softwarzed Networks Salvatore D Oro, Laura Gallucco, Member, IEEE, Sergo Palazzo, Senor Member, IEEE, Govann Schembra Abstract Softwarzaton of networks allows smplfyng deployment, confguraton and management of network functons. The drvng force towards ths evoluton s represented by Software Defned Networkng (SDN) that allows more flexble and dynamc network resource allocaton and management. Effcent resource allocaton and orchestraton are two prmary targets of ths softwarzaton process; however, centralzed methodologes result complex, and exhbt scalablty ssues. So, dstrbuted solutons are to be preferred but, n order to be effectve, should quckly converge towards equlbrum solutons. In ths paper, we focus on makng dstrbuted resource allocaton and orchestraton a vable approach, and prove convergence of the relevant mechansms. Specfcally, we explot game theory to model nteractons between users requestng network functons and servers provdng these functons. Accordngly, a two-stage Stackelberg game s presented where servers act as leaders of the game and users as followers. Servers have conflctng nterests and try to maxmze ther utlty; users, on the other hand, use a replcator behavor and try to mtate other users decsons to mprove ther beneft. The framework proves the exstence and unqueness of an equlbrum, and a learnng mechansm to converge to such equlbrum s proposed. Numercal results show the effectveness of the approach. Index Terms Game Theory, Softwarzed Networks, Orchestraton, Resource Management. I. INTRODUCTION The prolferaton of new servces and applcatons n the Internet wth dfferent requrements n terms of avalablty, servce qualty and reslence, s makng management of network nfrastructures a key challenge. In ths evolvng scenaro, Telco Operators (TOs) show ncreasng nterest n softwarzng ther networks, so makng deployment, confguraton, management and updatng of network functons faster and easer, and thus achevng numerous advantages n terms of both Captal Expendture (CAPEX) and Operatonal Expendture (OPEX). Two relevant key enablers of ths evoluton are Software- Defned Networkng (SDN) [ 3] and Network Functons Vrtualzaton (NFV) [4]. SDN allows a flexble management of All authors are wth the Dpartmento d Ingegnera Elettrca, Elettronca ed Informatca, Unversty of Catana, Italy; E-Mal: {name.surname}@dee.unct.t Ths work has been partally supported by the INPUT (In-Network Programmablty for next-generaton personal cloud servce support) project funded by the European Commsson under the Horzon Programme (Call H-ICT-4-, Grant no ). Ths paper has been accepted for publcaton on IEEE JSAC Specal Issue on Game Theory for Networks. Ths s a preprnt verson of the accepted paper. The fnal paper wll be avalable n short. Copyrght (c) 3 IEEE. Personal use of ths materal s permtted. However, permsson to use ths materal for any other purposes must be obtaned from the IEEE by sendng a request to pubs-permssons@eee.org. the network resources thanks to ts pecularty of separatng the network control from the forwardng plane. NFV, on the other hand, brngs vrtualzaton concepts from cloud computng to the network n order to let software-based network functons, also called vrtualzed network functons (VNFs), run on commodty hardware nfrastructures. The ntroducton of the jont SDN/NFV paradgm s seen by Telco Operators (TOs) as the way to move to more flexble networks where servces can be nstantly montored, controlled, blled, and managed on the fly, rather than requrng a set of complex, manual changes [5]. However, as compared to purpose-bult networkng hardware or mddle boxes devces, deterrents to ths approach are the achevable performance and the scalablty. Key elements for the desgn of these systems are resource allocaton, and network functon orchestraton. Although smlar desgn problems have been studed n cloud computng scenaros [6 9], there are mportant dfferences stemmng from the fact that servers n data centers are connected to each other through hgh-capacty and hgh-speed networks, so makng the specfcs of the underlyng network less mportant. On the contrary, n network functon deployment, network constrants (e.g. bandwdth and latency) are of crucal mportance. The choce of where runnng network functons has to be made by accountng not only for the ncreased load n the nodes hostng the functons, but also for the latency experenced to reach these nodes, whch can be dfferent for each flow [, ]. The frst step towards management and functon allocaton n an NFV scenaro was made n [], where the VNF-P algorthm was ntroduced to handle traffc load varatons by dynamcally nstantatng VNFs. In the same context [3] and [4] dscuss placement polces for specfc VNFs. Instead, the work [5] consders a heterogeneous scenaro wth VNFs characterzed by dfferent scalablty, relablty, and avalablty requrements, and proposes an extenson of the Openstack orchestrator to translate the ndvdual deployment requrements nto a placement of the VNFs n the cloud nfrastructure. Two works that are very close to ths paper n some aspects are [6] and [7]. In [6] VNF placement s made n a way that mnmzes the overall network cost, expressed n terms of the dstance between users and the locatons where servces are provded, and the cost of servce setup. Instead, the work n [7] consders the mult-commodty faclty locaton by consderng the exstence of more than one VNF nstance n the same network. The problem of placement of network functons mplemented as mddle boxes s consdered n [8], wth the target of

2 optmzng network operatonal costs and utlzaton, wthout volatng servce level agreements. Ths VNF Orchestraton Problem (VNF-OP) s addressed and formulated as an Integer Lnear Programmng (ILP) problem that s then solved through heurstcs. Another aspect that has to be consdered s system scalablty w.r.t. the number of functons and the customers. In fact, the complex tasks of management, orchestraton and resource allocaton are n charge of only one entty, the Orchestrator, whch therefore requres sophstcated algorthms whch generally results n NP-hard problems [6, 9, ]. Ths makes the deployment of the softwarzed paradgm unfeasble f TOs am at desgnng and managng ther networks whle optmzng costs and performance. So centralzed solutons (see Case A n Fg. (a)) or vrtually centralzed solutons (see Case B n Fg. (b)) could result unfeasble, and dstrbuted approaches to resource allocaton and orchestraton have to be pursued. Wth all ths n mnd, the reference scenaro addressed n ths paper addresses a sngle TO network doman where some customers, n the followng referred to as Servers, gve ther avalablty to support the TO n provdng network servces to the Users, not only sharng ther hardware facltes where runnng network functons, but also strongly allevatng management and orchestraton burden n the decson tasks of both placng functons and assgnng them to each actve user flow. We consder that the Server role s played by customers of the TO network and network functons are located n the CN premses (see Case C n Fg. (c)). So, both resource provsonng and management are dstrbuted wthn the consdered TO network doman. Wth respect to ths, let us note that, at the best of our knowledge, there s no work so far where the man tasks performed by the orchestrator, and thus related to management ssues, are dstrbuted among customers of the network,.e., the VNF Servers. A relevant ssue wth ths dstrbuted approach s related to system convergence towards equlbrum. In fact, the possblty to converge towards an equlbrum as well as the rapdness of ths convergence should be nvestgated to prove the feasblty of the approach. Accordngly, game theory s the natural way to model and characterze the system. In partcular, a new market s assocated to the proposed system, where the man actors are: ) the Servers, that are the sellers of the network functons; ) the Users, that play the role of buyers; 3) the TO, that coordnates the whole system. In such a context, Servers autonomously decde the prce and the bandwdth to be requested to the TO network n order to provde the network servces. Users, on the other hand, accordng to the prce specfed by each Server, and the correspondng expected performance n terms of both experenced latency and provded bandwdth, choose one Server for each VNF. In ths way the task of assocatng each flow to a Server s not decded by the Orchestrator, but n an autonomous and dstrbuted way, as a consequence of the nteracton between Users and Servers. To model and support nteractons between Servers and Users, we explot herarchcal and evolutonary game theoretc tools. Specfcally, snce Servers naturally act and make decsons by antcpatng the Users, we defne a two-stage Stackelberg game where Servers act as the leaders of the game, and Users as the followers. Servers have conflctng nterests among themselves, as ther objectve s to ndvdually and selfshly maxmze a utlty functon. Also, as commonly assumed n mult-player markets, Servers are expected not to cooperate wth each other, and do not exchange any nformaton wth other compettors. Therefore, ther nteractons are modeled by non-cooperatve game theory. Instead, Users are nfluenced by socal and mtaton behavor,.e., they observe other Users decsons and mtate those decsons f ths s expected to mprove ther beneft. Thus, ther nteractons are modeled by usng the replcator dynamcs from Evolutonary Game Theory (EGT) []. In more detal, we derve a closed-form soluton for the equlbrum condton of the replcator dynamcs whch s then used to solve the Stackelberg game. Accordngly, we show that the consdered game admts a Stackelberg Equlbrum (SE), and we prove that the SE s unque. We also propose a renforcng learnng procedure that provably converges to the unque SE, and llustrate an algorthmc mplementaton. We show that the learnng procedure can be mplemented n a prvacypreservng and dstrbuted fashon. Fnally, we present an extensve numercal result analyss to hghlght the dependency of the dynamc nteractons among players on the man system parameters, and evaluate the proposed market model, n the vew of provdng some nsghts n settng system parameters to maxmze revenues. A part of the numercal analyss s also amed to show that the proposed learnng procedure s scalable w.r.t. the number of Servers, and quckly converges to the SE. At our best knowledge, ths s the frst work where a gametheoretc approach s used n a softwarzed network scenaro to support nteractons between Servers and Users. Moreover, explotng the game-theoretc approach allows us to demonstrate that adoptng a dstrbuted framework, nstead of a tradtonal centralzed approach, s benefcal for all the nvolved stakeholders. On the one hand, the smplfed orchestraton and the possblty to offload VNFs on thrd-partes Servers s benefcal to the TO. Servers have economc beneft partcpatng to the VNF Market as sellers. Fnally, Users can choose the Servers that most ft ther needs. The remanng of ths paper s organzed as follows. In Secton II, the consdered network scenaro s descrbed. In Secton III, the game-theoretc framework whch models the consdered resource allocaton and orchestraton problem s proposed and studed. A numercal analyss of the proposed game-theoretc framework s presented n Secton IV. Fnally, n Secton V conclusons are drawn. II. SYSTEM MODEL The consdered reference scenaro s sketched n Fg.. It conssts of a network doman of a Telco Operator (TO) that provdes customers of ths doman wth network servces accordng to the NFV paradgm. The man roles n the system are played by the Orchestrator, the VNF Servers, and the Users. Users are the customers that generate flows and request VNFs for each of ther flows. They are located n Customer Networks (CNs) where a Customer Premse Equpment (CPE)

3 3 TABLE I NOTATION (a) (b) (c) Fg.. a) Case A: Centralzed market scenaro where all network functons are executed on a sngle server owned by the TO; b) Case B: Centralzed market scenaro where network functons are dstrbuted among several servers owned by the TO; c) Case C: Proposed dstrbuted market scenaro where network functons are executed on several thrd-partes servers. Varable Descrpton S, U Sets of VNF Servers and User Groups n p Fracton of users n U p connected to VNF Server p User Groups p U b p Bandwdth requested by S to serve users from p M Number of VNF Servers p (F) p Prce mposed by S to p U to access VNF f p F N p Populaton sze of User Group p p (B) Bandwdth-unt prce mposed by the TO to VNF Server F Set of VNFs U (U) p Utlty functon of a user n p connected to VNF Server f p F VNF functon requested by User Group p γ m Step-sze of the learnng procedure c p Cost of S to serve a sngle user n p U α, α, α 3 Weghng parameters β, β Fg.. The reference network scenaro. devce allows them to be connected to an access node of the TO core network, n the fgure ndcated as Provder Edge (PE) node. VNF Servers are NFV-complant nodes [, 3] owned by network customers that have decded to run VNFs n order to serve the TO network doman they belong to, and obtan economc benefts. A VNF Server can be ether a stand-alone computer, such as the one connected to the CN 6 n Fg., a set of servers organzed as a data center, whose resources are partally or totally dedcated to run VNFs, or an enhanced CPE (ecpe) node. As descrbed n [4], the latter s a CPE devce that s able to run VNFs n a vrtualzed envronment (e.g. the ecpe nodes connectng CN 5 and CN 7 to the TO network). Besdes the hardware facltes, VNF Servers need an amount of bandwdth that s provded them by the TO network. A VNF Server can provde more than one VNF and decde the sellng prces autonomously. In general, a VNF Server can also provde, manage and sell an entre servce chan realzed by connectng local component VNFs. However, for the sake of smplcty and wthout loss n generalty, n the sequel we wll refer to VNF Servers as servers that provde VNFs only. A very mportant role n the system s played by the Orchestrator, whch s n charge of management and orchestraton of the whole system. It runs on a dedcated server and communcates wth all nodes through the TO network. The man tasks performed by the Orchestrator are: The problem of dstrbuted servce chan composton s out of the scope of ths paper and s addressed n [5]. Exposng a lst of VNFs that the TO wants to provde to ts Users; Provdng the VNF Servers wth the VNF templates, contanng the deployment and operatonal behavor requrements necessary to realze each VNF and manage ts lfecycle; Assgnng a slce of bandwdth to the VNF Servers accordng to ther bandwdth request; Provdng each User wth the current lst of VNF Servers that are runnng the requested VNFs, ncludng nformaton regardng the prce appled by each VNF Server and the relevant performance parameters, n terms of experenced latency and receved bandwdth. Allowng Users to choose a VNF Server for each requested VNF functon by settng the flow table of the SDN swtches n the TO network n such a way that User flows traverse the chosen VNF Servers. Polces for management and orchestraton of the resources are a key element of the system as they strongly nfluence ts performance. The man performance parameter that depends on the appled polces s the latency. In fact, placng a VNF on a specfc node n the network determnes that all the flows usng t have to pass through that node; thus, f that node s very far from the sources of some flows, latency may result unacceptable for them. It s evdent that each VNF Server s characterzed by almost the same performance latency parameter for all the Users that enter the network through the same PE node, or through dfferent PE nodes whch are close to each other n the core network,.e., are connected to each other by hgh-speed lnks of a few mles. Another mportant parameter s the bandwdth that each VNF Server provdes to Users, whch depends on both the amount of bandwdth the VNF Server requests to the TO and the number of User flows usng ts VNFs. In the followng, we use the term User Group to ndcate the set of Users requestng the same VNF, whch are characterzed by the same latency from the VNF Servers provdng that VNF, and exhbt the same requrements n terms of delay and bandwdth.

4 4 8) The Vrtual Infrastructure Manager (VIM) block of the Orchestrator confgures the SDN swtches n the TO network n such a way that the specfc flow for whch the User u has chosen a gven Server, traverses ths Server. Fg. 3. Management process flow dagram. Fg. 4. The proposed game-theoretc framework for User Group p U. A. Dstrbuted System Management In ths secton, we descrbe the management framework of the dstrbuted reference system. Let us defne F as the set of VNFs provded by the TO. The man enttes nvolved n the management operatons from ts onboardng to ts usage by User flows are sketched n Fg. 3. Accordngly, for each VNF f F, the relatve steps can be syntheszed as follows: ) The Network Manager (.e. a human operator) creates a VNF template for f. All templates are onboarded and stored on the Orchestrator, and lsted n a VNF Catalogue; ) Servers that are nterested n provdng the VNF f, download the template from the Orchestrator, and create an nstance of f, assgnng to t a set of local resources. 3) Each VNF Server ndvdually decdes the amount of bandwdth requred to serve ts Users, partcpatng to the game G (S) descrbed n Secton III-B; 4) Each VNF Server that has launched the VNF f, requests to the Orchestrator to be regstered on the VNF Server lst as provdng the VNF f, also specfyng the decded prce to be appled to the Users and the requested bandwdth; 5) Each User that s nterested n the VNF f, n the followng ndcated as User u, contacts the Orchestrator to receve nformaton concernng the VNF market of f. More specfcally, the User receves the lst of all the VNF Servers that are runnng f and, for each of them, the bandwdth that ths VNF Server would assgn to u, the prce that t has decded to be appled to the Users of the same User Group, and the IP address of the consdered VNF server, so that the User u can autonomously derve the latency from t. 6) Thanks to the nformaton receved n the prevous tem, the User u chooses to whch VNF Server to connect to. Ths s done by partcpatng to the game G (U) that wll be ntroduced n Secton III-A. 7) The User u communcates to the Orchestrator the VNF Server chosen for f durng the prevous step. B. The Market Model Let us now dscuss the market model that supports the above management framework. Let F be the set of VNFs provded by the network, and U the set of User Groups. For each VNF f p F, let p U a User Group composed by N p Users that are nterested n f p, and S be the set of VNF Servers that provde t. Let d p be the latency encountered by the flows of the Users belongng to p to reach the VNF Server S. It s realstc to assume that User Groups act as dstnct tenants whch are allowed n prncple to connect to the same VNF Server, but, for securty reasons, cannot mx ther traffc wth those generated by other User Groups. Accordngly, requests from each User Group p have to be ndvdually accommodated. The enttes that wll partcpate to ths market are the VNF Servers that have nstalled and run f p, the Users that need f p for some of ther flows, and the Orchestrator that has decded to provde ts customers wth f p and that ntend to gan some economc beneft from t. For each User flow traversng a gven VNF Server, ths Server has to allocate a gven amount of computng and storage resources, and ths represents a cost for the VNF Server. Consderng the generc VNF Server S, we wll refer to the ncremental cost ncurred by the VNF Server to guarantee the requred resources to a new flow requestng functon f p as c p. For example, the cost c p may be an energy cost as n [4], that depends on the prce appled by the energy provder. Therefore, the cost for a VNF Server to manage all the User flows n p, C (F) p, s proportonal to the number of flows n p, that s C (F) p = c p n p () Another cost for the VNF Servers s due to the bandwdth that they receve from the TO network accordng to the requests ssued to the Orchestrator. Let b p be the bandwdth receved by the VNF Server to manage the User Group p, and p (B) the bandwdth-unt prce appled by the TO network to the VNF Server. Note that the value of p (B) does not depend on the User Group. Accordngly, the cost of the overall bandwdth used by that VNF Server s: C (B) p = p (B) b p () On the other hand, the revenue for the VNF Server assocated to the provson of VNF f p s proportonal to both the number n p of Users that are usng t, and the prce ˆp (F) p appled by ths VNF Server. Now, f we assume that the VNF Servers have to pay a commsson (or fee) to the TO, represented by the commsson parameter ψ [, ], the actual revenue of the In Table I we provde a lst of the symbols used throughout the paper wth ther meanng.

5 5 VNF Server related to the provson of VNF f p to the User Group p s R p = p (F) n p (3) where p (F) = ˆp (F) p ( ψ). The mechansm to decde the amount of bandwdth that each VNF Server requests to the TO network wll be dscussed n Secton III. It s amed at maxmzng a utlty functon defned as follows: [ U (S) p (b p) = β R p β C (F) p ] + C (B) p where b p = (b p, b p,..., b Mp ) s the bandwdth vector that contans the bandwdth b p requested to the TO network by each VNF Server, and β and β are approprate constants weghng the relatve relevance of revenues and costs. On the other hand, Users n User Group p, choose the VNF Server by also takng nto account the latency experenced to reach t, d p, and the current prce t s applyng to the VNF f p. However, the hgher the number of User flows usng the same VNF Server, the lower the bandwdth allocated to each of them. Specfcally, the beneft functon of each user n p s expected to be ncreasng n the amount of resource allocated to that user,.e., b p /n p ; and to be decreasng n both the prce ˆp (F) p, and the latency d p. Specfcally, and n lne wth standard economc assumptons [6], we assume that Users experence dmnshng returns as the value of b p /n p ncreases. Such an assumpton can be modeled through concave functons, e.g., logarthmc functons, and has been wdely used n economcs theory to reflect the concept of rsk-averson or satsfacton behavor of ratonal decson-makers. Wth all ths n mnd, each User selects the VNF Server that maxmzes the followng utlty functon [6, 7]: U (U) p (n p) = ln ( α b p n p (4) ) α ˆp (F) p α 3 d p (5) where n p = (n p, n p,..., n Mp ) s the state vector that contans the number n p of flows from the User Group p served by each VNF Server n S; α, α and α 3 are approprate constants that wegh the contrbutons to the utlty functon of the bandwdth receved, the prce appled by the VNF Server, and the latency encountered to reach that Server, respectvely 3. In the followng of the paper, we wll refer to α, α, α 3, β and β as the weghng parameters. C. Telco Operator revenues An mportant topc that deserves partcular attenton s related to the revenues generated by the TO. In centralzed markets such as n Cases A and B, depcted n Fgs. (a) and (b), respectvely, the whole amount pad by Users s receved by the TO tself. Therefore, the TO s able to monopolze revenues generated by the VNF provsonng. Instead, by dstrbutng the VNF provsonng process such as n Case C (see Fg. (c)), a part of the User payments go to VNF Servers because they gve a commsson on the sale of VNFs, and also pay the TO to get the necessary amount of bandwdth to serve ts users. 3 Note that the use of a logarthmc functon n (5) s justfed by the fact that such class of functons have been shown to be proportonally far. Accordngly, let U (TO) A, U (TO) B be the revenues n Cases A, B and C, respectvely. In Case A, we have that all VNFs requred by all User Groups are executed on a sngle server. Therefore, the revenue of the TO can be expressed as follows: U (TO) A = p U and U (TO) C N p ( ˆp (F) p c p ) where N p s the number of Users n User Group p U, ˆp (F) p s the prce charged to users n User Group p to access functon f p on the centralzed server, and c p s the analogous of c p for the centralzed server. In Case B, VNFs are all provded by servers managed by the TO. In ths latter case, the revenue of the TO s: U (TO) B = p U N p ˆp (F) p S (P ) (6) ( ) n (OPT) p c p (7) where S (P ) s the set of propretary servers and n (OPT) p s the optmal number of users n the User Group p on the TOpropretary server, for all p U and S (P ). Fnally, n Case C, VNF provsonng s dstrbuted among dfferent thrd-party VNF Servers, and the overall revenue of the TO s ) U (TO) C = p U ( S C (B) p + ψ S ˆp (F) p n p The frst term n (8) depends on the bandwdth requested by the VNF Servers, whle the second term depends on the commssons pad by those VNF Servers to the TO. The analyss on the effcency of TO s revenues n all the above cases wll be nvestgated n Secton IV-E where we wll show that, n many cases, dstrbutng the VNF market s much more proftable than centralzng t. III. GAME MODEL In ths secton, we llustrate the proposed game-theoretc model of the nteractons between VNF Servers and Users n the dstrbuted management framework. Decsons taken by VNF Servers and Users depend on both ndvdualstc nterests, e.g., maxmze ther own utlty, and decsons taken by counterparts, e.g., opponents strateges. For example, Users connect to one of the avalable VNF Servers dependng on the offered bandwdth and other relevant parameters such as proposed prce and expected communcaton delay. On the contrary, VNF Servers am to maxmze ther revenues and are not lkely to cooperate wth each other. Also, ther actons depend on the number of Users that are connected to them to use ther VNFs. In real scenaros, VNF Servers naturally act and make decsons by antcpatng the Users. Accordngly, nteractons among VNF Servers and Users can be modeled as a two-stage Stackelberg game where VNF Servers act as the leaders of the game and Users as the followers. In the addressed problem we should also consder Users that replcate other Users decsons. Such replcatve behavor naturally arses n those scenaros (8)

6 6 where multple enttes make decsons by replcatng other Users behavor [8 3]. In Secton III-A, we frst defne a game G p (U) among users of the same User Group p U where we explot Evolutonary Game Theory (EGT) and replcator dynamcs to model the decson-makng process of Users. Then, n Secton III-B, we use non-cooperatve game theory to defne the game G p (S) whch models compettve nteractons among the VNF Servers to serve users n the User Group p. Fnally, n Secton III-C we propose a dstrbuted and prvacy preservng renforment learnng procedure to compute the equlbrum of the game G p (S). The llustrated games wll be played each tme some condtons of the system change. More specfcally, a varaton of ether the latency of the VNF Servers from the Users of a User Group, or the prce decded by one of the VNF Servers for a gven functon f p, or the number of Users nterested n the functon f p, determnes a varaton n the utlty functon of some User, and ths stmulates the Users to start playng the game to change the Server. The consequent dstrbuton varaton of the Users on the Servers, n ts turn, stmulates the Servers to play the G p (S) game n order to decde a modfcaton of the bandwdth to be requested to the TO network. The games are terated untl all the enttes n the system reach a new steady state. In Secton IV we wll numercally analyze ths transent perod, and show that t lasts some teratons. The consdered game-theoretc model and ts herarchcal structure are shown n Fg. 4. G (U) p A. Evolutonary game G (U) p among Users Each User s ntrnscally selfsh as t makes decsons wth the am of maxmzng ts own utlty U (U) p, as defned n (5). However, the hgher the number n p of Users n the User Group p connected to the -th VNF Server, the lower the utlty U (U) p of that User. Therefore, the decson-makng process of each User s also nfluenced by decsons taken by the other Users belongng to the same User Group p. Also, f a User s aware that another User s achevng a better utlty, he can decde to mtate that User and mgrate to the same VNF Server to whch that User s connected [3]. In the rest of ths paper, we refer to ths phenomenon as mtaton behavor. Imtaton behavor often arses when consderng nteractons among enttes that ratonally try to maxmze ther beneft by mtatng other enttes decsons that provde better beneft. For example, mtaton s at the bass of a varety of decson makng problems n both wred and wreless networks [9 3] that are often modeled by explotng theoretcal tools from evolutonary game theory. In lne wth a vast body of lterature, we consder the wellknown and wdely used replcator dynamcs [33] as the mtaton dynamcs whch descrbe the nteractons among Users. Accordngly, for each User Group p U, we defne the evolutonary game G p (U) as follows: Populaton: t conssts of the set of the N p Users n User Group p U. Strategy: t s defned as the choce of the VNF Server S to whom each User n the populaton p decdes to connect; the strategy set of each User s S. Utlty: the utlty, or beneft, acheved by each User connected to the VNF Server S s equal to U (U) p as defned n (5). We can now defne the replcator equaton that descrbes how the number of Users n the populaton p that connect to avalable VNF Servers vares ṅ p = n p U (U) p (n p) N p j S n jp U (U) jp (n p) (9) where n p n p denotes the number of Users n the User Group p whch have chosen as a strategy to connect to the -th VNF Server. The frst term n the rght-hand sde of (9) represents the utlty of a User that connects to the -th VNF Server, whle the second term represents the average utlty of the populaton whch depends on the current dstrbuton n p of the populaton. Therefore, the growth rate ṅ p /n p of the number of Users n the User Group p connected to the -th VNF Server s equal to the dfference between the beneft when choosng the strategy, and the average beneft of the whole populaton. A general result from EGT shows that an equlbrum pont for the replcator dynamcs s a fxed pont of the replcator dynamcs such that all Users experence the same beneft,.e., U (U) p = U (U) jp for all, j S. In Proposton, we wll show that the replcator equaton (9) for each User Group p admts a unque soluton for any bandwdth vector b p. Furthermore, we characterze the equlbrum pont by dervng the resultng state vector n p at the equlbrum. To ths purpose, and for notaton purposes, let us defne an auxlary varable φ (p),j [ ( φ (p),j = e α ˆp (F) p ˆp(F) jp as follows: ) ] +α 3(d p d jp) () From (), t can be easly shown that the followng relatonshps hold for all, j, k S and p U φ (p), =, φ(p),j = /φ(p) j,, and φ(p) k,j = φ(p),j φ (p),k () Proposton. For all p U and any gven bandwdth vector b p, the replcator equaton (9) admts a unque evolutonary equlbrum n p. Also, the number of Users n p n User Group p connected to the generc VNF Server S at the equlbrum pont can be derved as follows: where b p b p. n p = N p b p j S b jpφ (p),j () Proof: The replcator equaton can be reduced to an equvalent system of ordnary dfferental equatons (ODEs). Thus, to show that the replcator dynamcs admts a unque equlbrum pont, t suffces to note that the rght-hand sde of the mean dynamc n (9) s contnuously dfferentable. Therefore, Lpschtz contnuty, and thus unqueness of the equlbrum, follow [].

7 7 Now, n order to determne the unque equlbrum, t s well known that t s reached when ṅ p =. Such condton mples that U (U) p = U (U) jp for all, j S,.e., all Users receve the same beneft. Accordngly, we can buld a system of equatons wth N p (N p )/ equatons that can be solved by explotng the relatonshp N p = S n p. Thus, after some easy analytcal dervatons, we obtan the result n (). For the sake of llustraton, n the followng we show how to derve () when N p =. However, the more general case can be treated n a smlar way. From (5) and (), and by mposng U (U) p = U (U) p we get ( ) bp n p ln = ln(φ (p), b p n ) (3) p Recall that N p = n p + n p. Thus, we get n p = N p b p b p + b p φ (p), and n p = N p b p b p + b p φ (p), whch s a specfc case of (). An mportant result that stems from the unqueness of the equlbrum pont s that the replcator dynamcs converges towards a unque stable pont. Therefore, unqueness avods possble oscllatons among two or more equlbrum ponts. Also, snce the equlbrum s unque, complex equlbrum selecton mechansms to determne the most effcent NE n the set of the feasble multple NEs are not requred. To estmate the replcator equaton n (9), each User n the User Group p has to evaluate ts utlty U (U) p (n p) and get the average utlty (n p) of the group. The utlty U (U) p N p j S n jpu (U) jp (n p) s defned n (5) and only requres the value of the rato b p /n p,.e., the amount of bandwdth that wll be assgned to the User. As already stated at pont 5) n Secton II.A, t s worth notng that the value of b p /n p s already avalable n the VNF Server Lst shown n Fg. 3. Instead, snce no User s aware of the strateges of the other Users n p, the average utlty of Users n p s broadcast by the Orchestrator to all Users. It s mportant to note that the above values do not carry any prvate nformaton about decsons taken by other Users. In fact, users do not know the number N p of Users n the group, therefore t s not possble to derve any prvate nformaton from both b p /n p and N p j S n jpu (U) jp B. Stackelberg game G (S) p (n p). between VNF Servers and Users As already dscussed before, VNF Servers act as leaders of the game between VNF Servers and Users. Also, n Proposton we have derved the dstrbuton n p of the populaton p U at the equlbrum of the replcator dynamcs. For the sake of notaton, let us frst defne the two followng auxlary varables ( ) p p = N p β p (F) p β c p (4) Accordngly, we can ncorporate (3), (), (4) and (5) n (4) to rewrte the utlty functon U (S) p of the generc VNF Server S as follows: U (S) p (b b p p) = p p k S b π kpφ (p) b p (6),k For each User Group p U, we defne the non-cooperatve game G p (S) as follows: Player set: t conssts of the set S of VNF Servers. Strategy: t s defned as the amount of bandwdth b p to be requested to the TO network to serve ts connected Users from User Group p. For each User Group n U, we assume that such amount of bandwdth s bounded by B. Thus, the strategy set s B = S B, where B = [, B ] and dentfes the Cartesan product 4. Utlty: the utlty of each VNF Server S s equal to U (S) p as defned n (6). By calculatng the frst-order dervatve of (6), t can be easly shown that p p leads to a non-postve frst-order dervatve of the utlty functon U (S) p. In other words, the best strategy for the -th VNF Server s not to partcpate n the game G p (S) for User Group p,.e., b p =. Therefore, those VNF Servers wth p p ext the game and they can be removed from the player set S. Accordngly, wthout loss of generalty, n our model we assume that the player set S s composed by only those VNF Servers such that p p >. In the followng, we analyze the Stackelberg game G p (S) and provde useful results about ts equlbrum ponts, referred to as SEs. Defnton. Let b p B. The strategy profle (b p, n p) s a SE for the game G p (S) f for all b p B and S, we have where n p s defned as n (). U (S) p (b p, n p) U (S) p (b p, n p) Defnton. Let b p = (b p, b p ), where b p s the bandwdth vector of all players except,.e., b p = (b jp ) j S, j wth b jp b p. The strategy b p = (b p, b p,..., b Mp ) s sad to be the Stackelberg strategy for the game G p (S) f for all S we have that b p = arg max U (S) p (b p, b p b, n p) p B Also, the value U (S) p (b p, n p) s denoted as the Stackelberg utlty of leader n game G p (S). In Proposton, we prove that the game G p (S) unque SE. Proposton. The game G (S) p admts a unque SE. admts a Proof: The man steps of the proof are as follows. Frst, we prove the exstence of the equlbrum by explotng concavty propertes of VNF Server utlty functons n (6). Then, and π = β p (B) (5) 4 We do not consder the varable ˆp (F) as a strategy for the VNF Server as we lmt our study to the case where, whle the bandwdth vares n tme, the prcng polcy remans constant durng each game executon.

8 8 we show that the Dagonal Strct Concavty (DSC) property holds. The DSC property mples that VNF Servers experence dmnshng returns along any drecton,.e., along all b p b p. Fnally, we explot results contaned n [34, 35] to prove that a unque equlbrum exsts. Let the margnal utlty v p (b p ) of each player S be U (S) p (bp) defned as v p (b p ) = b p. Therefore, from (6) t follows that the margnal utlty of the generc VNF Server s k S,k v p (b p ) = p b kpφ (p),k p ( ) π (7) k S b kpφ (p),k where p p s defned n (4). To show that the DSC property holds, t must be shown that: ) U (S) p (b p) s strctly concave n b p ; ) U (S) p (b p) s convex n b p ; and ) the functon ρ(b p, r p ) defned as ρ(b p, r p ) = S r p U (S) p (b) (8) s concave n b p for some r p = (r p, r p,..., r Mp ) such that r p > for all S. From (6), t can be shown that property ) holds as U (S) p (b p) s defned as the dfference between a strctly concave functon and a concave functon. To prove ), t suffces to note that the Hessan matrx of U (S) p (b p) has all non-negatve egenvalues,.e., the Hessan matrx s postve semdefnte. Let r p = / p p for all S. Accordngly, (8) can be rewrtten as follows: ρ(b p, r p ) = b p S b p + j b r jpφ (p) p π b p,j S b p = b p + j b + b kp jpφ (p),j k b kp + j k b jpφ (p) k,j S From (), we have that ρ(b p, r p ) = S r p π b p (9) b p b p + k b + b kp φ (p),k kpφ (p),k k b p + j b jpφ (p),j r p π b p = S r p π b p () Observe that ρ(b p, r p ) s a concave functon n b p as requred n ). Therefore, we have that DSC property holds and the general theory n [34, 35] ensures the unqueness of the equlbrum. In (6), we use the equlbrum condton n (). Therefore, nteractons between Users (.e., the followers) and VNF Servers (.e., the leaders) modeled through the game G p (S) produce a unque SE (b p, n p). However, recall that VNF Servers compete wth each other n the Stackelberg game. Accordngly, the strategy profle b p dscussed above also represents a Nash Equlbrum (NE) [35] for the compettve game among VNF Servers. Algorthm Exponental Renforcement Learnng (XL) Parameter: step-sze sequence γ m (default: γ m = /m). Intalze: m ; z p for all S. Repeat m m + ; for each VNF Server S do smultaneously requested bandwdth b p B [ + exp( z p)] ; measure margnal utlty v p from (7); update scores: z p z p + γ mv p; untl termnaton crteron s reached. C. Renforcement Learnng Procedure for game G (S) p In Proposton, we have shown that the game G p (S) admts a unque equlbrum. Unfortunately, we are not able to fnd a proper characterzaton of the equlbrum and provde closedform expressons. Thus, we need to provde a robust mechansm to allow VNF Servers to ndvdually reach the equlbrum of the game. Accordngly, n the followng we propose an exponental renforcng learnng [36, 37] procedure, whch provably converges to the unque equlbrum of the game. For each VNF Server S that serves Users n the User Group p U, we defne the followng learnng procedure { zp (m + ) = z p (m) + γ m v p (b p (m)) e b p (m + ) = B z p (m+) () +e z p (m+) where m represents the teraton ndex, b p (m) s the bandwdth vector at teraton m, and γ m s the step-sze of the learnng procedure whose mportance wll be explaned later. For each User Group p U, the algorthmc mplementaton of () s shown n Algorthm. In the followng Proposton 3, we show that the proposed exponental renforcng learnng procedure converges to the equlbrum of the game. Proposton 3. Let γ m be the step-sze of the learnng procedure and m γ m < m γ m = +. For any feasble ntal condton n B and User Group p U, Algorthm always converges to the unque SE of G p (S). The proof conssts n showng that ) the mean dynamc of (),.e., ts contnuous-tme verson, converges to the equlbrum of the game as tme goes to nfnty, and ) () s an asymptotc pseudo-trajectory (APT) [38] for the contnuoustme verson of (). For a detaled and rgorous proof, we refer the reader to Appendx A. From Proposton 3, we have that any varable step-sze rule n the form γ m = /m β wth β (.5, ] wll converge to the unque SE of the game G p (S). Let us note that, n order to compute b p (m + ) n (), each VNF Server S s requred to know v p (b p (m)) n (7), whch only depends on the term k S b kp(m)φ (p),k. To ths purpose, the Orchestrator has full access to the VNF Server parameters (e.g., d p, p (F) p, etc...). Then, at each teraton and for each VNF Server S, the Orchestrator s able to compute the overall sum k S b kp(m)φ (p),k and send t to the correspondng -th VNF Server. Note that, by so dong, the -th VNF Server cannot extract any prvate nformaton on other VNF Servers from the sum k S b kp(m)φ (p),k. Thus, t

9 9 Requested bandwdth at the SE, b Dstrbuton of Users at the SE, s 5 x VNF Server S VNF Server S Prce parameter of S, p (F) Fg. 5. Requested bandwdth and populaton dstrbuton at the equlbrum as a functon of the prce p (F) charged by S (Sold lnes: d = 5 DUs and d = 4 DUs; Dashed lnes: d = d = 4 DUs) means that the learnng procedure () can be mplemented n a prvacy-preservng and dstrbuted fashon. IV. NUMERICAL ANALYSIS In ths secton, we present a numercal analyss of the proposed dstrbuted orchestraton and resource allocaton scheme. In our smulatons, we assume a populaton sze of N = 3 Users and, unless otherwse stated, we consder the followng weghng parameters: α =, α =.5, α 3 =.35, β = and β =. Fnally, and unless explctly mentoned otherwse, we assume that the commsson parameter s ψ =, and the bandwdth-unt prce p (B) s equal for all VNF Servers n S and s set to p (B) = Prce Unts (PUs). For llustratve purposes, we prmarly focus on the two VNF Servers case (.e., M = ) as t allows us to hghlght the dynamcs of the nteractons among Users and VNF Servers together wth the mpact of the varous system parameters on the outcome of the game G (S). Moreover, we also provde extensve results also for the case M >, whch makes possble to show the feasblty of the proposed learnng procedure and to analyze the mpact of latency when multple VNF Servers are wllng to provde the consdered VNF. A. Impact of prcng on the SE In ths secton, we prelmnarly study the mpact of the prcng appled by the VNF Servers. To ths purpose, n Fg. 5 we show the outcome of the game as a functon of the prce p (F) charged by VNF Server S to ts Users, when the prce appled by VNF Server S s assumed constant and equal to p (F) = 6 PUs. Specfcally, we show the amount of bandwdth b that each VNF Server requests to the TO network and the number n of Users that connect to each VNF Server at the SE. Also, we consder two dfferent confguratons of the latences d experenced by Users connected to the -th VNF Server. For the sake of generalty, we wll express latency n terms of Delay Unts (DUs). In more detal, sold lnes llustrate the outcome of the game when d = 5 DUs and d = 4 DUs, respectvely. Instead, dashed lnes refer to the case when d = d = 4 DUs. As expected, when p (F) s hgh, Users are Revenues at the SE Costs at the SE Utltes at the SE x x 4 VNF Server S VNF Server S 4 6 x Prce parameter of S, p (F) Fg. 6. Revenues, costs and utltes of VNF Servers S and S as a functon of the prce p (F) charged by S (Sold lnes: d = 5 DUs and d = 4 DUs; Dashed lnes: d = d = 4 DUs). lkely to connect to VNF Server S because t apples a lower prce (.e., p (F) = 6 PUs). Accordngly, Users get hgher payoffs when they connect to VNF Server S ndependently of the experenced connecton latences d. On the contrary, when p (F) s low, n order to attract more Users, the strategy of VNF Server S conssts n requestng a hgh amount of bandwdth to the TO network. In ths way, as evdent from (5), the utlty of the Users ncreases as a consequence of the ncrease n the shared bandwdth. Such behavor holds for values of p (F) that are below a gven threshold, above whch requestng more bandwdth s no more the optmal choce. For values of p (F) hgher than ths threshold 5, the optmal strategy of the VNF Servers conssts n reducng the amount of requested bandwdth. Such behavor s motvated by the fact that an ncrease n the requested bandwdth causes an ncrease n the costs, whch also leads to a reducton n the utltes acheved by the VNF Servers. Accordngly, when the cost to provde more resources to the Users s hgher than the expected revenues, VNF Servers prefer to reduce the amount of shared resources to reduce costs and keep hgh revenues. Fnally, t s worth notng that when d = 5 DUs and d = 4 DUs (sold lnes), both VNF Servers request to the TO network a lower amount of bandwdth than n the case when d = d = 4 DUs. Ths s due to the fact that, when latences are equal (or smlar), there s no monopolstc behavor and VNF Servers have to compete to attract more Users, whch results n hgher requested bandwdth. In Fg. 6 we show revenues, costs and utltes acheved by VNF Servers S and S at the SE as a functon of the prcng parameter p (F). More n detal, revenues and costs are defned as the frst and second terms n (4), respectvely. Instead, utltes are equal to U (S), and are defned as n (4). From (3), we have that revenues acheved by each VNF Server depend on R and are determned by the number n of Users that connect to that VNF Server as shown. Accordngly, Fg. 6 shows that revenues vary accordng to the dstrbuton of Users at the SE beng consdered n Fg. 5. On the contrary, from () 5 In general, the threshold values are dfferent for the two VNF Servers.

10 α Dstrbuton of Users on S at the SE, s α 3 Dstrbuton of Users on S at the SE, s α experenced latency more than the cost to obtan a share of the resources, Users are more attracted by the VNF Server S. In Fg. 8, we show the bandwdth requested by both VNF Servers at the equlbrum as a functon of α and α 3. Fg. 8 shows that the behavor of both VNF Servers s smlar. For example, the requested bandwdth ncreases for low values of both α and α 3, and then decreases when ether α or α 3 decrease. Even though the behavor s smlar, the requested bandwdth consderably dffers for the two VNF Servers. In fact, Fg. 8 shows that the hghest value of b s 3.5 4, whle the maxmum value of b s 4. Fg. 7. Dstrbuton of Users at the SE as functon of the two weghts α and α 3 n (5). α Requested bandwdth by S at the SE, b α 3 Requested bandwdth by S at the SE, b α 3 x 4 4 Fg. 8. Requested bandwdth of the two VNF Servers at the SE as a functon of the two weghts α and α 3 n (5). and () we have that costs depend on both the number n of Users connected to VNF Server S and the requested bandwdth b at the SE. Therefore, as shown n Fg. 6 the resultng trend of experenced costs s a combnaton of both n and b, whch are shown n Fg. 5. Note that, when the value of the prce parameter p (F) s hgh, the number of Users connected to S asymptotcally tends to N,.e., the whole Users populaton s lkely nclned to connect to the VNF Server S whch provdes better performance for a lower prce p (F). Therefore, for hgh values of p (F), S requests a small amount of bandwdth such that ts revenues and costs asymptotcally tend to zero. B. Impact of the weghng parameters on the SE In ths secton, we estmate the mpact of the weghts α and α 3 that appear n (5) on the outcome of the game G (S) when M =. To ths purpose, n Fgs. 7 and 8 we show the dstrbuton of Users at the SE as a functon of α and α 3. In our smulaton we have assumed p (F) = 6 PUs, p (F) = PUs, d = 5 DUs and d = 4 DUs. Fg. 7 llustrates the dstrbuton of Users at the equlbrum. When α s hgh and α 3 s low,.e., Users are much more concerned about the prce charged by VNF Servers than the experenced latency, Users are much more attracted by the VNF Server S snce p (F) << p (F). On the contrary, when α 3 s hgh but α s low,.e., Users wegh the 3 C. Tme-varyng and PE postonng analyss In ths secton, we dscuss the mpact of the populaton sze N, the cost c to process each flow, and the PE poston on the outcome of the game G (S). To ths purpose, we smulated a scenaro where the number of Users requestng a gven VNF and the cost c vary n tme accordng to realstc nght/day usage patterns. More specfcally, let the number of Users N and the cost c vary n a 48-hours long temporal wndow as shown n Fg. 9(a). Fg. 9(b) llustrates both the strategy of each VNF Server,.e., the bandwdth requested to the TO network, and the dstrbuton of the populaton at the equlbrum when p (F) = p (F) = 8 PUs, d = 5 DUs and d = 4 DUs. Observe that when the cost to process flows s low, VNF Servers can support more User connectons. Accordngly, VNF Servers request more bandwdth to the network to attract a hgher number of Users. However, when d = 5 DUs and d = 4 DUs, even though the VNF Server S provdes Users wth a hgher amount of bandwdth, t also has a hgh latency. Therefore, to reduce the experenced latency, Users connect to the VNF Server S and thus n > n. Instead, when both VNF Servers have equal latency,.e., d = d = 4 DUs, Fg. 9(c) shows that the majorty of the populaton chooses the VNF Server whch provdes the hghest amount of bandwdth. To study the mpact of the PE poston,.e., the User entrance ponts to the network, w.r.t. the poston of VNF Servers on the outcome of the game G (S), we consder fve VNF Servers,.e., S = {S, S, S 3, S 4, S 5 }, and fve possble postons of the access PE, here denoted as PE k wth k =,,..., 5. We assume ˆp (F) = 6 PUs for all S. Each access PE poston corresponds to a dfferent latency confguraton. For example, n Fg. (a) t s shown that VNF Servers S and S provde low latences when Users access the network through provder edges PE and PE, and hgh latences when Users access through PE 4 and PE 5. For VNF Servers S 4 and S 5, what happens s exactly the opposte, whereas S 3 provdes low latences to Users ndependently of the poston of the access PE. When Users access through PE and PE, Fg. (b) shows that the majorty of them decdes to connect to S and S. Thus, as shown n Fg. (c), to attract such an expectedly ncreasng number of Users, S and S request a hgh amount of bandwdth to the TO network. As expected, the contrary holds n the case Users access through PE 4 and PE 5.

11 Populaton Sze, N Per user energy cost, c VNF Server S 5 VNF Server S Tme [hours] (a) Requested bandwdth at SE, b Dstrbuton of Users at the SE, s 6 4 VNF Server S VNF Server S Tme [hours] (b) Requested bandwdth at SE, b Dstrbuton of Users at the SE, s VNF Server S VNF Server S Tme [hours] (c) Fg. 9. a) Populaton sze N and prce parameter c as a functon of tme; b) Requested bandwdth and dstrbuton of Users at the equlbrum as a functon of tme when d = 5 DUs and d = 4 DUs; c) Requested bandwdth and dstrbuton of Users at the equlbrum as a functon of tme when d = d = 4 DUs Latency parameter, d Dstrbuton of Users at the SE, n Requested bandwdth at the SE, b PE PE PE 3 PE 4 PE 5 PE PE PE 3 PE 4 PE 5 PE PE PE 3 PE 4 PE 5 3 S S S 3 S 4 S 5 (a) (b) (c) Fg.. Latences (a), dstrbuton of Users (b), and requested bandwdth (c) at the SE for dfferent access PE postons. D. Convergence Analyss In ths secton, we nvestgate the convergence of the proposed learnng procedure n (). Specfcally, we are nterested n analyzng the convergence speed of () and ts scalablty w.r.t. the number M of VNF Servers. Results shown n ths secton are averaged over smulaton runs where we have assumed ˆp (F) = 9 PUs for all S, whle the latency d and the cost c parameters have been randomly generated as llustrated below. Also, for llustratve purposes we frst focus on sngle User Group case. Instead, n the sequel we wll also provde results for the case of multple User Groups, whch makes possble to show the adaptablty to network confguraton changes of the proposed learnng procedure. At each smulaton run, the latency parameters d are randomly generated from two Gaussan dstrbutons. Specfcally, the frst half of M/ VNF Servers are assocated wth a Gaussan dstrbuton wth mean values µ (d) = 3 and standard devatons equal to σ = /8µ (d). Instead, the second half of M/ VNF Servers are assocated to a Gaussan dstrbuton wth µ (d) = 45 and σ = 3/6µ (d). The cost parameters c are generated from a Gaussan dstrbuton wth mean value µ (c) = 85 and standard devaton σ = 3. To measure the convergence speed of the proposed learnng procedure, at each teraton we consder the normalzed Eucldean dstance between the bandwdth vector b(m) computed n () and the SE vector b as follows: ( d(b(m), b b (m) b ) = ) () S In Fg. (a), we show how fast the proposed learnng procedure converges to the unque SE of the game G p (S) when M =, for dfferent step-sze rules. Specfcally, we consder both varable step-sze (.e., γ m = /m β ) wth β {.5, }, and fxed step-sze rules (.e., γ m {, 3}). It s shown that fxed step-sze rules converge faster than varable step-sze rules. In addton, the convergence speed s faster when hgh values of the fxed step-sze are consdered,.e., γ m = 3. Recall that convergence of the learnng procedure under varable stepsze rules s ensured by Proposton 3. Unfortunately, the same s not true for fxed step-sze rules, as n ths case convergence to the SE cannot be proven analytcally. It s worth notng that very large fxed step-sze are prone to generate oscllatons around the SE 6. Therefore, to guarantee convergence to the SE whle achevng a fast convergence speed, a varable step-sze γ m = /m β wth β =.5 should be consdered. Fnally, n Fg. (b), we show how many teratons the proposed learnng procedure needs to reach the SE as a functon of the step-sze γ m for dfferent values of the number M of 6 To avod oscllatons, f generated, approaches smlar to Search-thenconverge (STC) [36] can be effectvely appled. B

12 Dstance from the SE.5.5 Varable (γ m =/m) Varable (γ m =/m.5 ) Fxed (γ m =) Fxed (γ m =3) Iteraton, m (a) Number of teratons M=6 M= M= M=5 3 Fxed Step sze, γ m (b) Per User Group Average Utlty x 4 SE Fxed (γ m =.8) Fxed (γ m =) Varable (γ m =/m.5 ) Tme, t Fg.. a) Dstance from the SE for dfferent step-sze rules; b) Number of teratons needed to reach the SE for dfferent number M of VNF Servers and values of the step-sze γ m; c) Adaptablty of the proposed learnng procedure as a functon of dfferent step-sze rules. (c) VNF Servers when we consder a fxed step-sze rule. More n detal, we let the learnng procedure run untl the stoppng condton s reached,.e., d(b (m), b ). for all S. As expected, an ncrease n the value of the step-sze mproves the convergence speed of the learnng procedure. Furthermore, n Fg. (b) we show the scalablty of the proposed learnng procedure w.r.t. the number M of VNF Servers. It s mportant to note that an ncrease n the value of the step-sze γ m allows to mprove the convergence speed of the learnng procedure even when hgh number of VNF Servers are consdered, e.g., M = 5. Thus, by properly ncreasng the value of the stepsze, t s also possble to mprove the scalablty of the learnng procedure. Now, we nvestgate the adaptablty of the proposed learnng procedure when network parameters change over tme. Specfcally, we consder M = VNF Servers whch serve fve dfferent User Groups each of whch s requestng a dfferent functon. Fg. (c) shows the per-user Group average VNF Server utlty defned as U M S p U U (S) p at the SE and that acheved by usng the learnng procedure for dfferent stepsze rules as a functon of tme. For each User Group p U, at tme nstant t =, both the games G p (U) and G p (S) are played and, as shown n Fg. (c), the SE s reached n few teratons. Then, at tme nstants t = {6, 8} we smulate a network confguraton change. More n detal, at t = {6, 8} we randomly generate a new latency parameter confguraton. Note that any change n the latency confguratons pushes users n each User Group to re-dstrbute themselves accordng to the new network confguraton through the evolutonary game G p (U). It follows that, to adapt to new network and user dstrbuton condtons, and to evaluate the amount of bandwdth to be requested to the TO for each User Group, VNF Servers need to re-execute the game G p (S). As shown n Fg. (c), at tme nstants t = {6, 8}, users wll re-arrange themselves dfferently, thus causng a devaton from the prevous SE. Accordngly, the learnng procedure s re-executed by VNF Servers and Fg. (c) shows that the system s able to quckly adapt to system fluctuatons,.e., the proposed learnng procedure s able to reach the SE n few teratons. As expected, the convergence rate s faster n the case of fxed step-sze rules. Specfcally, the hgher the step-sze, the faster the convergence speed and adaptablty of the learnng procedure. However, even though the consdered varable step-sze rules show bad performance n terms of number of teraton needed to reach the equlbrum, recall that they assures convergence. Instead, the same does not hold for fxed step-sze rules whch are fast but ther convergence to the equlbrum cannot be analytcally proved. E. TO revenue effcency analyss As ntroduced n Secton II, an mportant aspect that deserves partcular attenton s related to the revenues generated by the TO. In the followng, to measure the effcency of the proposed dstrbuted mechansm w.r.t. the revenues of the TO we consder the effcency rato, defned as the rato between the revenues generated by the TO under our dstrbuted market model and that acheved under monopolstc and centralzed models,.e., Cases A and B n Fgs. (a), (b). Specfcally, the effcency rato n Case A and Case B s denoted as ξ A and ξ B, respectvely, and defned as follows: ξ A = U (TO) C U (TO) A, ξ B = U (TO) C U (TO) B (3) where U (TO) A, U (TO) B and U (TO) C are defned n (6), (7) and (8), respectvely. An effcency rato hgher than or equal to means that our proposed mechansm provdes revenues to the TO whch are ether hgher or equal to that acheved n centralzed markets. In Case A, the poston of the unque centralzed server s of extreme mportance as t wll determne the latency that network users wll experence when connectng to that server and the cost parameter c. Therefore, to nvestgate the effcency of the proposed mechansms under dfferent network confguratons, we consder 5 possble latency and cost confguratons. Specfcally, the VNF Server poston confguraton and the cost parameters used by each VNF Server, that s c p, have been generated by usng the same Gaussan dstrbutons descrbed n Secton IV-D. In order the Case A to be comparable, also n ths case we consdered 5 smulaton runs, where the latences

13 3 TO Revenue Effcency p (B) =.5.5 Commsson parameter, ψ p (B) =. 8 Medan Mean 5% 6 75% Percentle 9% 9% Percentle Commsson parameter, ψ Fg.. Effcency rato ξ A as a functon of the commsson parameter ψ for dfferent values of the bandwdth-unt prce p b. TO Revenue Effcency.5.5 p (B) = p (B) =.5 p (B) =. p (B) = Commsson parameter, ψ Fg. 3. Effcency rato ξ B as a functon of the commsson parameter ψ for dfferent values of the bandwdth-unt prce p b. from the unque VNF Server and the costs, have been calculated as the average value of all the latences and the costs n the same run for the Case C. For llustratve purposes, n the followng we assume that the parameters p (B), ˆp (F) and ψ are fxed and equal for all VNF Servers S and we consder a sngle User Group. We consder all the above possble network confguratons and, for each network confguraton, we evaluate the effcency rato ξ A. Obtaned results are shown n Fg. where we show ξ A as a functon of the commsson parameter ψ for dfferent values of the bandwdth-unt prce p (B). Fg. shows that, n most realzatons, dstrbutng VNF functons s more effcent than havng a sngle server that manages and controls the VNFs. Also, t s shown that only under few network confguratons the proposed mechansm s not effcent. Instead, n the majorty of the cases hgh effcency s obtaned, ξ A >, f compared to the centralzed market scheme n Case A. Furthermore, an nterestng result s related to the bandwdth-unt prce p b. Specfcally, an ncrease n the value of p b also ncreases the effcency of the proposed dstrbuted market model. Instead, n Fg. 3, we consder Case B, where VNFs are executed on M servers whch are owned by the TO, and we compare t wth our proposed mechansm where the same servers are, nstead, VNF Servers owned by customers. Accordngly, for each smulaton run we have consdered the same poston and prce confguratons of the relatve run of Case C. Fg. 3 shows the effcency rato ξ B as a functon of the commsson parameter ψ for dfferent values of the bandwdth-unt prce p b. The proposed mechansm s not effcent for the TO for small values of the commsson parameter ψ. Specfcally, when the commssons on VNF provsonng sent by VNF Servers s low,.e., U (TO) C S C(B), the proposed mechansm fals n mprovng the revenues of the TO. Instead, t s worth notng that when small values of the commsson parameter are consdered, e.g., 4% 8% of the overall revenues generated by the sale of VNFs, our proposed soluton s more convenent for the TO than the centralzed ones. Ths s an mportant aspect whch show that the proposed model for the dstrbuton of the VNF provsonng s not only proftable for VNF Servers whch are now able to enter a new market, but t s also proftable for the TO. In fact, our dstrbuted market allows the TO to receve payments from VNF Server w.r.t. both bandwdth requrements and commsson fees on payments submtted by network users. Instead, Case B only allows payments from network users. Accordngly, by allowng thrd-party VNF Servers to access the VNF market, the TO can mprove ts revenues and also reduce the computatonal cost whch s outsourced to those VNF Servers. Also, note that an ncrease n the value of the bandwdth-unt prce p b mproves the effcency of the proposed mechansm aganst Case B. The numercal analyss of the proposed system shows that the three stakeholders can take advantage of the framework; ndeed the TO can smplfy the Orchestraton mechansm by usng the dstrbuted scheme and employng external servers for VNFs provsonng. The VNF Servers on ther sde are sellers and can ncrease ther economc beneft. Fnally, Users can also ncrease ther beneft n terms of prce reducton and performance mprovement. V. CONCLUSIONS In ths paper we have dscussed how game-theoretc tools can be effectvely used to address the problem of dstrbuted management, resource allocaton and orchestraton of softwarzed networks. Specfcally, we exploted herarchcal game theory to desgn a dstrbuted SDN/NFV system where VNF Servers partcpate n the VNF market as sellers of VNFs. The nteractons among VNF Servers and Users requestng VNFs have been modeled as a two-stage Stackelberg game where the former act as the leaders and the latter as the followers of the game. Unqueness of the SE has been proved, and a renforcng learnng procedure whch provably converges to the unque SE has been proposed. We accounted for mtatve and socal behavors of Users and we used the replcator dynamcs equaton from evolutonary game theory to model ther nteractons. Furthermore, a closedform equlbrum condton has been derved. Through the explotaton of game theory, we have shown that a dstrbuted framework results benefcal for all the nvolved stakeholders. On the one hand, the smplfed orchestraton and the possblty to offload VNFs on thrd-partes VNF Servers s benefcal to the TO. On the other hand, VNF Servers have

14 4 economc beneft by partcpatng to the VNF Market as sellers. Fnally, Users can choose the VNF Servers that most ft ther needs. The numercal analyss carred out has proved the feasblty and effcency of the proposed game-theoretc framework. In partcular, the results obtaned show that the framework s scalable and rapdly adapts to network changes. REFERENCES [] Open Networkng Foundaton, Software-defned networkng: The new norm for networks, ONF Whte Paper,. [] M. Yu, L. Jose, and R. Mao, Software defned traffc measurement wth OpenSketch. n Proc. of NSDI, vol. 3, 3, pp [3] C. Monsanto, J. Rech, N. Foster, J. Rexford, and D. Walker, Composng software defned networks. n Proc. of NSDI, 3, pp. 3. [4] ( Whte Paper.pdf) Whte paper on network functons vrtualzaton. [5] S. Raynovch. Lfecycle servce orchestraton (LSO) market overvew, SDX central market report, january 6. [6] F. Wuhb, R. Yanggratoke, and R. Stadler, Allocatng compute and network resources under management objectves n large-scale clouds, Journal of Network and Systems Management, vol. 3, no., 5. [7] M. Alcherry and T. Lakshman, Network aware resource allocaton n dstrbuted clouds, n Proc. of IEEE Infocom,. [8] X. Meng, V. Pappas, and L. Zhang, Improvng the scalablty of data center networks wth traffc-aware vrtual machne placement, n Proc. of IEEE Infocom, March, pp. 9. [9] J. W. Jang, T. Lan, S. Ha, M. Chen, and M. Chang, Jont vm placement and routng for data center traffc engneerng, n Proc. of IEEE Infocom,, pp [] A. Basta, W. Kellerer, M. Hoffmann, H. J. Morper, and K. Hoffmann, Applyng NFV and SDN to LTE moble core gateways, the functons placement problem, Proc. of the 4th Workshop on All Thngs Cellular: Operatons, Appl. and Challenges, Chcago, USA, 4. [] A. Lombardo, A. Manzaln, V. Rccobene, and G. Schembra, An analytcal tool for performance evaluaton of software defned networkng servces, Proc. of IEEE SDNMO, Krakow, Poland, 4. [] H. Moens and F. D. Turck, VNF-P: A model for effcent placement of vrtualzed network functons, n Proc. of CNSM 4, 4. [3] M. Bouet, J. Leguay, and V. Conan, Cost-based placement of vrtualzed deep packet nspecton functons n sdn, n Proc. of IEEE Mlcom, 3, pp [4] S. Gebert, D. Hock, T. Znner, P. Tran-Ga, M. Hoffmann, M. Jarschel, E.-D. Schmdt, R.-P. Braun, C. Banse, and A. Köpsel, Demonstratng the optmal placement of vrtualzed cellular network functons n case of large crowd events, n Proc. of ACM Sgcomm, 4. [5] M. Scholler, M. Stemerlng, A. Rpke, and R. Bless, Reslent deployment of vrtual network functons, n Ultra Modern Telecommuncatons and Control Systems and Workshops (ICUMT), 3 5th Internatonal Congress on, 3. [6] R. Cohen, L. Lewn-Eytan, J. S. Naor, and D. Raz, Near optmal placement of vrtual network functons, n Proc. of IEEE Infocom, 5, pp [7] R. Rav and A. Snha, Approxmaton algorthms for multcommodty faclty locaton problems, SIAM Journal on Dscrete Mathematcs, vol. 4, no., pp ,. [8] M. F. Bar, S. R. Chowdhury, R. Ahmed, and R. Boutaba, On orchestratng vrtual network functons, Proc. of th IEEE Internatonal Conference on Network and Servce Management (CNSM), November 5. [9] M. Bar, S. R. Chowdhury, R. Ahmed, R. Boutaba et al., On orchestratng vrtual network functons, n Network and Servce Management (CNSM), 5 th Internatonal Conference on. IEEE, 5. [] A. Gupta, M. F. Habb, P. Chowdhury, M. Tornatore, and B. Mukherjee, Jont vrtual network functon placement and routng of traffc n operator networks, 5. [] W. H. Sandholm, Evolutonary game theory, n Encyclopeda of Complexty and Systems Scence. Sprnger, 9, pp [] ETSI NVF GS, Network Functons Vrtualzaton (NFV) Infrastructure Overvew, NFV-INF V.., Jan 5. [3] ETSI NFV GS, Network Functon Vrtualzaton (NFV) Management and Orchestraton, NFV-MAN v.8., Nov 4. [4] G. Farac and G. Schembra, An analytcal model to desgn and manage a green SDN/NFV CPE node, Network and Servce Management, IEEE Transactons on, vol., no. 3, pp , Sept 5. [5] S. D Oro, L. Gallucco, S. Palazzo, and G. Schembra, Explotng congeston games to acheve dstrbuted servce channg n NFV networks. IEEE Journal on Selected Areas n Communcatons, ths ssue. [6] R. W. Shephard and F. Rolf, The law of dmnshng returns, producton theory. Sprnger Berln Hedelberg, 974. [7] A. Mas-Colell, M. D. Whnston, J. R. Green et al., Mcroeconomc theory. Oxford unversty press New York, 995, vol.. [8] A. Bosch-Domènech and N. J. Vrend, Imtaton of successful behavour n cournot markets, The Economc Journal, vol. 3, no. 487, pp , 3. [9] G. Nan, Z. Mao, M. Yu, M. L, H. Wang, and Y. Zhang, Stackelberg game for bandwdth allocaton n cloud-based wreless lve-streamng socal networks, Systems Journal, IEEE, vol. 8, no., pp , March 4. [3] J. Elas, F. Martgnon, A. Capone, and E. Altman, Non-cooperatve spectrum access n cogntve rado networks: a game theoretcal model, Computer Networks, vol. 55, no. 7, pp ,. [3] H. Tembne, E. Altman, R. El-Azouz, and Y. Hayel, Evolutonary games n wreless networks, Systems, Man, and Cybernetcs, Part B: Cybernetcs, IEEE Transactons on, vol. 4, no. 3, pp ,. [3] S. Gesendorf et al., The nfluence of nnovaton and mtaton on economc performance, Economc Issues, vol. 4, no., p. 65, 9. [33] P. D. Taylor and L. B. Jonker, Evolutonary stable strateges and game dynamcs, Mathematcal boscences, vol. 4, no., pp , 978. [34] C. Goodman, Note on exstence and unqueness of equlbrum ponts for concave n-person games, Econometrca: Journal of the Econometrc Socety, 98. [35] J. B. Rosen, Exstence and unqueness of equlbrum ponts for concave n-person games, Econometrca: Journal of the Econometrc Socety, pp , 965. [36] S. D Oro, P. Mertkopoulos, A. Moustakas, and S. Palazzo, Interferencebased prcng for opportunstc mult-carrer cogntve rado systems, Wreless Communcatons, IEEE Transactons on, vol. 4, no., pp , 5. [37] P. Mertkopoulos and A. L. Moustakas, The emergence of ratonal behavor n the presence of stochastc perturbatons, The Annals of Appled Probablty, vol., no. 4, pp ,. [38] M. Benaïm, Dynamcs of stochastc approxmaton algorthms, n Semnare de probabltes XXXIII. Sprnger, 999, pp. 68. VI. APPENDICES APPENDIX A. PROOF OF PROPOSITION 3 Proof: For the sake of clarty, n the followng we wll omt the subscrpt p whch dentfes the User Group p U. The mean dynamcs of () s {ż = v (b) b = B e z +e z (4) At any gven tme t, let b(t) be a soluton for (4). In system theory, such soluton s often referred to as soluton orbt or trajectory of the system. In the followng, we show that ) b(t) converges to b as t +, and ) () s an asymptotc pseudo-trajectory (APT) [38] for the mean dynamc (4), and converges to b f some mld condtons on the step-sze are satsfed. From Proposton, we have that U (S) (b) s a strctly concave functon n b. Therefore, v (b)(b b ) < for all b [, B ] by defnton. By explotng ths latter result, t can be shown that the functon V (b) defned as V (b) = ( B b ) ( b B ln + b ) B S b ln B b b B b (5) s a strct Lyapunov functon for (4). In fact, we have that V = dv (b)/dt = S v (b)(b b ) <, V (b ) = and V (b) > for all b b. It can be shown that V (b) s radally unbounded,.e., V (b) when b. Therefore, the equlbrum pont b s also globally asymptotcally stable (GAS), whch mples that b(t) converges to b as t +. Now, we prove the second part of the proposton whch conssts n showng that also the dscrete-tme algorthm asymptotcally converges to the equlbrum. By decouplng (4), we get ḃ = db ( dt = b b ) v (b) (6) B

15 5 The latter result wll be useful to show that the dscrete-tme algorthm tracks the contnuous-tme system up to a bounded error that asymptotcally tends to as ncreases. A second-order Taylor expanson of () leads to b (m) b (m + ) = b (m) + γm b(m) (7) v (b(m)) + µγm B for some bounded µ. Note that µ s bounded because b v (b) s bounded by defnton. Intutvely, (7) s the dscrete verson of (6) up to a bounded error. P P < Snce, by assumpton, m γm m γm = +, results n [38] show that b (m) s an APT for (4). It stll remans to prove that b (m) b. By decouplng z and b, we b obtan z = ln B b. By rewrtng V (b) n terms of z, we obtan V (z). By consderng a Taylor expanson of V (z), we obtan: X V (z(m + )) = V (z(m)) + γm (b (m) b ) v (b (m)) + µ γm S for some bounded µ >. Snce b s GAS, t follows that B s a basn of attracton for b. Therefore, there must exst a compact set L B contanng b, where B s the strategy set of the game G (S). So, f we prove that there also exsts a large enough m such that b(m ) L, then, the proof s concluded. Assume ad absurdum that such m does not exsts. Recall that v (b)(b (m) P b ) < by defnton. Therefore, t must exst some β > such that S v (b)(b (m) b ) β for a large enough m. It follows that V (z(m + )) V (z(m)) γm β + µ γm (8) X µ γm m (9) Sergo Palazzo (M9 SM99) receved the degree n electrcal engneerng from the Unversty of Catana, Catana, Italy, n 977. Snce 987, he has been wth the Unversty of Catana, where s now a Professor of Telecommuncatons Networks. In 994, he spent the summer at the Internatonal Computer Scence Insttute (ICSI), Berkeley, as a Senor Vstor. In 3, he was at the Unversty of Canterbury, Chrstchurch, New Zealand, as a recpent of the Vstng Erskne Fellowshp. Hs current research nterests are n modellng, optmzaton, and control of wreless networks, wth applcatons to cogntve and cooperatve networkng, SDN, and sensor networks. Prof. Palazzo has been servng on the Techncal Program Commttee of INFOCOM, the IEEE Conference on Computer Communcatons, snce 99. He has been the General Char of some ACM conferences (MobHoc 6, MobOpp ), and currently s a member of the MobHoc Steerng Commttee. He has also been the TPC Co-Char of some other conferences, ncludng IFIP Networkng, IWCMC 3, and European Wreless 4. He also served on the Edtoral Board of several journals, ncludng IEEE/ACM Transactons on Networkng, IEEE Transactons on Moble Computng, IEEE Wreless Communcatons Magazne, Computer Networks, Ad Hoc Networks, and Wreless Communcatons and Moble Computng. whch yelds to V (z(m + )) V (z()) β X m γm + P P < By assumpton m γm m γm = +. Thus, (9) leads to V (z(m + )), whch s a contradcton as V (z) s lower bounded by constructon. Therefore, [38] ensures that there must exst m such that b(m ) L and lmm + b(m) = b, whch concludes the proof. Salvatore D Oro (S ) receved the B.S. degree n Computer Engneerng and the M.S. degree n Telecommuncatons Engneerng degree both at the Unversty of Catana n and, respectvely. He receved the PhD degree from the Unversty of Catana n 5. In 3 and 5, he was a Vstng Researcher at Unverst Pars-Sud, Pars, France and at Oho State Unversty, Oho, USA. He s currently a postdoctoral research fellow at the Unversty of Catana. In 5, he organzed the st Workshop on COmpettve and COoperatve Approaches for 5G networks (COCOA), and served on the Techncal Program Commttee (TPC) of the CoCoNet8 workshop at IEEE ICC 6. In 3, he served on the Techncal Program Commttee (TPC) of the th European Wreless Conference (EW4). Laura Gallucco Laura Gallucco (M) receved the Laurea Degree n electrcal engneerng n and the Ph.D. degree n electrcal, computer, and telecommuncatons engneerng n 5 from the Unversty of Catana, Italy. Snce she was wth the Italan Natonal Consortum of Telecommuncatons (CNIT), workng as a Research Fellow n the FIRB VICOM and NoE SATNEX projects. Snce, she has been an Assstant Professor wth the Unversty of Catana. In 5 she was Vstng Scholar wth the COMET Group, Columba Unversty, New York. Her research nterests nclude unconventonal communcaton networks, software defned networks, and network performance analyss. She serves n the edtoral boards of Elsever Ad Hoc Networks and Wley Wreless Communcatons and Moble Computng journals. Govann Schembra s Assocate Professor at the Unversty of Catana. From September 99 to August 99 he was wth the Telecommuncatons Research Group of the Cefrel of Mlan, workng on traffc modellng and performance evaluaton n broadband networks. He was nvolved n several natonal and EU projects. In partcular, he worked for the Unversty of Catana n the European project DOLMEN (Servce Machne Development for an Open Longterm Moble and Fxed Network Envronment), and has been actng as WP leader n the NoE Newcom. Hs research nterests manly concern wth SDNs, NFV, traffc modelng, cloud computng and data center management, moble cloud networks.