D7.1.3 General procedure to run experiments on the TEFIS platform. José Roberto (FUSP), Caio César (FUSP)

Transcription

1 Deliverable Title D7.1.3 General prcedure t run experiments n the TEFIS platfrm Deliverable Lead: FUSP Related Wrk package: WP 7 Authr(s): Jsé Rbert (FUSP), Cai César (FUSP) Disseminatin level: Public Due submissin date: 31/05/2011 Actual submissin: 31/05/2011 Prject Number Instrument: IP Start date f Prject: 01/06/2010 Duratin: 30 mnths Prject crdinatr: THALES Abstract This dcument cver the steps t any experimenter submit, run, test and cllect the results f a generic experiment ver the TEFIS platfrm. Page 1 f 41

2 Prject funded by the Eurpean Cmmissin under the 7th Eurpean Framewrk Prgramme fr RTD - ICT theme f the Cperatin Prgramme. License This wrk is licensed under the Creative Cmmns Attributin-Share Alike 3.0 License. T view a cpy f this license, visit r send a letter t Creative Cmmns, 171 Secnd Street, Suite 300, San Francisc, Califrnia, 94105, USA. Prject c-funded by the Eurpean Cmmissin within the Seventh Framewrk Prgramme ( ) Page 2 f 41

3 Versining and Cntributin Histry Versin Date Mdificatin reasn Mdified by /04/2011 TC Jsé Rbert(FUSP) /05/2011 Dcument Sectins Jsé Rbert(FUSP), Cai César(FUSP) /05/2011 Prtal Figures Additin Jsé Rbert (FUSP), Cai César(FUSP) /05/2011 Send t partners review Jsé Rbert (FUSP) /05/2011 Reviewer Cmments Brian Pickering (ITI) /05/2011 Reviewer Cmments Ainha Gracia (SQS) /05/2011 Final versin Jsé Rbert (FUSP), Cai César (FUSP) Page 3 f 41

4 Table f Cntents Executive Summary Intrductin Abut the TEFIS platfrm Experiment Facilities Abut the Experiment General Architecture Overview The TEFIS Prtal the TEFIS Middleware (Back-end cmpnents and Cre Services) TEFIS Cnnectrs TEFIS User Tls Scenari Definitin Experiment Overview Hypthesis Definitin Experimental Methd and Prcedure Design f the Experiment Variables Metrics and Measurements Hardware and Sftware requirements General Overview Scheme Steps Experiment Implementatin Lgging int the prtal Specificatin f the experiment Request Prvisin / Deply Experiment Executin Run phase Cllect and Evaluate phases Acrnyms References Page 4 f 41

5 INDEX OF FIGURES Figure 1: High Level Wrkflw Figure 2: TEFIS Functinal Architecture Figure 3: e-travel use-case use f the TEFIS facilities Figure 4: Scheme f a generic experiment n tp f testbeds and a hardware/sftware relatinship Figure 5: TEFIS prtal - main page Figure 6: TEFIS Prtal - register page Figure 7: TEFIS Prtal - lgin page Figure 8: Creatin f a new experiment n the Experiment Design Flw Figure 9: Creating r searching fr an experiment Figure 10: Definitin f a new experiment Figure 11: List f existing experiments available Figure 12: TEFIS Prtal - resurces page Figure 13: Defining the experiment name and descriptin; and chsing test types, dmains and keywrds related t the testbeds used by the experiment Figure 14: Cnfiguratin f an experiment already defined Figure 15: Cnfiguratin f a generic experiment Figure 16: Arrangement f taskrder Figure 17: Insertin f the jb files int a task during experiment cnfiguratin Figure 18: Platfrm allwing fr the creatin f testruns fr already cnfigured experiments Figure 19: Creatin f a testrun fr new experiments Figure 20: List f created testruns Figure 21: Testrun cnfiguratin page with rdered tasks Figure 22: Executin f a testrun n the TEFIS prtal Page 5 f 41

6 INDEX OF TABLES Table 1: A generic experiment specified in simple terms and an example invlving an end-user trial f a multimedia scial netwrking applicatin... 7 Table 2: Descriptin f steps fr a generic experiment running n the TEFIS platfrm Page 6 f 41

7 Executive Summary The main bjective f Deliverable D7.1.3 General prcedure t run experiments n the TEFIS platfrm is t create a definitin f the three main activities necessary t run any generic experiment: the Scenari Definitin, the Design f the Experiment, and the Implementatin and Executin f the Experiment. Using the descriptin f aims and intentins f a Scenari, any experiment can be specified in simple terms. This infrmatin can be translated int the fllwing categries: Summary; Hypthesis definitin; Experimental methd and prcedure; Actrs invlved; Variables; Metrics and Measurements; Technical Requirements; In summary this with an example experiment invlving an end-user trial f a multi-media scial netwrking applicatin: Table 1: A generic experiment specified in simple terms and an example invlving an end-user trial f a multimedia scial netwrking applicatin Item Descriptin An Example Experiment Summary Tell us what yur experiment is abut and what yu are trying t achieve. Testing the latency in sharing multimedia files fr a scial netwrking applicatin running in a highly-distributed envirnment. Hypthesis definitin Experimental methd and prcedure What is the basic questin yu are trying t investigate? Tell us what yu are trying t prve (r disprve!) Tell us what steps are invlved in running yur experiment. Will latencies be less than 500 msec t transfer multi-media files irrespective f the number f users? A secndary issue is whether a 20 t 100 applicatin-server-t-data-server rati wuld be sufficient t supprt the prjected traffic. The intentin is t use a grup f real users wh will register with the service, take phts, anntate with audi, and then pst t a central site. In the secnd stage, users will brwse all cntent arbitrarily and dwnlad any files they wish t. They will be limited t 5 dwnlads in a given perid. Page 7 f 41

8 Actrs invlved Variables Metrics and measurements Technical requirements Tell us wh and hw many peple yu will need as end users. Tell us any values yu are trying t test and any values yu will be keeping cnstant. Tell us hw yu want t quantify the data frm the experiment. Tell us abut the envirnment and technical needs f yur experiment. 500 real users, with multi-media mbile phne devices, frm the age grup years. The verall latency f (a) uplading files (b) dwnlading files (c) brwsing files as a functin f the number f users and the sizes f the files. We will nly try t regulate the number f users and their usage. We will als investigate specific details abut the technical infrastructure such as cnnectin speed and bandwidth. We wuld like t measure the elapsed time in ms between request receipt at the server end until the whle file has been received and stred at the persnal device end. We will need t have access t 500 end-users, cnnected t and using the service. We expect the service t be installed and running n at least 20 servers and a bank f 100 servers each with apprximately 50 TB f strage, and with wireless access frm the user t the server and quasirealtime retrieval f data frm the data servers t the applicatin server. User devices shuld be f the Apple family. Page 8 f 41

9 1 Intrductin This dcument describes hw t run a generic experiment n the TEFIS Platfrm, aiming at general prcedures based n experience gained frm the experiments used in the early stages f platfrm develpment. It describes a walkthrugh t guide the experimenter in planning the experiment frm scratch thrugh t the cllectin f the results, via the design in the prtal and the deplyment f the applicatin n the platfrm. In this first sectin f this dcument the TEFIS platfrm will be intrduced and the testbeds available initially fr use by the experimenter and in the fllwing sectins the steps t run a successful experiment will be described in detail. 1.1 ABOUT THE TEFIS PLATFORM The TEFIS prject is develping an pen platfrm t access hetergeneus and cmplementary experiment facilities fr cmmunities f sftware/business develpers t test, experiment and cllabratively create knwledge. It prvides apprpriate tls and methdlgies t address the full develpment life-cycle f innvative services. The main feature f the platfrm is the single pint f access t a large number f varied testbed resurces, s the experimenter des nt need t g t different places fr different parts f his experiment. Instead he has access t all experimental facilities in a single prtal, where he can plan the experiment, run all necessary tests - even when a single experiment invlves separate stages run n different testbeds - and cllect the results. 1.2 EXPERIMENT FACILITIES Hetergeneus and cmplementary facilities are available in TEFIS fr: netwrk perfrmance experiments, sftware testing, and user-riented living labs. The initial Experimental Facilities (testbeds) that the experimenter will able t use in the TEFIS platfrm are: Planetlab: glbal research netwrk that supprts the develpment f new netwrk services, with a cmmunity f graduate students, researchers and industrial partners. One f its main purpses is t serve as a testbed fr verlay netwrks in which researchers can run experiments n a variety f glbal services (file sharing, ruting and multicasting verlays, scalable bject lcatin, etc) with the advantage f being able t run experiments under real-wrld cnditins and n a large scale. This testbed is als designed t supprt lng-running services that supprt a client base. PACA Grid: is a set f machines accessible via Graphical Interactive interfaces based n the PrActive Parallel Suite. This Clud aggregates dedicated machines, bth Linux and Windws, and spare desktp machines, dynamically added during the night and at week-ends. The facility mainly targets the scientific and industrial cmmunities that need t speed-up scientific simulatins, parallelize and distribute large scale applicatins, accelerate financial cmputatins and s n. Page 9 f 41

10 ETICS: e-infrastructure fr Testing, Integratin and Cnfiguratin f Sftware is a cllabratin prject crdinated by CERN and funded in part by the Eurpean Cmmissin. The gals f ETICS are t prvide state-f-the-art pen-surce tls, prcedures and resurces t manage the sftware prductin lifecycle fr research prjects develping grids and ther distributed sftware and t prmte the establishment f standard quality assurance methds and prcedures within the scientific research cmmunities t imprve the verall quality, stability and reliability f the grid-based research infrastructures. ETICS prvides a number f services including a cnfiguratin service, a distributed multi-platfrm test bed, an nline repsitry f packages, build and test reprts and metrics, and an ISO-cmpliant quality assessment mdel called A-QCM (Autmated Quality Certificatin Mdel). SQS IMS: this testbed is ffered as an infrastructure t validate and test applicatins ver IMS. The SQS testbed ffers bth the emulated IMS platfrm, and cnnectin t real envirnments (IMS, SS7) t allw tasks t be run and tested; als end-t-end validatin and testing services is prvided, such as functinal validatin f the system under test, netwrk perfrmance mnitring, measurement f end t end netwrk QS and interperability, fr the Telecmmunicatins Sectr. Btnia Living Lab: is the first and largest pen Living Lab fr human-centric ICT develpment. The basic idea is t engage end-users, individuals and stakehlder rganisatins, alng a targeted value chain, in the ttal prcess frm need-finding and idea-generatin, thrugh cnceptdevelpment and prttype/usability testing t pilt service validatin f market and marketing principles. KyaTera: is a testbed that cnnects academic institutins, institutes f research and funding agencies in an envirnment fr cllabrative wrk, based n a Fiber-t-the-Lab netwrk, dedicated t the study and develpment f science, technlgies and Future Internet applicatins. The Kyatera netwrk perfrmance meter will be created t measure netwrk quality, targeting the prblem f guaranteeing QS n multimedia data transmissin. 1.3 ABOUT THE EXPERIMENT The TEFIS platfrm aims t help identify experimental needs and frmalize planned experiments, the ntin f high-level wrkflw was presented t facilitate users running an experiment. The wrkflw is illustrated in the figure 1: Page 10 f 41

11 Figure 1: High Level Wrkflw The TEFIS prtal fllws these stages giving the user access t the related functinalities supprted by the back end and the testbed facilities. Thse stages are as fllw: Plan: is the first phase f the high level flw in which the user is respnsible fr defining the type f experiment, experimental restrictins, which testbeds are required and glbal infrmatin related t the experiment. In summary, this stage is used fr: Page 11 f 41

12 Expressin f needs Identificatin f testbeds, KPIs, tls Planning & methdlgy High-level Wrkflw frmulatin Request: is the phase in which the testbeds chsen are cntacted, their checked availability discvered, and if necessary, request reservatin. In this phase, a time plan fr the experiment is als prduced. In summary, this stage is used fr: Access requests/grants Resurce bking Prvisin / Deply: is the stage respnsible fr the cnfiguratin f the required resurces and deplyment f data and user applicatins Thse stages are gruped here because frm a user perspective, there is little interventin required, and s the transitin frm ne stage t the next is largely transparent. In summary, this stage is used fr: Test specificatin (executable wrkflws) Cnfiguratin Deplyment Run: is the realisatin f the definitin, planning and preparatin described in the first three stages. The utputs are the results f the user applicatin and perfrmance results f executin (KPIs measured during executin). In summary, this stage is used fr: Executin Testing (tls, Users) Data cllectin Evaluate: is the phase where results cllected frm the run stage are analyzed and used t verify final experiment results. Frm this pint, the user may chse t run anther experiment r change cnfiguratins if necessary. In summary, this stage is used fr: Experimental results analysis Experimental refinements Cllate: is the stage in which final results and cnclusins are prduced fr the experimenter. The results can als be published in a knwledge base, s future experimental cycles are enhanced. In summary, this stage is used fr: Access knwledge base Cmmunity disseminatin Page 12 f 41

13 2 General Architecture Overview Befre the experimenter starts the definitin and the design f the experiment, it is imprtant t knw abut the TEFIS architecture. A mre detailed explanatin can be fund in the dcument D Here the experimenter can find sme f the basic principles f the platfrm architecture. Figure 2: TEFIS Functinal Architecture The TEFIS framewrk is rganized int fur functinal blcks: the TEFIS Prtal, the TEFIS Middleware (including back-end cmpnents and cre services), TEFIS users tls, and the TEFIS testbed cnnectrs. 2.1 THE TEFIS PORTAL The TEFIS prtal will be the single access pint t specify the experiments, s they can be executed remtely n the testbeds. This prtal will allw testbeds t be searched fr, the retrieval f reference data, cnfiguratin f tests t be perfrmed, gathering f diagnstic parameters and mre. It will als prvide access t a wide range f tls. This functinal blck is divided int fur different interfaces: Page 13 f 41

14 Resurces Directry The directry prvides the repsitry f tls, hardware, sftware, dcumentatin and the list f experiment facilities (testbeds) and its resurces, being a fundamental infrastructure supprting the experiment lifecycle. The presentatin layer will be implemented in the TEFIS Prtal, while the actual directry implementatin is a backend cmpnent within the TEFIS Middleware. Identity Management This interface is respnsible fr creating and managing accunts and access plicies fr TEFIS users, prtecting the platfrm with simple and effective accunt management techniques. This system resides in the TEFIS middleware layer, while the TEFIS prtal implements the user interface. Experiment Manager This interface will wrk with the TEFIS resurces directry t list and cnfigure resurces in the testbeds, besides planning the experiment executin, allwing it t be executed n a cmbinatin f experiment facilities. TEFIS users will be given supprt in defining, executing and cllecting the required data in their experiment. The interfaces in this system will be adjusted t the needs f each user. Fr the end-user (experimenter/researcher) specifically, this will be implemented using these subcmpnents: Experiment designer This will help users t frmalize their requiriments and pint ut the crrect tls, testbed and methdlgy fr that experiment. In this step the inputs are the user needs and the specificatin and design f the experiment, while the platfrm utputs are the test strategy and a list with the types f tests t perfrm (functinal, integratin, etc.) Experiment planner Once the user has specified the experiment, cntrl switches t this interface in rder t rerganize the knwledge prvided by that user and link his needs t existing resurces via the cre services. In this step the user inputs are the test strategy and the user specific criteria fr it, while the platfrm utput is the test plan Experiment wrkflw manager This step will cmplete the wrkflw prcess, based n the specificatin f the characteristics f the experiment prvided by the user. It will send a request t the cre servicesand btain the cmplete list f activities and schedules. In this step the user input is the test plan, while the platfrm utputs are the partial results f the executin f the activities and an activities wrkflw Cnfiguratin assistant The Cnfiguratin Assistant is respnsible fr guiding the user in their experiment definitin, rganizatin and mnitring tasks. The mre specific the descriptin f the Page 14 f 41

15 experiment, the mre accurate will be the resurces, facilities, tls and data prvided t the user Experimental Data Interface This interface is respnsible fr prcessing data presented in the TEFIS platfrm. There are tw main types f data t distinguish: 1. Static data: data used t identify the user and establish cntractual infrmatin between user and test prvider, as well as the definitin f the experiment (cnfiguratin and applicatin data) 2. Dynamic data: data created during and after the experiment run (perfrmance, user data, cnslidated results and reprts) With nly tw types f data there are still distinctins t be made. The static data will be held lcally, since it has a small and tractable ftprint, and des nt require extensive strage. Meanwhile, the dynamic data will typically be remtely stred and accessed via a TIDS and it will require significant resurce t stre and manage, since this data will be large and ptentially pen-ended. The fllwing items identify the service prvided by this interface: Registratin Users can identify data and infrmatin assciated with a specific experiment r testrun Experiment Request User lks fr related experiments User gives details abut a new experiment they have run Experiment Mnitring User wants t knw hw the experiment is prgressing User wants t decide whether t restart the experiment User wants t evaluate the experiment utput Experiment Publicatin User wants t retrieve archived data User wants t send data elsewhere 2.2 THE TEFIS MIDDLEWARE (BACK-END COMPONENTS AND CORE SERVICES) The main gals f the TEFIS Cre Services are: Execute lw level wrkflws Since the TEFIS platfrm aims t prvide a cmmn framewrk t a set f several hetergeneus testbeds, it must be cnsidered that each experiment (r kind f experiment) will differ frm any Page 15 f 41

16 ther, s that sme f the steps in the high level wrkflw have t be made mre specific fr the lwer level wrkflw Mnitr resurce status, matching and identificatin Prvide infrmatin abut the current status f a given prcess; as well as prviding infrmatin n perfrmance. This will help future experiment prvisining. 2.3 TEFIS CONNECTORS TEFIS is expected t be dynamic (supprt testbeds jining and leaving) and hetergeneus (ptentially any type f testbed can jin). T manage thse interactins (frm an architectural pint f view), the system relies n the cncept f a cnnectr. The rle f cnnectrs in the TEFIS architecture is t handle the hetergeneity presented by the different testbeds, building an abstractin layer abve the testbed interface, ffering a cmmn interface t all testbeds in the TEFIS platfrm. A hmgeneus and unifrm prtcl is defined, in rder t ease cmmunicatin between the TEFIS platfrm and the testbeds. Due t their lcatin n the periphery f the infrastructure and because they are in charge f cmmunicatin with external cmpnents, the cnnectrs are als the central elements in ne aspect f security management. 2.4 TEFIS USER TOOLS The TEFIS user tls represent a lng term visin, nce the platfrm is peratinal, making it able t hst external tls n the platfrm, in the same way that new testbeds are integrated. 3 Scenari Definitin Befre running the experiment ver the TEFIS platfrm the user needs t fllw a few steps t create a gd plan fr their experiment. This planning descriptin starts with the Scenari Definitin, since a gd experiment needs a gd basis. Page 16 f 41

17 Accrding t the experiment test life-cycle, this and the next sectin crrespnd t the plan phase in the life-cycle wrkflw. The Scenari Definitin is divided int three phases: the experiment verview, the hypthesis definitin and the experimental methd and prcedure, as described belw. 3.1 EXPERIMENT OVERVIEW Befre the experimenter starts the frmal descriptin f his experiment, he can create a quick descriptin f what he is planning t d, making an abstract, t intrduce the experiment and its scenari. This stage can help the experimenter himself t think abut the experiment in a high-level view, s he can wrk ver this descriptin t g int detail abut the experiment, extracting the variables, metrics, and thers features that are necessary. Finally this sectin can help ther peple t understand the experiment. If a secnd persn needs t use r cntinue the experiment, here will be the first step s he understands it. 3.2 HYPOTHESIS DEFINITION When the experimenter starts the experiment specificatin, the first thing t cnsider is the scenari hypthesis, in which the experimenter will think and prvide a frmal definitin f the prpsed scenari hypthesis, including classes being studied, variables and expected causal relatinships. Infrmatin shuld als be prvided n hw the hypthesis was derived. This way the experimenter will have a frmal descriptin f what he intends t d, as well as the scenari backgrund t this experiment, and thereby describing the mtivatin fr this experiment. T facilitate later verificatin f what hyptheses are cvered, it is recmmended that the experimenter make a bulleted list, with a shrt resumé f each hypthesis t be tested. Making the bulleted list desn t mean that a mre extensive descriptin can t be dne. It is highly recmmended that the experimenter creates an explanatin f why he intends t verify this hypthesis, and why it is imprtant t validate the experiment. 3.3 EXPERIMENTAL METHOD AND PROCEDURE After the hyptheses are defined, the experimenter needs t fcus n hw he shuld prceed in rder t validate this hypthesis list. In this sectin he shuld describe the experimental Page 17 f 41

18 prcedure chsen fr testing the hypthesis, prviding a justificatin fr why it was selected. This will include hw independent variables are explred and results analysed. Cnsidering the testbeds as the principal way that the TEFIS platfrm ffers the experimenter the means t test and validate his experiment, here he will fcus n chsing the best testbed fr each hypthesis. S the best practice t d in this sectin is: View all testbeds that the TEFIS platfrm ffers, studying their features; List all testbeds that have features that will help validate and test all the hypthesis previusly defined; Create a shrt descriptin f hw each testbed will be used in the experiment, and why this test was chsen fr that hypthesis; Planning hw the testbeds will be used, i.e. which is the first testbed, if sme test can be dne in parallel, if any test will use the results f any previus test, etc; Create a scheme r a wrkflw t better visualize the interactin with the testbed. Figure 3 is an example f the wrkflw chart based n the e-travel case. A mre detailed explanatin can be fund in D2.1.1, sectin 4.2. Figure 3: e-travel use-case use f the TEFIS facilities 4 Design f the Experiment Page 18 f 41

19 After the experimenter has defined the scenari, i.e. created a backgrund t the experiment, defined all the hyptheses, chsen the testbeds t perfrm the tests, he needs t g t a lwer level in the experiment t extract variables, metrics, hardware, sftware and finally think abut the steps t run the experiment n the platfrm. In this sectin the experimenter needs t describe the verall bjectives f the experiment design, fcusing n the specific innvatin and technlgy that the experiment will need t prvide. The infrmatin extracted frm this sectin will g t fill sme f the fields n the prtal interface, when the experimenter needs t describe their experiment. It s gd practice t keep all this infrmatin dcumented in rder t easily extract it later. 4.1 VARIABLES In this sectin the experimenter shuld identify all variables in the experiment that he will want t measure, calculate r test. T assist the experimenter s wrk, he shuld aim t answer the fllwing questins: What d I change? (independent variable list) What d I bserve? (dependent variable list) What d I keep the same? (cntrlled variable list) What irrelevant variables might mediate the effect f the independent n the dependent variables? (extraneus variable list) The experimenter shuld understand that an experiment need nt present all types f variable, e.g.: an experiment may have nly independent variable. As an example, sme types f variables are listed belw: Independent variable: Number f users in the system Dependent variable: Ttal bandwidth used by all users in the system Cntrlled variable: Max number f users in the system Extraneus variable: Pwer line jitter 4.2 METRICS AND MEASUREMENTS Nw that the experimenter has all variables listed, he can prvide all metrics and measurements related t each variable. T d this, he shuld aim t prvide answers t the fllwing questins: What is the feature f interest? Name f metric: Hw is it measured? Page 19 f 41

20 Hw ften is the measurement made? Is it a raw measurement r pre-prcessed? What are the units f measurement? Is there any uncertainty in the measurement? As an example we describe the measure f time t transmit a frm t the database: Every time a frm is sent t the database the platfrm will measure the time, in millisecnds, that it takes t arrive. This measurement will help establish the system respnse time, in case f a frm sending. 4.3 HARDWARE AND SOFTWARE REQUIREMENTS In this step the experimenter will describe all hardware and sftware besides the TEFIS platfrm that he will need t perfrm his experiment. Smetimes the experiment needs an extra device, like a smart-phne r anther hand-held device t run an applicatin, r an external data server, cntaining a massive amunt f data t be prcessed by anther physical entity befre entering the TEFIS platfrm. Alng with such hardware items, additinal sftware may be necessary beynd what TEFIS prvides r which needs t be integrated with the platfrm t perfrm the tests. In this sectin, thinking f best practices, the experimenter can first list all the requirements related t the hardware and sftware, and describe their significance t the experiment. e.g.: An applicatin server t hst the applicatin web-services: Tmcat versin 6.0 A mbile device t run the data cllectin sftware, fr instance, a smart-phne with the capability t cnnect t the web (3G, 4G r any ther wireless Internet cnnectin) and a GPS system, t run the experiment applicatin. 4.4 GENERAL OVERVIEW SCHEME At this pint f the experiment design, the experimenter has a gd idea f what he needs in terms f sftware, hardware, f all the variables t be measured and f hw he will achieve it, alng with all the testbeds he will use and the sequence by which t run the experiment. Thereby, fcusing n best practices, the experimenter may create a scheme crrelating all the knwledge gathered until nw, t facilitate the creatin f the next and final stage f the design. Generally this scheme will cntain the testbeds in the backgrund, and n tp f that the sftware, hardware and the data flw inside the experiment and the applicatin. Page 20 f 41

21 A sample scheme is shw in figure 4, using three generic testbeds, running a generic experimental applicatin, which will generate randm data, pass n t an applicatin running ver a cellphne, sending the data ver a netwrk t a webservice, where the data will be prcessed. Figure 4: Scheme f a generic experiment n tp f testbeds and a hardware/sftware relatinship 4.5 STEPS The final stage f the experiment design is planning the steps fr running the experiment thrugh the TEFIS prtal. Nw the experimenter has all the infrmatin required t create the steps, as he is already aware f the platfrm architecture, frm the infrmatin available in the dcument r in the brief explanatin in sectin 2 f this dcument, and he knws all abut his experiment, as planned in the previus sectins. S the experimenter will be able t write the steps t run the experiment. Page 21 f 41

22 The steps are brief descriptins f what he will d, sequentially, in each part f the prtal, including the inputs and utputs expected in the platfrm. T easily implement the steps, the experimenter can use the scheme created in the preview sectin tgether with the platfrm architecture, that is mainly the user interface part f the platfrm (prtal). In table 2, the steps fr a generic experiment are listed t help him with his wn experiment. Table 2: Descriptin f steps fr a generic experiment running n the TEFIS platfrm Step Summary Descriptin 1 Sftware develper creates a TEFIS accunt 2 Specificatin f the experiment by the user via the prtal 3 Create descriptin f the desired SLA, stressing factrs (number f resurces and input lad) All the infrmatin required fr the accunt are prvided Once an initial experiment has been described in TEFIS and a test prfile created, then fr subsequent test runs, the user simply edits what is already held in the TEFIS Experiments Data Manager 4 Stre experiment descriptin in the TEFIS platfrm 5 Selectin f the mst suitable testbed fr the experiment 6 Initial cnfiguratin f the chsen testbed is dne thrugh TEFIS; any ther cnfiguratin activities are dne using the WS-API; reprts are fed back fr presentatin via the prtal t the user Chice made based n hardware requirements and the metrics wanted This is related t the TEFIS wrkflw definitin, testbed cnfiguratins and then the actual submissin f the experiment via the TEFIS platfrm. The user needs t define the set f resurces first that they wuld like t use frm amng the available testbeds, then the rder/cnditins in which they are t be used and finally cnfigure the testbeds directly frm their TEFIS wrkspace. 7 Prvide Applicatin and sftware 1. The user prvides the cnfiguratin which 8 Prepare a wrkflw t cnduct the test (tasks and number f ndes) cntains all dependencies required t build the applicatin. Either the user specifies where t get them r they are upladed directly, if is the case. 9 Send a jb, which will be executed when resurces becme available 2. Depending n the result f the build and test suite prcesses, TEFIS deplys the applicatin t the testbed. The cnfiguratin f the applicatin is very dynamic, hence these cnfiguratin files need t be Page 22 f 41

23 tagged t shw that they will need mdificatin fr a successful deplyment; and relevant pieces f infrmatin need t be updated nce they becme available (like the machine hsting the given services, fr instance) 10 Prepare Virtual machines (install required sftware: database, libraries, and s frth.) 11 Deply the cde t the ndes f the selected testbed(s) 12 Execute the tests 13 Cllect the data manually r directly frm directly frm the mnitring facility f the testbed in use, if direct access t the testbed is available. Otherwise data will be requested via the TEFIS supervisin manager 14 The user is ntified f the end f the experiment These steps wuld be TEFIS internal; the user des nt expect t have t cnfigure things manually whilst the applicatin is being deplyed r has already been deplyed. Any required sftware, cnfiguratin files and s frth need t be dynamically mdified and prvisined n the target ndes The user is ntified that sme metrics are available. Depending n the results, he may then decide interactively t run step 2 f the experiment again r exit the experiment. In the case where the applicatin is t be run again, the experiment flw ges back t steps 6 t 9 where the input fr the experiment may be mdified and the values t be cllected refined 15 Experimenter can decide t re-run the experiment depending n the data cllected. He can launch the test again and decide t include new wrker ndes t the experiment, change experiment input, and refine the data set t be cllected. At this pint, he can als exit the experiment. 16 The experiment restarts frm step 9 Page 23 f 41

24 5 Experiment Implementatin At this pint the experimenter will start implementin, passing thrugh the steps described in the preview sectin, fcusing n the first steps, ging frm accunt creatin t sftware deplyment. The life-cycle f the experiment is cmpleted by the implementatin sectin, by the executin f the request and by the prvisin/deply phases. As mst f the steps are dne in the prtal, the experimenter may refer t D3.1.1 [2] fr a mre cmplete verview f the prtal. 5.1 LOGGING INTO THE PORTAL The first thing t d in the TEFIS prtal is t lg in, s the user gains access t the platfrm and is able t design and run the experiment. If the user desn t have a prtal accunt, he must create ne t jin the platfrm. This is dne n the initial page f TEFIS platfrm, and it is shwn in the figure 5. Figure 5: TEFIS prtal - main page Page 24 f 41

25 Figure 6: TEFIS Prtal - register page T register, the user simply cmpletes all fields in the frm and presses the submit buttn, as shwn in the figure 6 t register int TEFIS platfrm. Page 25 f 41

26 Figure 7: TEFIS Prtal - lgin page Once the user has created his accunt and lgged int the prtal, he will have access t the dashbard areas (PTMs, resurces and experiments). 5.2 SPECIFICATION OF THE EXPERIMENT Lgged int the prtal, the user enters the experiment area in the dashbard, and will be able t specify the experiment. The user can create the descriptin f the desired SLA, stressing factrs (number f resurces and input lad), variables that are t be measured r tested and the relative KPIs. He will then stre his experiment descriptin n the platfrm and save its specificatin. In the experiment interface the user will use the Experiment Design Flw, in which he will select the phases t create the experiment. In this first step, the user will enter in the Select r create an experiment just by clicking the link, as shwn in the Figure 8. Page 26 f 41

27 Figure 8: Creatin f a new experiment n the Experiment Design Flw Once in this sectin, the user is presented with a screen in which he can search fr r select an existing experiment, r create a new ne by clicking n the Advanced experiment cnfiguratin link. Figure 9 shws the screen in questin. Figure 9: Creating r searching fr an experiment At this pint, if the experimenter chses t create a new experiment, the prtal will g back t the Experiment Design Flw and the user will be able t prceed t the Experiment Definitin step, as shwn in figure 10. Page 27 f 41

28 Figure 10: Definitin f a new experiment With this step, the experimenter will use the Advanced Experiment Definitin, where he can chse a name and a descriptin fr the experiment. Once an initial experiment has been described in TEFIS and a test prfile created, then fr subsequent test runs, the user simply edits what is already held in the TEFIS data stre, i.e. he desn t need t describe r specify the experiment every time he wants t perfrm a test. If the experimenter has spent sme time t dcument the experiment, as shwn in the previus sectins, this part f the implementatin will nly be a cpy f what he has already dne. The user can select the previus experiment, entering n the select r create an experiment in the Experiment Design Flw Select an existing experiment. After this, the user will reach the fllwing page shwn in the Figure 11. Page 28 f 41

29 Figure 11: List f existing experiments available T pen an experiment, just select the ne desired and hit the Lad link. 5.3 REQUEST In this phase the experimenter will fcus n the testbeds, mainly n gaining access t the resurces available. A list f available resurces and testbeds, at the time f the experiment, can be fund in the prtal, under the resurces page, which is accessible after lgin t the prtal. If the experimenter is fllwing this dcument, at this pint he has already chsen the resurces and the testbeds t be used in the experiment, s he can check n this page if the resurce is available at the mment f the experiment. In the main prtal, the user can view a list f the available resurces that can help the experimenter t chse the crrect testbed fr his experiment. Page 29 f 41

30 Figure 12: TEFIS Prtal - resurces page All the steps f this phase will be dne essentially in the TEFIS prtal, s the user desn t need t interact directly with the testbeds. Hwever, sme testbeds used in the experiment need t be cntacted t verify availability and t perfrm any reservatin if needed, thugh fr the mst part the TEFIS platfrm will handle this. Cases d exist, due t the individual testbed s plicy, in which the experimenter will need t make this cntact himself. In the same lcatin where the user creates the name and descriptin f the experiment, he needs t chse the Test Types, the Dmains and the Keywrds, all related t the testbeds that the experiment will use. Figure 13 shws this page, in which the user can select multiple Test Types, Dmains and Keywrds fr the experiment, by clicking then n the Save this resurce list. After that the prtal will return t the Experiment Design Flw page. Page 30 f 41

31 Figure 13: Defining the experiment name and descriptin; and chsing test types, dmains and keywrds related t the testbeds used by the experiment After all testbeds have been selected, the user will need t cnfigure them. The initial cnfiguratin will be dne directly in the TEFIS prtal, but any ther cnfiguratin activities may be dne using the WS-API, prvided by the testbed. This will prduce reprts cntaining feedback fr presentatin t the user at the prtal. After the prtal returns t the Experiment Design Flw page, the user will ntice that he may nt select the definitin again, since the experiment has already been defined. Hwever the user can cnfigure his experiment using the Experiment Cnfiguratin step, shwn in the Figure 14. Page 31 f 41

32 Figure 14: Cnfiguratin f an experiment already defined Inside the Experiment Cnfiguratin page, the user will find the Available Resurce Types, in which the experimenter can drag and drp the resurces needed, change the prperties f each resurce cntainer and, after everything has cnfigured, click n the Save experiment link. Figure 15 shws a generic experiment cnfiguratin, where the prtal shws the pp-up bx t alert t the Save prcess. It can be seen in this figure, that the experimenter can change the clur f each resurce t help visualize the flw, and he can add tasks t help in the experiment wrkflw, discussed later in this dcument. Page 32 f 41

33 Figure 15: Cnfiguratin f a generic experiment The next step is t prepare the wrkflw t cnduct the test. Fr this, the user can rearrange the tasks in the Order tasks link under the Experiment Cnfiguratin. Figure 16: Arrangement f taskrder T arrange the tasks the user can just drag and drp the ne selected t the desired place. Once the testrun is dne, the experimenter cannt change the wrkflw. T d this, it is necessary t delete the lder testrun, g back t this phase and red the task rder. Page 33 f 41

34 After wrkflw preparatin, the user will need t send a jb t the prtal, which will be executed as planned in the wrkflw and when resurces becme available, in accrdance with the testbed cntract fr this experiment. The user needs t g back t the Experiment Cnfiguratin page, select a task and insert the jb file in the cnfiguratin windw as shw in the Figure 17 Figure 17: Insertin f the jb files int a task during experiment cnfiguratin After sending the jb t the prtal, the user will then prepare Virtual machines, i.e. he will install the required sftware t run the experiment: database, libraries, ther third-party sftware etc. Finalizing the cnfiguratin, the request phase f the life-cycle is dne. But the experimenter can cme back t this phase t chse thers testbeds, resurces r d anther cnfiguratin after he has cllected the results f the test. Whenever back at this phase, all previus data are stred in the TEFIS database, s the user desn t need t cnfigure them again. He can simply edit the data r start ver again. As a reminder nce the experiment is described, the user can t red it; he will need t create a new experiment frm the first step, in the event that he has missed a resurce type r wants t change the experiment type. Page 34 f 41

35 5.4 PROVISION / DEPLOY After the request phase, with the testbeds prperly selected and cnfigured, the user will start the prvisin and deply phase. In the prvisin phase, the user prvides the cnfiguratin which cntains all the dependencies required t build the applicatin, including any third-party sftware that the experiment needs. Either the user specifies where t get them r they are upladed directly int the prtal. Depending n the result f the build and test suite prcesses, the TEFIS platfrm will deply the applicatin t the crrect testbed. The cnfiguratin f the applicatin is very dynamic. In cncequence, the cnfiguratin files must be identifiable as needing mdificatin t make the deplyment succeed; and relevant pieces f infrmatin will need t be updated nce they becme available, i.e. the machine hsting the given services. All this is dne in the Test Run phase f the Experiment Design Flw. After the experiment cnfiguratin has been dne, the prtal will turn t the test run task, and the user will be given access t this phase as shwn in the Figure 18 Figure 18: Platfrm allwing fr the creatin f testruns fr already cnfigured experiments Entering this task, if a new experiment, the TEFIS prtal will ask the experimenter t create a new test run, as shwn in Figure 19. Page 35 f 41

36 Figure 19: Creatin f a testrun fr new experiments After the user cnfirms the creatin f a new testrun clicking in the OK buttn, the system will shw the prcess status and when it s dne, the prtal will shw the testruns list as can be seen in Figure 20. Figure 20: List f created testruns If this experiment has been laded, the prtal will directly shw the testrun list with all the testruns that have been previusly saved. T cntinue the experimenter the testrun just needs t be selected and click n the lad buttn, and the testrun cnfiguratin page will pen. Figure 21 shws the testrun Cnfiguratin page, with the rdered tasks. Page 36 f 41

37 Figure 21: Testrun cnfiguratin page with rdered tasks Nw that all has been set up in the prtal, the last thing t be dne befre the test battery is started is t deply the experiment cde t the ndes f the selected testbed, and the experiment is ready t be tested by the platfrm. It shuld be remembered that these final steps are internal t TEFIS; the user is nt expected t cnfigure things manually whilst the applicatin is being deplyed r has already been deplyed. Any required sftware, cnfiguratin files and s frth need t be dynamically mdified and prvisined n the target ndes. Page 37 f 41

38 6 Experiment Executin 6.1 RUN PHASE With everything having been set up in the prtal, the experimenter can finally run the tests. Fr that the TEFIS platfrm just needs t have the range f pssible actins initiated, depending n the type f testbed and the type f experiment. In TEFIS there are three appraches t execute actins n testbeds: 1. Actins are perfrmed manually by the experimenter n the testbed 2. A wrkflw is executed by the TEFIS Resurce Manager 3. The TEFIS system explits, if available, the testbed s executin engine t submit a specific wrkflw fr executin, using the testbed cnnectr, which is respnsible fr wrapping the executin engine in an abstract and general interface usable within the TEFIS system. Figure 22: Executin f a testrun n the TEFIS prtal Page 38 f 41

39 6.2 COLLECT AND EVALUATE PHASES As the platfrm finishes the tests the experimenter can cllect the data manually r directly frm the mnitring facility f the testbed in use, if direct access t the testbed is available. Otherwise data will be requested via the TEFIS Supervisin Manager. Once all the test infrmatin has been cllected, the user will be ntified abut the end f the experiment and nw he has three ptins: - He can simply decide that the infrmatin available at this pint is enugh and finish the tests. - He can decide just t re-run the test withut mdificatin t see if any f the results change in a secnd run. - Finally he can launch the test again frm the beginning, with new wrkings, changing the experiment input and refining the data set t be cllected. Page 39 f 41

40 7 Acrnyms 3G, 4G 3 rd /4 th Generatin A-QCM Autmated Quality Certificatin mdel ETICS e-infrastructure fr Testing, Integratin and Cnfiguratin f Sftware GPS Glbal Psitining System IMS IP Multimedia Subsystem ISO Internatinal Organizatin fr Standardizatin KPI Key Perfrmance Indicatr PTM PanLab Testbed Manager QS Quality f Service SLA Service Level Agreement SQS Sftware Quality Systems SS7 Signaling System 7 TIDS Testbed Infrastructure Data Service WS-API Web Services Applicatin Prgramming Interface Page 40 f 41

41 8 References [1] TEFIS deliverable: D2.1.1 [2] TEFIS deliverable: D3.1.1 [3] TEFIS deliverable: D3.3 [4] TEFIS prtal: [5] TEFIS experimente manager: [6] CMn: Page 41 f 41