Europe s Premier Software Testing Event Stockholmsmässan, Sweden Testing For Real, Testing For Now Evolution of New Feature Verification in 3G Networks Michael Monoghan, LM Ericsson Ltd,. Ireland WWW.EUROSTARCONFERENCES.COM
Evolution of new feature verification in 3G Networks Eurostar Stockholm, December 2009 Michael Monaghan, LM Ericsson (Ireland) Ltd
Short overview of LM Ericsson Founded 1876, Stockholm, Sweden Presence in 175 countries > 75,000 employees ~24,000 patents > 40% of all mobile traffic passes through Ericsson networks Ericsson supports networks which serve > 1 billion subscribers Business Units Networks Multimedia Global Services Sony Ericsson In this presentation, we will visit the 3G Mobile Network Development area 3G refers to 3 rd Generation, a family of standards for wireless communication A glossary of terms appears at the end of this presentation
Feature Verification in 3G Networks The scenario Very complex systems Very complex features Very difficult to recreate real world environment Many levels of test from block test to complex network test Many tools needed The aim To verify that features behave according to requirements To make a quality statement about new features, so a decision can be made about whether to release them To satisfy Time to Market and cost goals for the feature
Feature Verification in 3G Networks EM OSS-RC RNC RBS RXI EM Operations Support System Radio & Core Radio Network Controller Radio Base Station Radio Network Integrator Element Manager Element management WCDMA: Wideband Code Division Multiple Access, an air interface standard
Downlink 3.6 Mbps 14 Mbps 15 codes 64QAM 2x2 MIMO HSPA Evolution Uplink 20-40 Mbps 12 Mbps Multi Carrier 16QAM Multi Carrier 21 Mbps 28 Mbps Multi Carrier 64QAM + MIMO Higher Modulation Combinations 42 Mbps 80-160 Mbps 5.8 Mbps 1.4 Mbps 0.4 Mbps 2 ms TTI Refer to glossary for explanation of terms...
Major Advances in Verification of 3G Network Features Evolution from Function Based to Feature Based Verification More generic development leads to verification savings Evolution of Feature Verification Looking to the Future Summary
Advances in RNC Feature Verification Function Test Generic Design/Test Strategy FT to FV Move Parallel Testing Future 2004 2005 2006 2007 2008 2009 2010 FT: Function Test FV: Feature Verification
Function Test Function Test 2004 2005 RNC verification was done from a functional viewpoint Several functional teams triggered the same test case and then checked one part of the result A test case most often consists of Setting up a particular type of call (Control Plane) Sending data (User Plane) Moving the phone (Mobility)
Function Test Function Test 2004 2005 Mobility Team Control Plane Team User Plane Team Feature1 Mobility User Plane Feature2 Mobility Control Plane Feature3 Control Plane User Plane Feature4 Mobility Control Plane Feature5 Mobility Control Plane User Plane
Function Test Function Test 2004 2005 Small area of expertise for each engineer Little feature overview knowledge in verification Led to rigid test organisations and verification projects
Feature Based Verification FT to FV Move 2006 2007 Verification based on Use Cases Verify features end to end Verify each Use Case in the most suitable environment Many challenges had to be overcome
Feature Verification FT to FV Move 2006 2007 Mobility Team Control Plane Team User Plane Team Feature1 Mobility User Plane Feature2 Mobility Control Plane Feature3 Control Plane User Plane Feature4 Mobility Control Plane Feature5 Mobility Control Plane User Plane
Feature Based Verification FT to FV Move 2006 2007 Feature Verification has led to significant cost savings Very responsive and flexible test organisation An enabler for a feature based release strategy
Feature Based Verification FT to FV Move 2006 2007 Feature Verification, an evaluation: Feature #TCs Cost per Test Case (hours) Total Cost (hours) #TCs Cost per Test Case Feature1 168 9 1512 34 8-20 (Avg 14) Total Cost (hours) 476 69% saving!! Function Test Feature Verification
RAB (Radio Access Bearer) Concept Generic Design/ Test Strategy 2005 2006 Data Data Speech plus data Speech plus data Speech Speech
Generic Design/Test Strategy Generic Design/ Test Strategy 2005 2006 Market pressure for many radio connection combinations e.g. different interactive packet RAB rates and combinations Development began to be based on generic code units which made it easier to introduce new RAB rates and combinations In P6 the number of RAB combinations exploded : (13 in P3, 18 in P4, 39 in P5), > 70 in P6, Not possible to verify all transitions SP64 CS Streaming Speech SPHS Int. 128/384 Int. 384/384 SRB Int. 128/128 Int. 384/128 Other int. RAB state Int. 64/384 128/64 384/64 Int. 64/128 PS Streaming+Int 8/8 UDI (CS64) SP0 64/64 UDI+Int. 8/8 HS Int. FACH SRB
Generic Test Strategy Generic Design/ Test Strategy 2005 2006 Test strategy was to verify the generic code once and then run a greatly reduced test scope on new RABs/RAB Combinations going forward Reduced scope testing based on RAB type groups
Generic Test Strategy Savings Generic Design/ Test Strategy 2005 2006 Feature Pre-Generic (including all error and exception TCs) Generic Test Principle 1 (Removal of error and exception TC) Generic Test Principle 2 (Rab type groups) Generic Test Principle 3 (Some functions moved outside FV) Generic Test Principle 4 (Functional based changes costs removed) Feature2 ~ 196 TC 96 TC (-100) 88 TC (-8) 58 TC (-30) 43 TC (-15) ~ Exe cost 1764 hours ~ Exe cost 864 hours ~ Exe cost 792 hours ~ Exe cost 522 hours ~ Exe cost 387 hours 79% saving!
Generic Test Strategy Effectiveness Generic Design/ Test Strategy 2005 2006 Feature #TCs run in RNC Feature Verification Faults found by RNC Feature Verification Faults found by IODT (Mobile phone Verification) Faults found by RAN I&V Slipthrough analysis to RAN I&V Feature3 102 34 8 2 2 Analysis: Same faults would have been found in RNC but testing was done in parallel Actual TRs on feature Analysis: 1 due to tools issues, 1 due to intro of fault after test case had been run. Generic test is very effective when applied correctly
More Demanding Environment Parallel Testing 2007 2008 2 releases per year Market demand for many and more complex features (Mobile Broadband) Continuous improvements in Time To Market required Need for continuous improvements in operational excellence Need to streamline verification further: Introduce more parallel testing Minimize repetition in testing at all levels Maximise test effectiveness
Parallel Testing Parallel Testing 2007 2008 RNC node test work more closely with higher layer test organisations Form a RAN Level Joint Test team with participation from network nodes (RNC, RBS) and RAN Make Phased Deliveries for key features Split scope of feature test between RAN and nodes
Node/Subsystem I&V Integration Node/ Subsystem I&V Merge 2008 2009 Subsystem I&V is a very useful, low cost and under-utilised test environment Was Best Effort/Happy Test Integration of test activities means node level TCs are passed in a Subsystem I&V environment Improves verification lead times at RNC node level Reduces repetition between Subsystem and RNC test documentation and execution Need to minimise quality risk to RNC and Subsystem
Maximise Test Effectiveness Future 2009 2010 Merge of RNC node feature verification with RAN feature verification Pass RAN level test cases in a subsystem verification environment Remove RNC node verification as a layer of verification Eurostar conference Ericsson verification problems are industry-wide We compare favourably with other companies in many respects However there are some methodologies we should study more Risk Based Test Case Selection Each test case ranked by likelihood and impact Trial was inconclusive Estimate number of remaining faults Tools exist which predict number of latent faults in a feature Too much emphasis now on slip-through Preventive Testing
Summary Summary 2004 2010 We have come a long way Time pressure will continue to mount Borders between layers of the development process are being broken down
Contact Details Michael Monaghan, LM Ericsson (Ireland) Ltd Michael.A.Monaghan@ericsson.com
Glossary EM - Element Manager A simplified management interface for RNC and RBS nodes FT - Function Test a test strategy where new features are verified by verifying the functions which realise the features one by one FV - Feature Verification a test strategy where new features are verified as a unit HSPA - High Speed Packet Access a collection of protocols which provide high bandwidth and improved spectral efficiency in 3G networks Interfaces The communication channel and protocol between network nodes Iu - interface between the RNC and the core network Iub - interface between the RNC and the Radio Base Station Iur - interface between 2 RNC nodes Mun interface between other management systems and OSS-RC Mur - operator management system interface towards RNC Mut - operator management system interface towards RXI Mub - interface from operator management system towards RBS Uu - interface between the RBS and the Ue
Glossary (contd...) I&V - Integration and Verification a generic term in Ericsson to refer to all verification activities MIMO - Multiple-input and multiple-output use of multiple antennas at both the transmitter and receiver to improve communication performance Multi-Carrier a method of allowing a mobile device listen to two HSPA data streams from different cells simultaneously in downlink, thus facilitating higher bandwidth OSS-RC Operation Support System for Radio and Core a network management node P6 Project or Phase 6 Ericsson RAN product releases are numbered P2...Pn RAB - Radio Access Bearer a UMTS service providing an application using the UMTS network with the ability to send and receive data over the UMTS network with a specific Quality of Service (QoS). RAN - Radio Access Network part of a mobile telecommunications system which implements the radio access technology, so the mobile equipment can communicate with the backbone networks
Glossary (contd...) RBS - Radio Base Station a 3G network node containing radio frequency transmitter(s) and receiver(s) used to communicate directly with the User Equipment RNC - Radio Network Controller a network node in a 3G network which controls the base stations which are connected to it. It also carries out other functions including radio resource management and mobility management RXI An Access Network Aggregator an ATM switch supporting handling of various network topologies and technologies TC - Test Case For the purposes of this presentation, a test case generally involves establish a specific type of phone call (voice and/or data) Ue - User Equipment any device used directly by an end-user to communicate. It can be a hand-held telephone, a card in a laptop computer, or any other device. It connects to the base station WCDMA - Wideband Code Division Multiple Access an air interface standard used in 3G networks 16/64 QAM -16/64 Quadrature amplitude modulation methods of coding data for transmission over the air
Glossary (contd...) 2MS TTI - 2MS Transmission Time Interval Transmission Time Interval refers to the length of an independently decodable radio transmission 3G - 3rd Generation a family of standards for wireless communications. Main characteristics are wide range of services, high throughput and spectral efficiency