Robotesting: Are you ready for that yet? Testing of robots Testing with robots Rik Marselis October 2017 Who has a robot? In 10 years all of you will!! Sogeti 2017 2 Sogeti 2017 Page 1
1980 Workgroup -member 2007 2008 2009 2012 2012 @rikmarselis 2014 Rik Marselis The members of TestNet say hello to you. And trust that this conference will be just as interesting and fun as their conference last week! www.testnet.org (take a look, but beware, it s all in Dutch ;-) Sogeti 2017 Page 2
Our first bit of robotesting How can we stop R0B3 without touching it? Sogeti 2017 5 Robots in practice Over 50% of the dairyfarms in the Netherlands uses one or more robots. Sogeti 2017 6 Sogeti 2017 Page 3
Manure robot Low intelligence for specific tasks ( dull and/or dangerous ) Sogeti 2017 7 Milking robot Higher intelligence both replacing human labour and gathering of valuable data Sogeti 2017 8 Sogeti 2017 Page 4
What is a Robot? It s a machine that gathers information about its environment by input of sensors and based on this input changes its behavior. Combined with machine learning and machine intelligence the robot s reactions over time get more and more adequate. The use of Internet of Things, Big Data Analytics and Cloud technology make a robot versatile. A Robot can come in many different shapes and forms. It s not just the metallic man. It may just as well be a smart algorithm on social media, an autonomous vacuum cleaner or a self-driving car. This definition was assembled by Rik Marselis from various sources. Sogeti 2017 9 Robotesting Testing of robots October 2017 Sogeti 2017 Page 5
Different angles of quality for robotics Mechanical Electrical Information Processing Social Impact Machine Intelligence For these traditional tests we already have the knowledge and experience for decades. And we Business have many tools to help us testing Impact efficiently and effectively. Sogeti 2017 11 New risks having impact on business process Product Risk Analysis New risks Well-known approach Other outcoumes, for Test strategy Test techniques Test coverage Sogeti 2017 12 Sogeti 2017 Page 6
Artificial intelligence (from the film Ex Machina 2015) Artificial intelligence will come. The question is when, not if. Sogeti 2017 13 About Artificial Intelligence Artificial General Intelligence Human-like, i.e. capable of any task Artificial Narrow Intelligence Focused on one specific task Machine Intelligence Machine Intelligence (MI) is a unifying term for what others call Machine Learning (ML) and Artificial Intelligence (AI). When we called it AI, too many people were distracted by whether certain companies were true AI, and when we called it ML, many thought we weren t doing justice to the more AI-esque like the various flavors of Deep Learning. (Source: Machine Intelligence Executive introduction, SogetiLabs) Sogeti 2017 14 Sogeti 2017 Page 7
Different angles of quality for robotics Mechanical Electrical Information Processing Machine Intelligence new Business Impact new Social Impact new Sogeti 2017 15 Machine learning <<< vervangen door filmclip >>> Sogeti 2017 16 Sogeti 2017 Page 8
Testing the machine learning capabilities Testing machine learning is much different from traditional testing. Traditionally the output could always be predicted. With machine learning the output will be (must be!!) different over time. One of the ways to handle this is also focusing on the input of the learning machine. Sogeti 2017 17 Sogeti 2017 18 Sogeti 2017 Page 9
Testing machine learning: it s about the input!! The tester as psychologist of the intelligent machine Sogeti 2017 19 Social impact of robots Since the introduction of the milking robot, the farmer and his family can now have dinner together at 18:00 o clock (as usual in the Netherlands) De aardappeleters. Vincent van Gogh Sogeti 2017 20 Sogeti 2017 Page 10
Testing new quality attributes Traditionally testing was mostly focused on IT. With new technology business impact and social impact become more and more important. Think of quality attributes like Ethics and Embodiment For example: Do we want robots to take care of elderly people? And if so, what should such robot look like? Picture from film Robot and Frank 2012 Sogeti 21 Image recognition (how hard can that be?) Sogeti 2017 22 Sogeti 2017 Page 11
Robotesting Testing with robots October 2017 Robot automatically performs regression tests Sogeti 24 Sogeti 2017 Page 12
Robot automatically performs regression tests Low-cost ( 300,-) robot-arm tests mobile devices. Connected to test management tool. Test cases check if the quality goals are met and risks are covered. Sogeti 2017 25 Robotesting Use of Artificial Intelligence October 2017 Sogeti 2017 Page 13
Self-exploring testing tools Self-exploring testing tools Would this be possible? See in this video how Google s DeepMind learns to play breakout. In my opinion Artificial Intelligence will be able to assist testers in exploring software in the very near future. Read more in the article of Sander Mol and Rik Marselis Sogeti 2017 27 http://labs.sogeti.com/testing-self-learning-self-exploring-testing-tools/ In the mean time, use robots and AI for Functional testing Generate test cases Execute test cases Analyse the results Brute-force testing Generate a huge number of test cases and execute them Non-functional testing Intelligent performance testing And so much more!!!!! Sogeti 2017 28 Sogeti 2017 Page 14
Machine intelligence: test data management What is a great challenge with using live data? Privacy issues. Use machine intelligence to analyze the live data and find all different situations. These situations are your test situations. Based on these test situations the intelligent machine can generate synthetic data. Advantages: - The dataset contains only the test situations your want (small file) - All variations will be in the testset (good coverage) - The data is not in conflict with privacy-regulations Sogeti 2017 29 Machine intelligence: path coverage What is a great challenge with testing business processes? Covering the business process in an optimal way. Use machine intelligence to analyze the possible process paths. These situations are your test situations. Based on these test situations the intelligent machine can generate test cases. And for each test case find relevant test data. Advantages: - The dataset contains only the test situations your want (small testset) - All variations will be in the testset (good coverage) - No hassle to determine the right test data Sogeti 2017 30 Sogeti 2017 Page 15
Machine intelligence: output prediction What is a great challenge with testing business processes? Predicting the expected outcome. Use machine intelligence to analyze the test situation and to predict the expected outcome. Advantages: - The machine intelligence is better at strictly applying rules (no errors in output predictions) - Less manual labour Disadvantage: The rules must be very clear Sogeti 2017 31 Cognitive QA The next step in test automation COGNITIVE QA PLATFORMS Testdata management, service virtualization, etcetera, with Artificial Intelligence Test Execution Analysis Test Coverage Optimization SMART ANALYTICS FOR QA COGNITIVE QA INTELLIGENT QA AUTOMATION Test tools augmented with Artificial Intelligence www.cognitive-qa.com PREDICTIVE QA DASHBOARDS Predictive Analysis of test results Sogeti 2017 32 Sogeti 2017 Page 16
My tip if you fear for your future career Make sure you are good at collaborating with robots, so you will be the last to be made superfluous Sogeti 2017 33 Robots will change your testing!! Questions? Rik Marselis October 2017 Sogeti 2017 Page 17