Primary Research Report. In association with. IT Ops as a Digital. the lights on. Lessons from Digital Disrupters. July 2016

Transcription

1 Primary In association with IT Ops as a Digital Exploiting Business Enabler the Software More than just Advantage keeping the lights on Lessons from Digital Disrupters July 2016 October 2015

2 Introduction Some argue that IT operations doesn t matter anymore. It s all about innovation in today s fast moving digital world, so IT teams must focus their efforts on software development. Some argue that IT operations doesn t matter anymore. It s all about innovation in today s fast moving digital world, so IT teams must focus their efforts on software development. This is the impression we get from many pundits and technology companies nowadays. Developers, developers, developers has been the battle cry of more than one IT vendor executive. If IT operations is mentioned, it s often equated to keeping the lights on in a dismissive and sometimes derogatory manner. Even when speaking of DevOps, the emphasis is typically on Dev, with the Ops piece regarded as a necessary evil, there purely to serve the needs of upstream developers. Some have then been heard to propose NoOps as the ultimate goal. Our aim in this paper is to re-balance the discussion. We ll do this by walking you through our latest research on how the IT operations game is changing. Central to this are the results of a study (see Appendix A for sample distribution and methodology) in which we gathered responses from over 400 European IT professionals on the evolution of demands and challenges, and how Ops teams in medium and large organisations are responding. As we shall see, those taking a smart and modern approach are not just keeping the lights on, but enabling services that directly help their businesses compete in the market. The digital business context Let s start out by verifying some basic assumptions, namely the broad and critical role of IT in business nowadays, and the emerging requirements around digital (Figure 1). Figure 1 The digital business context for IT operations today IT is critical for our organisation to differentiate and compete in the market Most of our major business initiatives now have a significant IT component to them Competing effectively requires enabling the best digital customer experience Strongly agree Agree Maybe Disagree Strongly disagree Unsure The picture we see here is very clear, as is the prevailing view among respondents in our study of the operational implications (Figure 2). Figure 2 Operational implications of the latest business imperatives Effective IT security and data protection is now a key element of business risk management IT systems that are fast and reliable are now mandatory, not optional The range of applications considered critical today is much broader than 5 years ago 41% 41% 31% 16% Strongly agree Agree Maybe Disagree Strongly disagree Unsure It seems that today, it s best to assume critical status, secure 24x7 operation, and a need for responsiveness by default, not least as part of delivering a good customer experience. Copyright 2016, Freeform Dynamics Ltd Page 2

3 Escalating demands create operational strain. Ops teams feeling the strain If we add general growth to escalating business expectations, the result is operational strain across a whole range of different areas (Figure 3). Keeping up with escalating compute requirements 46% 46% 8% Protecting systems against cyber security threats 42% Keeping up with escalating storage requirements 52% 8% Figure 3 Have changing requirements and imperatives put operational strain on IT in any of the following areas? QoS management (availability and performance) Identity and access management Managing distributed data security and compliance Coordinating on-premise and cloud activity Keeping operational costs under control Managing rapid software release and deployment Workload management (scaling, migration, etc) Basic asset management (tracking what you have) Troubleshooting and remediating effectively 51% Keeping track of component/service dependencies Yes, significantly Pressure is building No problems (yet) Unsure Looking within IT itself, the digital need for rapid, iterative delivery of software, along with easy, secure access to systems and data, then adds to the pressure (Figure 4). Figure 4 Our use of DevOps increases the pressure on operational monitoring and management Aggravating factors as a result of evolving IT practices Increasing use of APIs makes planning, monitoring and management more of a challenge 46% Strongly agree Agree Maybe Disagree Strongly disagree Unsure One of the key questions arising from all this is why operations teams find it so hard to keep up. If you have any involvement in IT service delivery yourself, particularly in a larger enterprise environment, you will already know the answer to this. But for the record, here are the results from the research that sum it up at a high level (Figure 5). Our IT infrastructure is unavoidably complex Figure 5 The IT operations challenge in a nutshell Infrastructure complexity makes effective monitoring and management an ongoing challenge 17% Strongly agree Agree Maybe Disagree Strongly disagree Unsure Complexity cannot be avoided. We ll drill into this problem in more detail in a little while, then take a look at some of the options the research suggests you can explore to make things better. But before getting into that, let s remind ourselves of why your IT infrastructure is unavoidably complex in the first place. Copyright 2016, Freeform Dynamics Ltd Page 3

4 The unavoidable problem of complexity A huge number of factors make complexity management an ongoing challenge. Figure 6 How much have the following contributed to technology variation and complexity in your environment? This next chart has a lot on it. We make no apologies for this because it illustrates the sheer number of factors that make complexity management an ongoing challenge in any medium or large enterprise IT context (Figure 6). Mergers and acquisitions (bringing together different standards/legacies) Geographic distribution (local market supplier/support differences) Influence of the business (more opinionated stakeholders, shadow IT ) Differing application needs (matching of platform options to requirements) Mix of skills and experience (teams select the technologies they know) IT staff preferences (people select the technologies they like) General accumulation over time (new tech comes, but old tech remains) Chasing the cheapest option (heavy focus on immediate cost of acquisition) Deliberate multi-sourcing strategy (use multiple suppliers to minimise lock-in) Weak asset lifecycle management (don t track, monitor and review well enough) Lack of funding for modernisation (not enough budget for upgrade and replacement) Poor procurement discipline (standards and guidelines not adhered to) Inadequate procurement policy (standards and guidelines not well enough defined) Fads and fashions (desire to use the latest hot technologies) 50% 46% 50% 53% 52% 41% 55% 54% 16% 17% 17% Big factor Some impact No impact Unsure New tech comes, but old tech remains. Of course each organisation experiences its own range of contributors to the complexity problem, but the data behind this chart tells us that around 9 out of 10 organisations identifies at least one of the items listed as a big factor, with most citing four or more. Picking up on the reference to new tech comes, but old tech remains, a quick look at expected lifetimes of basic infrastructure components explains why, against the backdrop of high growth, multiple generations of technology accumulate over time (Figure 7). Figure 7 Very roughly, for review, budgeting and planning purposes, what kind of lifetime do you generally assume for the following types of assets nowadays? Up to 3 years 51% 3 to 5 years 6 to 7 years 8% 8 to 9 years Servers Storage devices Network equipment 6% 4% 1% Longer Copyright 2016, Freeform Dynamics Ltd Page 4

5 The extent of the complexity problem Significant variation is evident across all technology categories. We can begin to get a feel for the extent of the complexity problem if we broaden our view. Looking beyond servers, storage and networking to the broader range of core elements present in almost every enterprise IT landscape, we see examples of significant variation in all categories (Figure 8). Business applications Collaboration systems Web/application middleware Database technology 42% Virtualisation platforms Figure 8 How would you summarise the level of consistency in each of the following areas? Server operating systems x86 server hardware Non-x86 servers Storage platforms Networking infrastructure 46% Security infrastructure 50% Hosting/cloud service arrangements Monitoring and management tools High Medium Low Unsure/NA But even this is just the start, as we must also consider newer or emerging technologies that find their way into the infrastructure as the organisation takes advantage of the latest industry developments (Figure 9). Converged or hyperconverged architecture Private/hybrid cloud architecture Container technologies Modern flash/hybrid based storage arrays 16% Scale-out storage systems (Ceph or similar) Figure 9 Big data platforms (Hadoop or similar) Do you have any of the following in production yet, or are you moving in that direction? In-memory databases (e.g. HANA) and related apps Other NoSQL / alternative database technology Supercomputing or HPC platforms Intelligent networks / software defined networking 17% 16% Desktop virtualisation technology (Citrix or similar) 9% Mainframe accelerators for DB2, Java, etc 8% Mainframe Linux environment In production Currently implementing Currently investigating No activity Unsure Various database management systems, application stacks, storage solutions, etc, are combined in myriad ways to meet differing application needs. Of course complexity doesn t just stem from the range of technologies in place and the variation that exists within each category. We must also consider the many ways in which the elements are brought together and configured to deliver business services. If you look across your infrastructure you are likely to see different database management systems, application stacks, storage solutions, etc, combined in myriad ways to meet the needs of the hundreds, maybe even thousands, of applications that you run. You will undoubtedly set standards and try to keep things as consistent as possible, and periodically rationalise, consolidate and re-platform to regain some order, but the huge set of ever-changing application requirements and dependencies makes ongoing complexity inescapable. Copyright 2016, Freeform Dynamics Ltd Page 5

6 Operational management capability Tools and processes usually do the job, but often not that efficiently. Figure 10 How would you describe your operational monitoring and management capabilities in relation to the following? Enterprise IT ops is about monitoring and managing the kind of complex environments we have described, and the IT team clearly needs the processes and tools to do this. But how well do our study respondents stack up in this regard? The good news is that the majority report effective monitoring and management capabilities in most core technology areas. However, many also tell us that what s in place is not very efficient (Figure 10). Business applications Collaboration systems Web/application middleware Database technology Virtualisation platforms Server operating systems x86 server hardware Non-x86 servers Storage platforms Networking infrastructure Security infrastructure Hosting/cloud service arrangements Mainframe environment 55% 51% 50% Effective and efficient Effective but not very efficient Not very effective Unsure/NA 41% 16% And the same is true when we home in on those using newer technologies (Figure 11). Figure 11 How would you describe your operational monitoring and management capabilities in relation to the latest generation of technology options you use? Converged or hyperconverged architecture Private/hybrid cloud architecture Container technologies Modern flash/hybrid based storage arrays Scale-out storage systems (Ceph or similar) Big data platforms (Hadoop or similar) In-memory databases (e.g. HANA) and related apps Other NoSQL/alternative database technology Supercomputing or HPC platforms Intelligent networks / software defined networking Desktop virtualisation technology (Citrix or similar) Mainframe Linux environment Mainframe accelerators for DB2, Java, etc 56% 51% 51% 52% 50% 53% 46% 8% 10% 10% 10% Effective and efficient Effective but not very efficient Not very effective Unsure/NA Inefficiencies can result from outdated or disjointed toolsets. Figure 12 Tooling and process redundancy and disjoints Such inefficiencies can arise because of outdated or inadequate tooling, for example, or excessive reliance on manual processing. Whatever the cause, the result is high overheads, more mistakes and a slower response to change. Just as with infrastructure complexity, we then have overarching aggravating factors coming into play as we look at monitoring and management capabilities from a higher level perspective (Figure 12). We have multiple tools that do essentially the same job in many areas We simply have too many tools and processes that aren t integrated with each other Strongly agree Agree Maybe Disagree Strongly disagree Unsure The net of this is that in many cases, we can add tooling complexity and disjoints to the complexity we saw previously at an infrastructure level. Copyright 2016, Freeform Dynamics Ltd Page 6

7 Impact on service delivery A key question is how much good operational capability matters. A key question when considering the state of an enterprise IT infrastructure and the capabilities in place to help monitor and manage it is how much any of it matters. In order to examine this, we captured a qualitative view of how well participants in the study thought they were doing in relation to five service delivery indicators. Survey responses were then converted into numeric scores and averaged, yielding a performance index that could be used to segment organisations into three groups (Figure 13). Maintaining operational service levels 54% 9% Delivering overall value to the business Managing IT and information related risks 51% 17% Controlling the cost of IT delivery and operations 51% Figure 13 How well would you say IT is delivering at the moment against the following? Responding quickly and effectively to new needs 53% 17% Optimally (little room for improvement) Adequately (but could do better) Inconsistently (range of good and bad) Poor (generally falling short) Unsure Segmentation according to average overall performance scores PERFORMANCE SEGMENTS High achievers ( of sample) Mixed achievers (46% of sample) ( of sample) are more likely to have both better infrastructure consistency and superior operational management capability. Once we had our performance groups, we played a similar trick with the responses shown previously in Figures 8 and 10 so we could easily figure out if respondents were above or below average in relation to infrastructure standardisation and monitoring/management capability. The point of all this number crunching should be pretty obvious when you look at the chart below. The analysis tells us that are much more likely to have made efforts on infrastructure consistency, and they also tend to have superior monitoring and management capabilities in place (Figure 14). SERVICE DELIVERY PERFORMANCE 42% 75% 74% 58% Figure 14 Above average Below average INFRASTRUCTURE STANDARDISATION How well would you say IT is delivering at the moment against the following? SERVICE DELIVERY PERFORMANCE 79% 79% 55% Above average Below average CORE MONITORING AND MANAGEMENT CAPABILITY We must be careful about assuming causality when looking at data like this, but the correlations we see would seem to be logical. Copyright 2016, Freeform Dynamics Ltd Page 7

8 The basis on which you engage with the business is important. Other enhancers of service delivery While better processes and tooling, and trying to maintain as much infrastructure standardisation as possible undoubtedly helps to serve the business more effectively, operations teams clearly don t exist in a vacuum. With this in mind, the basis upon which they engage with the business is important, and we see some striking differences here between the different performance groups (Figure 15). Figure 15 How would you describe what you have in place in the following areas? A clearly defined set of services delivered by IT Formally defined service level agreements Measurement of user and stakeholder satisfaction 61% 66% 83% 61% Comprehensive Selective Ad hoc Non-existent/Unsure These kinds of efforts to build and maintain business alignment manifest themselves in other ways. If you are actively engaged in conversations with executives and stakeholders about needs and expectations, then they are much more likely to understand and appreciate operational pressures and realities (Figure 16). Figure 16 How well do business executives/stakeholders appreciate IT operations pressures and realities? 8% Very well in general They get it but need to be constantly reminded They only take notice when something goes wrong No interest, they just blame IT when issues arise Varies too much to generalise The objective is to create a virtuous circle of dialogue, delivery and investment. This kind of reciprocal empathy sets up a virtuous circle the more open the dialogue, the better you can prioritise, and the easier it is to justify investment in modernisation and tooling. And the more effectively you deliver, the more stakeholders are willing to engage in further dialogue. The thing to beware of, of course, is that the opposite is also true, which can lead to a vicious circle in which struggling IT teams with low credibility within the business find it increasingly hard to invest and deliver. If you ever find yourself in this position, then talking it out with those concerned should be the immediate priority. A similar set of principles apply within the IT function itself. Historical silos and lines of demarcation, e.g. between development and operations teams, can undermine your overall ability to serve the business effectively, efficiently and responsively. It s therefore no surprise to find that have generally made significantly more progress in implementing DevOps and Continuous Delivery (Figure 17). Figure 17 57% 3% How would you describe your capabilities in relation to DevOps and/or Continuous Delivery? 66% 8% 50% Mature Evolving Basic, not evolving No capabilities Unsure/NA 31% Copyright 2016, Freeform Dynamics Ltd Page 8

9 The role of cloud and hosting The only thing that ultimately matters is whether cloud and other forms of hosting help you to deliver a better service to the business. You may be wondering why we haven t mentioned cloud and hosting so far. After all, some of the more extreme IT industry rhetoric we hear suggests that getting rid of all your infrastructure and running everything in the cloud magically makes all of your operations problems disappear. Evangelism and vested interest to one side, the only thing that ultimately matters is whether cloud and other forms of hosting help you to deliver a better service to the business. In this respect, it is interesting to see a much weaker correlation between cloud and hosted service adoption and service delivery performance than previously observed in relation to other performance enhancing factors (Figure 18). Figure 18 To what degree are your servers hosted by a third party provider at the moment? The reason for this will be obvious to anyone who has experience of trying to scale cloud adoption in an enterprise environment, especially as the number of providers and service arrangements proliferates over time. Far from representing a magic bullet, cloud and hosting tend to create almost as many problems as they solve (Figure 19). Extensively Significantly A fair amount Modestly or not at all Managing security/identity between cloud services and in-house systems Assuring appropriate data protection of data stored in the cloud Figure 19 Do you see any of the following as issues in relation to the use of cloud services? Managing security/identity across cloud service silos (i.e. between providers) Keeping track of what data is being stored where Assessing the QoS and security characteristics of specific cloud services Keeping track of where cloud services are being used Troubleshooting applications and services that span on-prem and the cloud 42% 41% Managing QoS of applications and services that rely on cloud elements 31% Major challenge Significant distraction Not a big problem Unsure The data suggests that success with cloud, just as with on-premise infrastructure, is largely about effective operational management across silos and architectures (Figure 20). Figure 20 How would you describe your monitoring and management capabilities in relation to the cloud and hosting? 72% Effective and efficient Effective but not very efficient Not very effective Unsure/NA This lesson is particularly important to take on board as hybrid architectures continue to blur the lines between on-premise infrastructure and hosted cloud resources. Copyright 2016, Freeform Dynamics Ltd Page 9

10 Let s not forget the mainframe are more likely to be using the mainframe and regarding it as a strategic part of their IT infrastructure. The last aspect of IT operations we will look at is in relation to the IBM mainframe - that hugely valuable asset that commentators frequently overlook when considering the future. With 320 of those participating in our study reporting investments in this area, it makes sense to consider the mainframe s role and how well it is dealt with operationally. On that point, it is telling that are not only more likely to be using the IBM System z platform, they are also more inclined to regard it as a strategic and fully integrated part of their IT landscape (Figure 21). 63% 6% Figure 21 Does your organisation use IBM mainframe (System z) technology? Yes, as a strategic and fully integrated part of our overall IT infrastructure Yes, as a strategic platform, but operationally separated from everything else Yes, but we regard the mainframe as legacy No, but we are considering mainframe adoption No use and no plans Mainframes are extremely well suited to the needs of digital business, but they need to be managed well. Figure 22 How would you describe your mainframe monitoring and management capabilities? Figure 23 How would you describe your mainframe monitoring and management capabilities? This makes sense when we consider how many organisations have been waking up to the fact that mainframes are extremely well-suited to the needs of digital business. Robustness, security and scalability are key attributes for the back-end systems that enable online web and mobile customer engagement, IoT platforms and applications, and many of the associated information management and analytics requirements. Despite the mainframe heritage of discipline and control, however, we know from anecdotal evidence that the environment is not immune to the problems of inadequate, outdated and disjointed tooling and processes. It is therefore not wholly surprising that less than half of our mainframe users are satisfied that their operations environment is both effective and efficient (Figure 22). Effective and efficient Effective but not very efficient 4% A familiar pattern is then seen when we look at how this varies by performance group, underlining that operational capability matters, regardless of the environment (Figure 23). Not very effective Unsure 71% Effective and efficient Effective but not very efficient Not very effective Unsure 41% Copyright 2016, Freeform Dynamics Ltd Page 10

11 Acting on the insights It makes sense to focus on both reducing infrastructure complexity and boosting your operational capability. As you have been reading this report, you may have been trying to work out where your organisation sits on the delivery performance spectrum. If your IT team s level of achievement is mixed or you are generally falling short in terms of service delivery more broadly, then it makes sense to focus on both reducing infrastructure complexity and boosting your operational capability. If we visualise the research data in a slightly different way, it s clear that both of these dimensions are important if you want to improve your chances of becoming a High Achiever (Figure 24). Figure 24 What makes a difference? CORE MANAGEMENT CAPABILITY High (above average) Low (below average) 60% 7% 31% 9% 55% 54% 7% Low (below average) High (above average) INFRASTRUCTURE STANDARDISATION Your IT environment must constantly change to meet evolving business needs, so optimising it then locking it down is not the objective, even if it could be achieved. When looking to act on insights of this kind, however, it s important to keep things in perspective. Aiming for a totally standardised infrastructure and the perfect set of processes and tooling is unrealistic. Many of the factors highlighted earlier that contribute to ongoing complexity are simply facts of life over which you have limited control if you work in IT. Indeed, with the rate at which technology continues to advance and deliver greater value, it doesn t make sense to take a lockdown approach. Your enterprise IT environment is, and always will be, in constant flux, and this is an essential part of keeping up with business demands and market opportunities in today s digital world. Against this background, the way you act will depend on your current position. If parts of your environment have drifted into a particularly chaotic or unnecessarily complex state for whatever reason, then consolidation and/or modernisation work will often pay back convincingly. This will be especially true if your team is coming under increasing pressure to cut overheads, innovate more, improve service levels (availability, performance, etc), reduce operational risk and/or generally become more responsive. It can be hard to find the time and resources to spend on such exercises, but if an important part of your IT environment is highly fragmented or dependent on old or obsolete technology, then it will continue to hold you back until the issues are addressed. Copyright 2016, Freeform Dynamics Ltd Page 11

12 put a strong emphasis on good management capability at the early adoption stage. When it comes to introducing newer or non-core technology options the advice is to think about the operational dimension right from the outset. We don t have room to go through all of the technologies previously listed, but the following examples illustrate how put a strong emphasis on implementing good management capability at the early adoption stage (Figure 25). Desktop virtualisation platforms 52% Associated management capability 76% 41% Figure 25 Early adoption behaviour In production Currently implementing Currently investigating No activity Unsure In-memory databases/apps 17% In production Currently implementing Currently investigating No activity Unsure Container technologies Effective and efficient Effective but not very efficient Not very effective 72% Unsure Associated management capability Management capability percentages relate to those using the technology Effective and efficient Effective but not very efficient Not very effective 64% Unsure Associated management capability Management capability percentages relate to those using the technology 53% 10% 46% In production Currently implementing Currently investigating No activity Unsure Effective and efficient Effective but not very efficient Not very effective Unsure Management capability percentages relate to those using the technology Rushing too far into adoption of any technology without paying attention to operational requirements is likely to limit the value you get from it. The Ops is at least as important as the Dev in DevOps from a business perspective. The patterns we see here are consistent across all of the emerging or advanced options we looked at in the study. This reinforces the notion that rushing too far into adoption of any technology without paying attention to operational requirements is likely to limit the value you get from it. Final thoughts In an industry that has recently become obsessed with developers and software development, we hope the research and associated commentary we have presented in this report go some way towards rebalancing the discussion. If you work in a born in the cloud organisation you may be lucky enough to escape having to think about the kind of complexity that results from decades of investment and the accumulation of IT heritage over time. You may even be able to take a NoOps approach, though how easy that is to maintain as an organisation grows and diversifies is hard to imagine. Back in the world of enterprise class IT, effective operational monitoring and management is essential. Success with digital business, for example, requires the delivery of good user experiences. This depends not just on appealing mobile apps and web interfaces, but also back-end systems that are fast, reliable and secure, and flexible enough to keep pace with modern DevOps delivery. Against this background, regarding IT operations as simply about keeping the lights on is a recipe for poor business performance. The bottom line is that the Ops is at least as important as the Dev in DevOps from a business perspective. Copyright 2016, Freeform Dynamics Ltd Page 12

13 Appendix A: Research Sample The study upon which this report is based was designed, executed and interpreted by Freeform Dynamics Ltd in collaboration with CA Technologies. Data was gathered from 401 respondents via an online survey completed in May The sample distribution for the online survey was as follows: Sample by geography Sample by organization size Germany Italy 250 to 5,000 employees UK 5,000 to 25,000 employees France Spain More than 25,000 employees Switzerland Sweden 9% 9% Sample by industry sector Netherlands 9% Telecoms 16% Denmark 8% Cloud/hosting and managed service providers Figure 26 Composition of research study sample Involvement in IT Operations Overall strategic responsibility 53% Retail Healthcare Financial Services Day-to-day management responsibility Utilities and Energy Technical specialist / practitioner Public Sector Automotive 10% 10% Sample by role Overall head of IT Head of IT operations Ops or DevOps team leader/manager IT professional (architecture/design) 17% IT professional (operations) Note on methodology The survey was conducted online and respondents self-selected into the study. We must therefore be aware of possible sample bias towards more advanced respondents who are generally more enthusiastic and more likely to respond to a research call to action. This does not affect the commentary or conclusions contained in this report, but should be borne in mind when considering the data in another context. Copyright 2016, Freeform Dynamics Ltd Page 13

14 About Freeform Dynamics Freeform Dynamics is an IT industry analyst firm. Through our research and insights, we aim to help busy IT and business professionals get up to speed on the latest technology developments, and make better-informed investment decisions. For more information, and access to our library of free research, please visit About CA Technologies CA Technologies (NASDAQ: CA) creates software that fuels transformation for companies and enables them to seize the opportunities of the application economy. Software is at the heart of every business in every industry. From planning to development to management and security, CA is working with companies worldwide to change the way we live, transact, and communicate across mobile, private, and public cloud, distributed and mainframe environments. Learn more at Terms of Use This document is Copyright 2016 Freeform Dynamics Ltd. It may be freely duplicated and distributed in its entirety on an individual one to one basis, either electronically or in hard copy form. It may not, however, be disassembled or modified in any way as part of the duplication process. Hosting of the entire report for download and/or mass distribution by any means is prohibited unless express permission is obtained from Freeform Dynamics Ltd or CA Technologies. The contents contained herein are provided for your general information and use only, and neither Freeform Dynamics Ltd nor any third party provide any warranty or guarantee as to its suitability for any particular purpose. Copyright 2016, Freeform Dynamics Ltd Page 14