Sometime back, I had lunch with a CTO of a well-established financial services enterprise. I posed the question – what do you guys think about doing big data processing in the cloud? That created a fairly violent reaction, followed by a less violent discussion eventually leading to a few really interesting points and questions in my mind.
My CTO friend has serious and legitimate concerns about moving big data to the cloud. To better understand why, I will pose his response as a thought experiment around three key perceptions:
- Is the cloud simple or complex?
- He said complex, in fact really complex
- Is the cloud secure or not?
- He said not secure
- Is the cloud cheap or expensive?
- He said expensive
I have spoken with numerous people about this topic since our lunch, and I have realized there is much confusion in the market. Obviously there is no right or wrong, but it’s important that we focus on data and facts.
So in the spirit of collecting data and facts (or at least quantifying perceptions), Cazena backed a first-time GigaOM Survey that polled enterprises on their view of doing big data in the cloud. Andrew Brust, Research Director, Big Data & Analytics at GigaOM, seemed like the perfect guy to lead the research. Andrew is a practitioner with a healthy skepticism in everything he regards.
Over 300 enterprises were contacted at a senior level, across verticals, and across business & IT roles. All enterprises had at least 500 employees, though a majority had a much larger employee base. For full survey results, click here.
The study is pretty remarkable for its findings. Here are my top 5 observations.
- Interest: A lot more enterprises are actively exploring the cloud for big data than the perception may be.
- 25% are actively exploring how to leverage the cloud, while 28% are already in the cloud. That is a total of 53% of enterprises active in the cloud.
- 13% said they have no intention to move to the cloud (like my erstwhile CTO friend above!) BUT 40% of this group said they would reassess their position once they had a deeper understanding of the cloud security posture.
- Drivers: Agility & cost reduction were primary drivers for moving to the cloud. While roughly half of the audience was business (vs. IT), remarkably both business and IT were fairly aligned across these drivers.
- Cost – 47%
- Agility – 31%
- Other drivers: customer demand, mobility, new apps, divesting infrastructure to focus on core competence
- Amazingly, 14% said they felt the cloud would be more secure than their enterprise! Interestingly, the higher the title, the more confident they felt about cloud security. At the President/CEO level, 33% felt the cloud would be more secure. I am not sure if this is a state of delusion or enlightenment!
- Workloads: Data Science gets all the press but the majority of enterprises are actively exploring how to leverage the cloud for existing and traditional analytics. Batch processing and real-time processing each came in at 43%, while data science was only 17%.
- Data: By the way, enterprises are not just thinking about playing in the cloud; they are eager to see it as a production platform. Fully 56% of participants said they would have more than 10 TB in the cloud, while 20% said they’d place more than 100 TB in cloud systems.
- Concerns: The top 3 concerns about moving critical big data processing to the cloud were security, privacy & complexity.
- 63% were concerned about security & breaches
- 39% said privacy & regulations were an obstacle
- 23% were concerned about complexity, expertise, and data movement
However, 55% of all participants said they would reassess their position on the cloud after better understanding the security posture of the cloud. In general, participants felt they needed more education around security & privacy capabilities. They also felt the need to reduce the complexity of the cloud.