Big Data DevOps Costs: Market Data

We've been collecting more data about the costs of big data DevOps and the costs of running your own big data platform as a service (PaaS) in the cloud. We'll be sharing that over the next few months. One interesting area of research was the cost of assembling the team required to build and run your own Hadoop/Spark PaaS in the cloud. To do this, we identified the basic skills required for a big data team to operate a PaaS, and then researched available jobs online to get a sense of the job market. The available jobs and salary ranges are significant, representing the diversity of interest and companies in this area. 

Why are we so interested? Cazena's Big Data as a Service is fully-managed and automated with DevOps built-in. That means that companies don't need to hire a team to build and operate the platform. Everything, including automation and big data DevOps, is included with Cazena's subscription. That represents a massive savings in salaries and administration time for operations, and allows companies to focus resources on more strategic activities. 

The goal of this project was to help teams think through the costs of running their own platform, and whether it's really worth it, given the alternatives. The job search site Indeed.com tracks a lot of data and offers interesting analytics. Indeed tracks salary averages for common job categories, including salary lows and highs. They also track number of available jobs, which gives a sense of the competitiveness of some of these positions. Using data from Indeed.com's website, we put together this infographic on the roles and salary ranges required to run a big data platform in the cloud:

Big Data Team Salary Research - Cazena.com

To be exact, the range is $629,000 to $1,186,000 for annual team salaries for Big Data DevOps. We also noticed that some of these have a huge number of roles available, indicating high demand. And some roles, such as DevOps, have an average tenure of less than a year, according to Indeed.com's summary. That indicates the high potential risk of turnover.  Now, markets change, so we've included links, so you can click through for current data on your region. When you search a job, look at the bottom right to see the salary averages. Click through and check out the vast range of salaries depending on company size, industry and location. We pulled numbers within the last week, but you can use our links below to click through and get current data. 

Salary Ranges of Big Data DevOps Team

Cloud DevOps
*Salary range: $122,980 to $230,000

Description: The Cloud Development Operations Engineer, more commonly known as DevOps, is responsible for administering cloud accounts and resources, and managing the cloud infrastructure.

View current salaries & job listings on Indeed.com
*Salary range, averages and max, for Development Operations Engineer category from Indeed.com, May 2018. 

 

Hadoop Platform Administrator
*Salary range: $118,312 to $225,000 

Description: The Hadoop Administrator will provision and tune Hadoop/Spark nodes, with attached data stores and centralized object store required to deliver workload performance. This will be needed for open-source or commercial Hadoop software from Cloudera, Hortonworks or others, or cloud Hadoop solutions, such as EMR.  

View current salaries & job listings on Indeed.com
*Salary range, averages and max, for Hadoop Developer category from Indeed.com, May 2018. 

 

Cloud Security Architect
*Salary range: $143,769 to $225,000 

Description: The Cloud Security Architect will oversee all aspects of security, from platform to network to data. They administer security controls such as encryption, key-management, identities and role-based access control, as well as establish and ensure compliance controls. This requires a comprehensive knowledge of security protocols, and will require specific knowledge relative to the deployment style (AWS, Azure.) 

View current salaries & job listings on Indeed.com
*Salary range, averages and max, for Cloud "Director of Information Security" from Indeed.com, May 2018. 

 

Data Management Lead
*Salary range: $137,113 to $253,000

Description: Responsible for managing and administering data ingestion, data governance and logging as well as managing user access from a variety of data engineering, machine learning and SQL tools.

View current salaries & job listings on Indeed.com
*Salary range, averages and max, for Big Data "Senior Solution Architect" from Indeed.com, May 2018. 

 

Data Production Operations
*Salary range: $111,958 to $216,000 

Description: This role will cover first and second-line alerting, support, root-cause analysis and upgrade/patching/validation issues. This is also a catch-all capability required for technical issues like sprint tracking, billing, SLA monitoring and management.

View current salaries & job listings on Indeed.com
*Salary range, averages and max, for "Data Engineer" from Indeed.com, May 2018. 

 

Summary: Big Data Teams May Vary, But Still Pricey

Obviously, teams will vary depending on the size of the company, cluster and location. Note that the sample above is fairly conservative, assumes a medium size cluster and does not include any of the people needed for actual analysis and data science. This is simply the team required to operate the PaaS -- you may even need a few more for the setup (depending on how fast you want to get started) or more for larger global deployments. 

You'll need to plan for both development, deployment and ongoing operations. The basic big data team roles will require some crossover training, for 24x7 support, and additional resources would be required for major upgrades. In corporate environments, you may have access to additional resources from other IT or security groups, but beware of splitting platform roles between too many teams or people, as that may get risky for security. 

There are alternatives.

Cazena is the First Fully-Managed Big Data as a Service with DevOps Built-in, so you don't need to hire a team just to run your PaaS. Cazena provisions secure cloud environments in just a few hours, ready for data loading and analytics. That means you can start quickly, scale spending with platform use and save 50%+ over the cost of assembling and managing your own team. To learn more about what's included, read about "Fully-Managed (Big Data DevOps Built-in) vs. Big Data PaaS" or get in touch to talk more: info@cazena.com