DataOps Principles: How Startups Do Data The Right Way

If you have been trying to harness the power of data science and machine learning — but, like many teams, struggling to produce results — there’s a secret you are missing out on. All of those models and sophisticated insights require lots of good data, and the best way to get good data quickly is by using DataOps.

What is DataOps? It’s a way of thinking about how an organization deals with data. It’s a set of tools to automate processes and empower individuals. And it’s a new DataOps Engineer role designed to make that thinking real by managing and building those tools.

DataOps Principles

DataOps was inspired by DevOps, which brought the power of agile development to operations (infrastructure management and production deployment).  DevOps transformed the way that software development is done; and now DataOps is transforming the way that data management is done.

For larger enterprises with a dedicated data engineering team, DataOps is about breaking down barriers and re-aligning priorities. For smaller startup teams like ours, DataOps enables the tackling of large and complex data problems that previously were beyond reach.

There have been some other takes on the principles that make up DataOps, but this is our list:

• Self-Service Data Over Data Requests
• Autonomation Over Manual Processes
• Frequent Small Changes Over Infrequent Large Changes
• Reproducibility at All Levels
• Data Scientists / Analysts and DataOps Engineers Must be on the Same Team
• Value From Data is the Primary Measure of Progress
• Continuous attention to Technical Excellence and Good Design
• Simplicity As a Design Requirement
• The Best Architectures, Requirements, and Designs Emerge From Self-Organizing Teams

Full article >>>