DataOps: Still an unsolved challenge for many organizations

In our digital age, everyone has understood the importance of data—collecting information, sharing it across the organization, and using it to drive business-critical decisions. But many organizations still don’t know how to do these things well enough and fast enough. The unfortunate reality is that some valuable data is never shared with the right people at the right time. Worse, decision makers can’t always be sure that the data they rely on is accurate and up to date.

It is possible to build new technical processes and shape cultural expectations to successfully manage data. In fact, the software world has already achieved this goal with its development processes and could serve as a model for Dataops today. Data managers are just beginning to elevate their “product” to the same high level of organization, quality, and trust.

A uniquely valuable asset

Data is “a driver of growth and change” in the 21st century^St Century, just like the oil was 100 years ago economist writes – and it is just as valuable. “Data streams have created new infrastructures, new companies, new monopolies, new policies and, crucially, new economic systems.”

In a data-driven world, everyone is realizing that data is no longer just a by-product of important business processes, but a core asset with unique value. Any company in any industry can use its data to find and retain new customers, improve customer brand experiences, study sales trends, and refine marketing strategies.

But not every company makes full use of its data today. According to a survey by Deloitte Analytics, data management is hampered by three common challenges:

• Low data quality (only 34% of survey respondents considered this data to be “excellent” or “good”, defined as integrated, accurate, and centralized)
• Lack of analytics technology (49% had only basic reporting tools and limited predictive analytics tools)
• “Power struggles” over data ownership stemming from insufficient executive leadership (38% reported localized analytics with limited sharing of tools, data, and people; 20% reported “uncoordinated niches” of analytics efforts)

Still, 49% of participants agreed that analytics improved their decision-making skills. “Basically, analytics is about making good business decisions,” an analytics director told Deloitte. “Just giving reports with numbers doesn’t help. We must provide information in a way that best suits our decision makers.”

Finding balance in a flood of data

Organizations are overwhelmed by the “deluge of data” that is poured into them on a daily basis. The burden of data management falls mostly on IT teams and specialized data teams, who are typically the only people trained to analyze data. Even when new specialists come on board to reinforce a team, the learning curve is long and slow.

It doesn’t help that teams aren’t unified, and neither is the data they consume. Information comes from many sources, not always easy to trace, and ends up in multiple silos. Different teams manage slices of data without consistent processes and use different tools. The simple need for better tools all around further complicates the effort. Sometimes it seems like data just disappears into an inexplicable black hole.

According to the Deloitte survey, truly insight-driven companies — those who base decisions on data — are in the minority today, and “the most common culprit is culture.” The study concludes that “purchasing and using analytics tools is not difficult – changing behavior is”. The most fundamental change is the “democratization” of data, which means training a wide range of employees in analytics and empowering them with tools that even non-experts can effectively manage.

Finding the right balance between changing technology and changing culture is critical for data managers. It’s also hard to do. A Forrester Research study found that 88% are neglecting “either their technology and processes, or their culture and skills—or both.” Only 12% say they have achieved a workable balance between culture and technology by learning to focus on both without forgetting either. Forrester calls these rare organizations “data champions.”

What are best practices?

As defined in a Gartner glossary, dataops introduces enterprise-wide collaborative data management to improve “communication, integration and data flow between data managers and data consumers”. dataops automates the design, delivery and management of data delivery in a “dynamic environment” and uses metadata to increase the value and usefulness of the data. By ensuring “predictable delivery and change management of data, data models and related artifacts,” dataops promises almost by design to adopt best practices.

Organizations can derive even more value from their data by deploying an advanced data lineage platform that provides an automated, granular, and multi-dimensional view of the data journey. This gives data managers—and other users, from executives to the front lines—a comprehensive view into the flow of data, with the ability to map where data is coming from and where it’s going, from its original source through to reporting and analytics. This type of lineage uses automated and advanced methods to create a comprehensive cross-system view of all of an organization’s data, no matter what silo it resides in, including all data flows and dependencies.

And, perhaps most importantly, the Data Lineage solution empowers anyone to become their own data expert. Anyone can view the entire data landscape on one screen, pull data from any source onto an automated platform, and do it themselves without IT or data specialist support. This is the answer to uniting many teams with new tools that work for everyone and unifying data on a transparent, trusted platform. Data teams also benefit from an added benefit – freedom from repetitive manual tasks.

A source of truth

Businesses today may be overwhelmed with data, but they want it to keep coming. According to Forrester, data-driven decision-makers have a growing “thirst for data,” even as they struggle to absorb their data. “Seventy percent of data decision makers are gathering data faster than they can analyze and use it, but 67 percent say they consistently need more data than their current capabilities.”

Add to that the well-known fact that “the cost of bad data” costs the world $3.1 trillion each year, and it’s not hard to imagine why organizations are considering dataops and accelerating the journey to best practices. All of this will depend heavily on their ability to establish an enterprise-wide source of truth. That means the data every team wants is always there, always clear, and always trusted—no more black holes.

And it’s good to remember that we all have the means to become data-driven, insight-driven companies from top to bottom, across all teams and ventures.

Yael Ben Arie is CEO of Octopai.