Everyone has moved their data to the cloud – now what?

Businesses of all shapes and sizes are increasingly understanding the need to continuously improve competitive differentiation and avoid falling behind the world’s digital-native FAANGs – data-centric companies like Google and Amazon have used data to dominate their markets. Additionally, the global pandemic has stimulated digital agendas, data and agile decision-making for strategic priorities that are dispersed across remote workplaces. In fact, a Gartner Board of Directors study found that 69% of respondents said COVID-19 has prompted their organization to accelerate data and digital business initiatives.

Migrating data to the cloud is nothing new, but many will find that cloud migration alone will not magically transform their business into the next Google or Amazon.

And most organizations find that after migration, the latest cloud data warehouse, lakehouse, fabric, or mesh no longer helps harness the power of their data. A recent study by TDWI Research of 244 companies using a cloud data warehouse/lake found that a staggering 76% experienced most or all of the same on-premises challenges.

The Cloud Lake or Warehouse only solves one problem – accessing data – which, while necessary, is not a data usability solution and definitely not at absolute scale (which gives the FAANGs their “byte”)!

The usability of data is key to enabling truly digital businesses – businesses that can access and use data to make every product and service hyper-personalized and create unique user experiences for every customer.

The road to data usability

Data is difficult to use. You have raw information filled with errors, duplicate information, inconsistent formats and variability, and disconnected systems.

Moving data to the cloud simply shifts these issues. TDWI reported that 76% of organizations acknowledged the same on-premises challenges. They may have moved their data to one place, but they are still riddled with the same issues. Same wine, new bottle.

After all, the ever-increasing amounts of data have to be standardized, cleaned up, linked and organized in order to be usable. And to ensure scalability and accuracy, this needs to be automated.

Only then can companies begin to uncover the hidden treasures, new business ideas and interesting connections in the data. In this way, companies can gain a deeper, clearer, and more comprehensive understanding of their customers, supply chains, and processes, and convert them into monetizable opportunities.

The goal is to establish a central intelligence unit at the heart of which are data assets – monetizable and easily consumed layers of data from which the business can extract value when needed.

Easier said than done given the current obstacles: highly manual, complicated and complex data preparation implementations – namely because there is not enough talent, time or (the right) tools to handle the scale required to prepare data for the prepare for digitization.

If an organization doesn’t run in “batch mode” and data scientists’ algorithms rely on constant access to data, how can current data prep solutions running on routines run once a month survive? Isn’t the promise of digitization to send every company to the full at any time and anywhere?

Additionally, few companies have enough data scientists to do this. Research from QuantHub shows that there are three times as many data scientist job postings compared to job searches, leaving a current vacancy of 250,000 vacancies.

Faced with the twin challenges of data scale and talent scarcity, organizations need a radically new approach to achieving data usability. To use an analogy from the auto industry, just as BEVs have revolutionized how we get from point A to B, advanced data utilization systems will revolutionize the ability for any company to create usable data to truly go digital.

Solving the usability puzzle with automation

Most see AI as a solution to the decision side of analytics, but the biggest discovery of the FAANGs was the use of AI to automate data preparation, organization, and monetization.

AI must be applied to the core tasks of solving data usability – to simplify, streamline and enhance the many functions required to create, operate and maintain usable data.

The best approaches simplify this process into three steps: ingest, enrich, and distribute. Algorithms correlate data from all sources and systems in speed and volume for ingestion. Second, those many floating bits are concatenated, assigned, and fused for immediate use. This usable data must then be organized to enable flow and distribution across customer, business, and enterprise systems and processes.

Such an automated, scaled, and end-to-end data utilization system frees data scientists, business professionals, and technology developers from tedious, manual, and vulnerable data preparation while providing flexibility and speed as business needs change.

Most importantly, this system allows you to understand, leverage and monetize every tiny bit of data on an absolute scale, enabling a digital business that can match (or even beat) the FAANGs.

Ultimately, that’s not to say that cloud data warehouses, lakes, fabrics, or whatever the next hot trend is going to be is bad. They solve a much-needed purpose – easy access to data. But the path to digitization does not end in the cloud. Data usability at scale will put a company on the path to becoming a truly data-centric digital company.

Abhishek Mehta is Chairman and CEO of Tresata