Getting started with data warehouse modernisation

Data Warehouse

Simon Spring, Account Director EMEA, at WhereScape outlines how we should begin our journey modernising our data warehouses.

In the last few years, data automation technology has really ‘crossed the chasm’ into the mainstream. Leaders no longer see automation as a nice-to-have luxury, but a necessity to compete in an increasingly data-centric market. A report by IDC predicts that by 2025 data creation will grow to an enormous 163 zettabytes (ZB), ten times the amount of data produced in 2017.

In anticipation of the surge in the amount of data that consumers and businesses are creating, data warehouse development is under pressure to modernise. We will start to see automation and machine-to-machine technologies shifting the bulk of data creation away from traditional sources. Consequently, organisations should start to consider realigning current business goals, provisioning data for existing and future business cases, and leveraging new platforms and data-driven tools.

Data is beginning to be viewed holistically with a company-wide strategy, and more often than not automation holds the key.

An ‘automation first’ philosophy

Automation is becoming central to this process, because it provides the modern tooling required for data warehouse design, development and administration. With an ‘automation first’ philosophy guiding data warehouse development, developers can fix outmoded development methodology and practices. In doing so, it becomes possible to address the shortcomings of traditional approaches where productivity, flexibility, reuse and adherence to standards are limited.

This isn’t just about how much data can be stored and processed or how quickly meaning can be derived from it. Adding data warehouse automation software to the mix can deliver even more efficiencies and value in a much faster timeframe than hand-coding or using native tools without automation –  in fact, it simplifies development to minimise both effort and risk in data integration and infrastructure projects. This allows companies to focus their effort and resources on providing analytic value to the businesses.

The key infrastructure enabler, however, is the adoption of cloud data warehousing as a solution to cope with growing data volumes. A cloud data warehouse is a database service hosted online by a public cloud provider. It has the functionality of an on-premises database but is managed by a third party, can be accessed remotely and its memory and compute power can be reduced or increased instantly.

In complete contrast to legacy approaches, these cloud-based services have empowered organisations to pay only for the resources they need. As such, data warehouses built on cloud infrastructure offer a foundation for modernisation in businesses committed to data-driven excellence.

Cloud data warehousing brings additional benefits to analytical data infrastructures, from agility and cost-effectiveness to scalability and performance. And specifically, building a cloud-based data warehouse with the benefits of automation enables end-users to design or prototype new analytic components without having to spend large sums of money on infrastructure. This fast-tracks new infrastructure projects and increases development and operation capabilities.

What to consider when selecting a cloud data warehouse

Choosing a cloud data warehouse solution should begin with a cost analysis to estimate how much money it could save the business. Different cloud providers have different pricing structures, with more established names, such as Amazon and Microsoft, renting out nodes and clusters, so every user has a defined section of the server. This makes pricing predictable and constant, but can be a disadvantage in that shared servers sometimes require maintenance.

Other modern cloud-based data warehouse platforms, such as Snowflake, provide elastic compute functionality allowing companies to ensure that they can easily adjust where and how the resources, such as costs, are being used. The ultimate goal is to make the design, development, deployment and operation of data warehouses quicker and cheaper, so teams can deliver projects in hours and days.

Each cloud provider has its own suite of supporting tools for functions such as data management, visualisation and predictive analytics, so these needs should be factored when deciding on which provider to use.

Understanding the benefits and options available in modernising the data warehouse, organisations can create effective strategies that not only optimise the selection of tools and data migration processes but build effective coordination between teams and stakeholders. Modernising the data warehouse and looking to the cloud can have a transformational effect; supporting businesses in the influx of big data, automating manual processes and maximising the return on cloud investment.

Bekki Barnes

With 5 years’ experience in marketing, Bekki has knowledge in both B2B and B2C marketing. Bekki has worked with a wide range of brands, including local and national organisations.

Ab Initio partners with BT Group to deliver big data

Luke Conrad • 24th October 2022

AI is becoming an increasingly important element of the digital transformation of many businesses. As well as introducing new opportunities, it also poses a number of challenges for IT teams and the data teams supporting them. Ab Initio has announced a partnership with BT Group to implement its big data management solutions on BT’s internal...

WAICF – Dive into AI visiting one of the most...

Delia Salinas • 10th March 2022

Every year Cannes held an international technological event called World Artificial Intelligence Cannes Festival, better known by its acronym WAICF. One of the most luxurious cities around the world, located on the French Riviera and host of the annual Cannes Film Festival, Midem, and Cannes Lions International Festival of Creativity. 

Bouncing back from a natural disaster with resilience

Amber Donovan-Stevens • 16th December 2021

In the last decade, we’ve seen some of the most extreme weather events since records began, all driven by our human impact on the plant. Businesses are rapidly trying to implement new green policies to do their part, but climate change has also forced businesses to adapt and redefine their disaster recovery approach. Curtis Preston,...