We certainly live in challenging times. An uncertain economic future due to a global coronavirus pandemic is affecting our societies and organisations, and businesses will increasingly be looking to reduce costs and become more efficient and productive. Every department, including IT, is trying to make savings and improve processes, which is a positive response especially for ensuring future competitiveness.
In this guide, we’re going to take you step-by-step through the required actions to take for implementing a successful and modern data warehousing solution. To help illustrate important steps in the journey, we’ll include key points from a real-life Antares customer story.
Our customer, GWA Group, a building supply firm, was grappling with a range of IT and data issues. They had a data warehouse in place, but it wasn’t scalable, was plagued by multiple bugs and issues, and suffered from very slow data processing. As a result, IT staff were inundated with time-consuming requests, while the business was being held back by reduced efficiency and productivity.
Fortunately, by working closely together, we were able to help them achieve much faster data processing, better data accuracy, and productivity gains – all without significant impact to their operations. Let’s take you through this journey now.
For definition purposes, a data warehouse is a single, centrally managed database that collates various information sources – internal and external. Unlike purely operational or transactional databases, a data warehouse contains a larger volume of historical information, stored in one place to help enable deeper analysis, forecasting, reporting, and business decision making.
Kick off your project with a comprehensive documentation of your existing environment. Describe how your current solution is operating – what it does well, and where it falls short to identify where the issues lie. Make sure you have input from every department in the company to understand exactly how they are using data and what pain points people are encountering.
Working with our client GWA, we entered into a “discovery phase”, where we investigated their current data warehousing solution. Our aim was to investigate and document how it functioned and unearth the root causes of key issues. GWA’s warehouse had evolved incrementally over time, and as a result, lacked structure and often experienced performance failures, such as failing to load. Despite upgrades, there were ongoing concerns around accuracy, which meant the data was not reliable enough to inform planning and decision making.
The next phase of your project is to set specific and measurable goals that describe exactly what you want to get out of your new data warehouse. For example, is data variety or accuracy a key goal, or perhaps you need shorter time to reporting to enable better decision making? In any case, decide your objectives so everyone is working with the same goals in mind.
The Antares data team worked closely with GWA business and technology leaders and various team members in order to fully understand the project’s goals. This allowed us to document a shared vision for successful outcomes, and forecast with greater accuracy the metrics which could be used to measure the end solution result.
It’s now time to map out your strategy for achieving your goals. Start by asking what the relevant data sets are that you will need to store and analyse, and match them to the correct source points. This may be operational systems, external information, financial systems, and so on. Based on your organisation’s infrastructure and existing systems, decide whether you are best served by an on-premise, cloud, or hybrid solution to facilitate your data strategy.
Applying the above methodologies, the GWA and Antares team devised a clear and actionable strategy to resolve data issues and produce greater value from data into the future. A shift to a structured enterprise data warehouse was central to the action plan. And with two workable options for either a hybrid or all-cloud approach, GWA opted for a hybrid solution.
Now is the time to prototype the solution by mapping the end reports, visualising what they will look like, and what problems they will help solve. Without this step, it’s harder to plan, design, and implement a data solution of value that addresses the real, day-to-day issues your organisation faces. Start by building reports and dashboards based on experimental data you have imported. Test with users and assess whether they are raising query performance issues.
The Antares and GWA teams worked together to prototype the end state and desired solution outcomes, focusing on delivering more valuable system reporting, data accuracy, and business planning. To launch, the new enterprise data warehouse project would start by serving the commercial management area of the business.
Create a detailed architecture that maps your new proposed system in its entirety, including technologies, connection points, processes, and data sources. Like a blueprint, a data model helps you gain a common understanding of data sets, and creates a data flow diagram to show how you want the data to travel through your organisation. Document the best ways for modelling data from your new system with various business inputs, such as sales transactions or historical service records.
This project started by sourcing data from multiple source systems including: com, MS SQL Server, IBM DB2, API, and several other file sources used across the business. Data from these systems were integrated within the data vault layer so they could be consumed by a presentation layer modelled with Kimball Methodology. The presentation layer was then exposed to business intelligence tools, including Jedox, for commercial management reporting.
This step is an extension of your new data warehousing strategy, architecture, and data modelling. You want to model the transactions that need to take place between your nominated data sources to ensure your systems and infrastructure are sufficiently robust to handle the
volume and types of transactions.
Throughout the project, Antares carefully defined and embedded strong enterprise data warehouse governance to prevent issues that GWA had previously experienced with its corporate data warehouse. For example, we applied a structured approach to working with multiple layers to ensure that data flowed seamlessly from one layer to another.
When you begin your build, you want to pay attention to applying a good Extract, Load, Transform (ELT) approach. This is the process you’ll use to pull data out of your current stack into your new data warehouse for analysis. A poor ELT process will slow you down, but a good one will optimise your speeds and ability to build easy, replicable data pipelines between your existing architecture and your new data management and analysis tools. This step is all about creating value for your organisation.
Using a best-practice approach to ELT, our team got to work. We involved GWA on this step, updating them throughout the process, particularly through creating user stories around data flow, allowing them to see whether all aspects of the project were remaining on track.
An IP accelerator can help significantly with your data warehouse build, so use one if you can. All based on automation, it fast-tracks a project’s development side, automatically profiling source systems, importing metadata, and generating the packages and procedures needed to move key data sets automatically between all of your assigned layers.
By applying the Antares Accelerator (our internal IP accelerator tool) to this project, we were able to deliver the technical requirements faster, cutting costs on the build and saving substantial amounts of time, to deliver on budget. An automated, meta-data driven approach, our IP accelerator added new value to the project, shifting emphasis from the technical development, allowing for more focus on GWA’s business processes, objectives, and outcomes.
For speed and flexibility, it’s a good idea to deliver your new data warehousing project using agile methodology. This approach helps culturally, keeping all stakeholders on the same page with regular communication to ensure that each deliverable meets peoples’ requirements and expectations.
As a large program of work, GWA’s new enterprise data warehouse was delivered using the agile methodology to ensure value was delivered fast across the organisation. To enable agile methodology success and ensure frequent outcome delivery, we applied our ‘enterprise data warehouse framework’, which is fully automated, meta driven, and designed to culturally align with each client’s environment.
Before you go live, it’s critical to test your new system with the people who will be using it every day. Will the system or application crash? Will it use only the correct amount of system resources? What is the load time like? Will users find the solution easy to use? Among the last phases of the testing process, UAT is essential for reducing both time and cost, whilst increasing customer satisfaction.
This UAT phase was critical as the final step in the GWA data warehousing project, helping our team to iron out a number of final processes and data flows to ensure the solution was robust for all stakeholders.
You’ve set goals, decided on strategy, mapped architecture and data flows, tested and finally delivered the project. Now it’s time to go live and launch your new data warehouse.
Be sure to keep iterating, improving, and enhancing your new system as you proceed.
After 6 months of close collaboration and development, we went live with GWA’s new data warehousing solution. One of the most important elements of launching was ensuring security in the live environment was of the highest standard. To maintain robust security in the cloud, we enabled security and firewalls and provisioned the database in an Australian data centre for maximum data security.
With help from Antares, GWA now manages and operates a faster, more efficient, and more intuitive data warehousing and analytics solution. Data now takes the GWA team a fraction of the time to process, despite having deeper layers of complexity. The company’s data can now also be accessed faster and more easily, while the data itself is higher quality and more accurate.
This means better strategic decisions are now being made every week as a result. Furthermore, the productivity of the team has also improved significantly as team members no longer have to wait a whole day for IT to get back to them with reports and data.
If you’d like to find out how your organisation can achieve its data management and analytics goals, reduce costs and make more strategic, data driven business decisions – Antares is holding a free, virtual workshop “Be a More Data-Driven Organisation.” In it, we’ll show you how to use data to gain clear foresight, improve operational efficiency, uncover revenue opportunities, strengthen existing customer loyalty, improve organisational responsive, and much more.