Digitalization provides access to large quantities of highly detailed data, which today is seen as “the new gold”. The value lies in its scope for analysing data and ensuring a better decision-making basis. However, large quantities of data also involve big challenges: How should we obtain, store, make accessible, use, structure and secure our data in the best possible way?
Data Management is all about how you can best exploit the full potential of your digital gold.
Interest in Data Management is increasing as more and more companies get under way with advanced Business Intelligence operations, such as Self-Service BI and predictive analysis with Data Science/Machine Learning. This in turn has resulted in an increased need for data from multiple sources – and a need to ensure that the right data, of maximum quality and the right level of detail, is used in these analyses.
The most common problems are that data is divided into several different source systems, and that they do not communicate properly with each other. The traditional solution has been to build a data warehouse based on a server in a server room, and to subsequently fetch, convert and store data from these data sources in the data warehouse. The relevant data is then available for reports and analyses from that location.
However, a lot has happened in recent years and right now Data Management is undergoing major changes. Trends such as cloud-based data, Self-Service BI and the demand for proactive, predictive or even prescriptive insights have all increased the pace of change in many companies.
Today even traditional operations are investing in analysis departments staffed by data scientists and data analysts. Data architecture and data management solutions are thus becoming increasingly important. Among the questions that need to be resolved: How should the model be structured? How and for whom should the information be available? What should the solution look like so that the information is as correct as possible? How quickly can I obtain the information?
The new challenges, and at the same time the opportunities that companies and organisations are facing, have changed the perception of data and how its potential can be fully exploited.
Many Data Management projects have collapsed owing to an excessively rigid process or manually coded data warehouse. It takes a long time to write code and, what is more, the code may be difficult to administrate over a long period of time as the number of solutions continues to grow. In order to speed up the development process and implementation, and at the same time create a uniform structure, various technologies have been developed in recent years. You can regard them as a kind of Best Practice for a secure and reasonably future-proof Data Management platform.
BIML – the first step away from manual coding
BIML is a script language based on XML that is designed for DW development. Instead of building on manual routines, tables, views, procedures with code and/or SSIS packages, generic script is developed in BIML which can then be run on different platforms. These are utilised primarily at the start-up of a data warehouse project or when it is time for an upgrade or change of the platform on which the data warehouse operates.
Instead of manually writing code or using BIML script, the data warehouse can be modelled and built using a Data Warehouse Automation tool. Objects, relationships and operations are modelled in a graphic interface, after which the tool automatically generates necessary objects on the database platform that forms the foundation. This saves a lot of time in projects, and administration is made simpler in the longer term since all objects and all logic are built the same way, no matter who the builder is.
As your Data Management solution is filled with data, there is an increasing need to be organised. In order to retain your solutions and their contents over time, the contents need to be documented and catalogued. This also makes the contents searchable.
Moreover, metadata (data about the data) needs to be updated so its relationships to other objects can be traced. This type of relationship traceability is generally known as Data Lineage, with the objects catalogued in a Data Catalog. Some tools have built-in functions for Data Lineage, while others require that you build this manually. Irrespective of method, it is worth the investment to do this.
New modelling techniques have therefore been developed for greater flexibility and agility than before. We can see that the Data Vault modelling concept is gaining ground and is now a serious alternative to the classic Kimball modelling approach. This is particularly apparent where the operation’s regulatory framework changes over time and there are high demands on traceability.
Data volumes are expanding all the time and in general they are doing so at an ever faster pace. In this context, the guiding principle is to not store the same data multiple times. This has driven the creation of Data Virtualization (Data Sharing and Data Cloning) – techniques that can spread data without duplicating or moving it. Data Virtualization does not replace a data warehouse, but it can be a supplement that supports various models in a Business Intelligence framework.
There is often a traditional BI solution with readymade reports, but for more agile reports via a Self-Service BI tool Data Virtualization can be part of a solution. In a Data Virtualization tool new data can be added quickly and simply, for instance to test a theory or conduct an analysis on a one-off basis.
This far we have in principle only examined various technical solutions in the form of systems and tools. However, Data Management itself has no intrinsic value. It is only when the right data of the right quality is put to practical use through various reports and analyses that any value is created. A good Data Management platform must therefore be built on the strategies, business models and control models that form the basis of your operation. Nonetheless, these should be queried and if necessary changed so as to better exploit the power of your data. This may also apply to existing competence, working methods and organisation.
Master Data Management consists of the definitions, processes, divisions of responsibility and technical platforms used for creating and maintaining master data. Master data is the type of data that is standardised for a company or a public authority. Examples of such information include customers, organisation numbers, employees, ID numbers, suppliers, partners, contracts, accounts or products. Such data is often shared by many departments, business processes and IT systems.
Master data is particularly important data to which we wish to ascribe descriptive terms, such as ID numbers where we want descriptive information such as first name, surname, address, phone number and customer number. An ID number is master data to which the attribute belongs, and the master data together with the attribute constitutes a master data domain. We can work in a similar way with products, customers, employees, ledgers and so on. Master data is often a prerequisite for being able to perform other actions such as successful digitalization, self-service BI of various kinds, reporting, conforming to legislative requirements (for instance GDPR) and data science; if the quality of the data is not good, these initiatives will not succeed.
Data Management is a complex area that encompasses several different sectors. Our specialists here at Advectas will help you see the complete picture – everything from starting up a small, traditional data warehouse to developing unique strategy documents and roadmaps for large projects.
Choosing the right Data Management platform in a fast-changing world requires specialist competence. The same applies to implementation of the necessary projects. Over a period of many years Advectas has built up the necessary expertise and experience. Our customers include many of Sweden’s largest companies and organisations.
Advectas is also independent in terms of its product solutions, always recommending what is most suitable for particular requirements. Having said that, we work together with a number of strategic partners – specialists whose products are at the forefront of their respective areas as regards development, maturity and market presence.
For example, taking a decision to move all or part of your company’s data to the cloud is a process encompassing many different aspects that must be taken into account. Using tried and tested methods, we support and secure the entire process from initiative and POC to full-scale solution. Over the years we have helped customers with both creation of new Cloud Data and Data Warehouse/Data Lake solutions as well as “lift and shift” of existing on-premises solutions to the cloud.
When it comes to Cloud Data Warehousing, Advectas partners with Amazon, Microsoft, Google and Snowflake, Matillion and Birst. All of them are built to be able to harness all the notable cloud benefits. This area is undergoing swift change and we are constantly at the forefront, which may generate additional partnerships in the future.
Irrespective of the individual Data Management area, we always aim to ensure that all our consultants are at the cutting edge in terms of their knowhow as new technologies and new areas of use arise and other areas transition to maturity.
Unlike many other players in this area, Advectas has expertise in the entire Business Intelligence sphere. This means that we have specialist competence not only in Data Management but in everything related to it: budget and planning, consolidated reports, Self-Service BI, Data Science and so on. This means we can deliver Data Management solutions that are optimally tailored for your specific BI needs.
Advectas also has specialist Management consultants with considerable insight into BI and Data Management. This is a virtually unique resource that can contribute strategic advice and practical support in terms of the prerequisites and organisational abilities needed to work in a perceptive manner. Read more about Advectas Management Consulting.