Data engineering is a line of activities from designing, implementation, to operation data systems that are purposed to extract maximum value out of data asset. Until recently, data systems have been used for multiple goals such as reporting, dashboard, and statistical analysis. However, coming in the Big Data era, they are now under heavy pressure of change as volume and variety of data to process is increasing and demand for data gets inflated and complicated.

Data warehouse, a data repository designed to store required data from various sources, opened up the early days of data systems. Unlike OLTP (Online Transaction Processing) DBMS, data warehouse is suitable to process large data in bulk and were swiftly adopted by many organizations.

Data warehouse is a central piece of the system. ETL (Extract, Transform, Load) tools connect to multiple data sources, transform gathered data, and load them in data warehouse. BI (Business Intelligence) and OLAP (Online Analytical Processing) tools enable users to analyze and visualize data stored in DW.

But this architecture begins to show weakness in Big Data era.
In order to overcome these shortcomings, a number of technologies and products emerged in the early 2000 opening up the Big Data era.

Big Data, in its simple definition, refers to a phenomenon where digital data explodes both in volume and variety. Data systems were required to go in tandem with this trend by ingesting various data formats, processing large data timely, and allowing users to freely access necessary data so that they can develop business solutions through experiments. These are major aspects of big data platform, a data system of today.

Big data platform has different architecture compared to its predecessor containing new components so as to accommodate these new requirements. The above diagram is one typical example.

Since big data platform is a complex mixture of many components, data engineering of today necessitates not only technical understanding in each component but also comprehensive view over the entire workflow to integrate all the components.

Softline has been a dedicated data service company since its foundation. As a group of data experts, our thorough expertise in data systems is our ground to provide high-level data engineering service for big data platform.

Softline Data Engineering Service