Member-only story
Evolutionary Data Infrastructure
From Monolith to Self-service Platform
All systems start as a small monolith. In the beginning, when resources and manpower are not sufficient, the monolith is the choice we have to make, even the data infrastructure is no exception.
But as requirements increase, there are more and more scenarios that cannot be achieved by the current architecture, and the system must therefore evolve. Each time the system evolves, it is to solve the problems encountered, so it is necessary to understand the different aspects that need to be considered, and to use the most efficient engineering methods to achieve the goal.
In this article, we will still start with a monolith, as we have done before.
But this time, our goal is not to serve a production environment, but to provide the data infrastructure behind all production environments.
A data infrastructure is a “place” where all kinds of data are stored, either structured data or time series data or even raw data. The purpose of this big data (and they are really big) is to provide material for data analysis, business intelligence or machine learning.
In addition to internal uses, there may also be user-facing functions, for example, a list of…
