Elasticsearch Index Lifecycle Management in a Nutshell
Understanding its Application Scenarios and Limitations
When we develop data-intensive applications, we usually classify data into frequently used and infrequently used, i.e. hot data and cold data. We have different ways of handling data at different “temperatures”, for example, cold data is stored in lower-cost storage, but with a lower access performance.
If the databases we use have this kind of temperature management built in, then the operation effort will be greatly reduced.
Fortunately, Elasticsearch has such a feature, called index lifecycle management (ILM).
Before we get into ILM, let’s jump to the conclusion.
There are two common ways to use Elasticsearch, one is to treat Elasticsearch as an OLTP database, just like a regular database where we CRUD each document. The scenario is we need the search capabilities provided by Elasticsearch, so each document will be a corresponding entity.
Take an e-commerce website as an example, in order to be able to search for product titles, content, etc., we will add a product as a document to Elasticsearch, and when there are any changes to the product, we will directly find out the corresponding document and modify the fields in it.
So, the modeling in Elasticsearch will be an index, let’s say products, which contains many documents representing products.
On the other hand, Elasticsearch can also be used as a time-series database. All data is continuously written to Elasticsearch, and the written documents are rarely modified. A common example of this is logs. We often use Elasticsearch’s powerful search capabilities to find specific patterns in logs. So in Elasticsearch, there may be a time-divided index, such as log-2024–04–01, log-2024–04–02 and so on.
Elasticsearch’s ILM only benefits from the latter.
ILM Concepts
Why ILM can only work on time-series data scenarios and not on OLTP scenarios?
Briefly, the unit of ILM is index, as indicated by the first letter I. Therefore, when only one index exists, there is no way for ILM to manage it. Either the whole index is hot, or the whole index is cold, and there is no way to distinguish which documents are hot and which are cold within this index.
The whole concept of ILM in Elasticsearch is to label the index with hot, warm and cold. With different labels, there will be different behaviors, for example, a cold index can only be read but not written. In addition, the nodes in the cluster are labeled so that different temperature indexes belong to specific nodes. In this way, the cold index can be placed in a lower cost machine to save cost.
In addition to ILM, Elasticsearch has another mechanism that is often paired with ILM, which is Rollover. When time-series data is constantly written to an index, the performance of the index will inevitably degrade. Therefore, Rollover allows the index to generate a new index when certain conditions are met, and the original index will no longer accept write requests.
The principle of Rollover is to have a dedicated alias for writing to the underlying index, e.g. index-000001, and when a new index is generated, the alias is automatically switched to the new index, e.g. index-000002.
Based on the configured ILM policy, index-000001 will be converted from a hot index to a warm index, and then to a cold index, achieving a lifecycle.
This is the whole concept of data tiering in Elasticsearch.
Conclusion
Although Elasticsearch has built-in ILM, there are still many people who misunderstand the concept of ILM. The built-in ILM is indeed a very powerful capability, but it is not a silver bullet as there are limitations on the scenarios it can be applied to.
Therefore, this article mainly explains the concept of ILM and the applicable scenarios. As for the specific operation is not difficult, the official document has a detailed description, I will not dive into it.
If we want to use Elasticsearch in an OLTP scenario and want to benefit from ILM, then I think we may need to change our modeling approach so that each CRUD creates a new entity, which might be simpler in this case.
Here’s a possible approach, let’s continue with the products example.
When we add a Japanese apple at the first index, products-01.
POST /products-01/_doc/1
{
"product_id": "0001",
"product_name": "Apple",
"description": "Made in Japan",
"price": 100
}After some time, Japanese apples were out of stock, so we switched to importing American apples. But at this time, the index has been changed to products-02 due to Rollover, and the original products-01 has been cooled down so we can't update it.
Then we need to write the same apples and updated information in products-02.
POST /products-02/_doc/1
{
"product_id": "0001",
"product_name": "Apple",
"description": "Made in USA",
"price": 50
}All of these indexes will have the same alias, products-search, for searching.
POST /_aliases
{
"actions": [
{
"add": {
"index": "products-01",
"alias": "products-search"
}
},
{
"add": {
"index": "products-02",
"alias": "products-search"
}
}
]
}Finally, we search the results directly on products-search but sort them in order, and the top one will be the final result.
POST /products-search/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"product_name": "Apple"
}
}
]
}
},
"sort": [
{
"_index": {
"order": "desc"
}
}
]
}So the client has to de-duplicate the data using product_id, which is a case of updates. Similarly for deletions, a document labeled with a soft delete flag needs to be added so that the client can be aware that the data is invalid.
In order to implement data tiering on the backend, extra processing must be done on the client side. If the system wasn’t designed this way from the beginning, there would be a lot of development on the client side, which is not always a good approach.
Alternatively, use additional ETL to move documents that meet the criteria to a different cold index. In this way the client can maintain its existing behavior without awareness.
Anyway, there’s not much ILM can do for OLTP scenarios, it all depends on the system design.