Hey everyone!
As you may have already seen, Elastic earlier this week announced some changes to their licensing of the synthetic _source feature as part of the release notes for 8.17. You can find the full knowledge base article about this change here.
What does this change mean for me?
Synthetic _source is used in TSDS (among other things), so if you are currently using the TSDS feature to save up to 70% of storage space and you do not have an enterprise license applied to your Elastic nodes, you might see a significant increase in storage requirements once you update to Elastic v8.17.
Are there options to be grandfathered in?
Based on the knowledge base article linked above, it seems like Elastic offers grandfathering in until the end of all contracts that are valid as of 4 Feb 2025, which might make it beneficial for you to extend your Elastic agreement before that time. Please get in touch with either your Elastic or ElastiFlow account team (depending on who you purchased Elastic licenses from) to see if this is a feasible option for you or if there are other avenues for you to continue using TSDS.
What is ElastiFlow doing to reduce storage requirements without TSDS?
Our engineers are working on a solution that will achieve part of the storage savings gained through TSDS, but without using the synthetic _source feature. This should allow you to retain about 40% of the total storage savings gained through TSDS and would be available for all Elastic nodes, including the open-source on prem deployment. It should also improve query performance, as building the synthetic _source at query time would no longer be required. We expect to release this feature in Q1 of 2025. Note that this will not provide any storage savings on top of TSDS, but can only be used when TSDS is disabled.
If you have any questions, please reply to this post.