hi, i’m trying to get a rough feel for what normal dashboard performance looks like with the large cluster setup from the docs.
if you have about a day of data coming in at close to 48k flows per second, how long do the standard dashboards usually take to load? i’m especially interested in the overview, top talkers, traffic details, and flow records dashboards.
even rough numbers would be really helpful. i’d love to know what the first load looks like when things are cold, what repeat loads look like once the caches are warm, and roughly what elasticsearch version and hardware you’re using. if you happen to have query took times, those would be useful too.
i’m not looking for a formal benchmark, just a real-world ballpark. thanks!
Unfortunately it isn’t a 100% straight forward answer, as the cardinality of the data has a real impact on query performance. To figure the Top-N from 1M IP addresses or 10K is going to be a different load. It also depends on how far back you are trying to query. To give you an answer I would say 15-60 secs if looking at the last hour or two of data.
The first thing you can do to improve query performance is increase the shard count. This allows Elasticsearch to better parallelize the work of the query. 3 or 4 shards per hot node, rather than 1 or 2, can make a very noticeable difference.
Since you a referring to the XL example, which includes warm nodes, you can also reduce the load on the hot nodes by rolling data over to the warm nodes more frequently, e,g, every 4 hours. The negative impact that the active indexing of data has on query performance is more than most people assume. By moving data more quickly to the warm nodes you will be able to query over longer time spans with good performance, as most of the data included in the query is coming from the warm nodes. In this case you would shift some of the storage capacity from the hot nodes to the warm nodes.
Are you already using or evaluating ElastiFlow up to 48K/sec? If so our team can help you better understand the architecture choices.