UDP Server to Flow Decoder is 90% full - works for a couple minutes

Network Infrastructure
1

I am getting the UDP Server to Flow Decoder is 90% full message repeatedly.

This is 1 Elastiflow instance (7.2.1) writing to 1 Elasticsearch instance on a different system/VM. Elastiflow is running concurrently with a logstash container that is gathering syslog for us (about 1K msgs/sec burst). We are sending IPFix to Elastiflow at a sustained rate of about 300 msgs/sec with a 1K msgs/sec when it bursts (coming from 3 switches).

After starting Elastiflow it works for a minute or two, writes records to Elasticsearch, then the 90% msgs start and nothing is written again. Restarts tend to have the same pattern.

The base server is well resourced with
24GB RAM
6 vCPUs
50GB HDD (at about 12GB capacity currently)

We also have a premium license good until Dec 31 of this year.

docker stats reveals less than 1% CPU usage and approx 2% memory usage.
Example:

CONTAINER ID   NAME                        CPU %     MEM USAGE / LIMIT   MEM %     NET I/O           BLOCK I/O   PIDS
d4aad778b15a   pensando-elastiflow-7.2.1   0.06%     500MiB / 23.47GiB   2.08%     5.83GB / 1.72GB   82MB / 0B   13

The logstash container stats are 2% CPU and 6.2% Memory - so not a lot of contention there.

I have debug logs but apparently can’t upload them here. I can post a portion of them in a topic entry if needed.

Help is appreciated because this is driving me nuts as to why its not working.

2

If you have a premium license I would suggest you open a support ticket at https://support.elastiflow.com or email support@elastiflow.com.

Also, if you haven’t reviewed this article, it has some helpful information:

Regards,
Dexter

3

Thanks @dxturner - I hadn’t seen that article. It looks like it may be my elastic server that is under-resourced. Will up those resources and see if that fixes it.

4

So the article that @dxturner linked help me figure out WTF was going on.

This is a demo system that we use to showcase our telemetry and we use Elastiflow as the collector for our IPFix. The problem was that we have 2 elasticsearch servers that we feed from Elastiflow. One of those elasticsearch servers was shut down (which I didn’t know) and it appears that since Elastiflow couldn’t send to that 1 server, it stopped sending to my server as well. So, I removed the other server from the docker-compose, restarted and it all works fine now.

1 Like