Can I exclude/filter IP ranges from being ingested in ElastiFlow 7.x?

I am running ElastiFlow 7.x using the Docker setup and trying to find a way to exclude specific IP ranges from being ingested or at least filtered before they make it to Elasticsearch.

I know about the EF_PROCESSOR_ENRICH_IPADDR_METADATA_USERDEF_PATH option and updating the ipaddrs.yml, but that only handles enrichment. It does not actually stop the data from being stored.

I also know I can filter things out in Kibana, but that still means the data is ingested and stored, which is what I am trying to avoid.

The issue is that vendors like Palo Alto do not let you filter the NetFlow export itself, so everything just gets sent to the collector. I want to avoid storing flows for internal traffic or other noisy sources.

Is there any way in version 7 to drop or ignore flows based on source or destination IP before they get indexed?

Appreciate any help or ideas.

You can drop certain “types” of records at the collector using EF_OUTPUT_ELASTICSEARCH_ALLOWED_RECORD_TYPES, but there is no mechanism to exclude certain IPs or address ranges.

I would probably look into creating an ingest pipeline in Elasticsearch with a ‘drop’ processor and drop the record before ingest.

Might make a good feature request though! Can you give some more specifics or examples (a PCAP is helpful) on what you would exclude?

Thanks for the reply. That makes sense. I just didn’t know if the collector itself could do filtering like that yet.

One use case I have is our network is segmented across a bunch of VLANs, like most corporate environments. We have separate VLANs for data, voice, printers, cameras, staff WiFi, guest WiFi, etc. This spans 40+ remote sites, all with the same layout.

Let’s say for example that each site has a camera VLAN and we follow a consistent subnet scheme where the 2nd octet represents the site and the 3rd octet is the VLAN. So something like 10.0.30.0/24, 10.1.30.0/24, 10.2.30.0/24, etc.

In this case, I don’t care about the camera traffic and would want to just exclude those ranges altogether from ever being ingested. Something like what METADATA_USERDEF_PATH does with the enrichment:

exclude:
cidr:
- 10.0.30.0/24
- 10.1.30.0/24
- 10.2.30.0/24

Or even with regex:

exclude:
regex:
- “^10\.(25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})\.30\.0/24$”

I’ll look into doing it with an ingest pipeline for now, but yeah, being able to drop traffic like this right at the collector would be a great feature add.