I am running ElastiFlow 7.x using the Docker setup and trying to find a way to exclude specific IP ranges from being ingested or at least filtered before they make it to Elasticsearch.
I know about the EF_PROCESSOR_ENRICH_IPADDR_METADATA_USERDEF_PATH option and updating the ipaddrs.yml, but that only handles enrichment. It does not actually stop the data from being stored.
I also know I can filter things out in Kibana, but that still means the data is ingested and stored, which is what I am trying to avoid.
The issue is that vendors like Palo Alto do not let you filter the NetFlow export itself, so everything just gets sent to the collector. I want to avoid storing flows for internal traffic or other noisy sources.
Is there any way in version 7 to drop or ignore flows based on source or destination IP before they get indexed?
You can drop certain “types” of records at the collector using EF_OUTPUT_ELASTICSEARCH_ALLOWED_RECORD_TYPES, but there is no mechanism to exclude certain IPs or address ranges.
I would probably look into creating an ingest pipeline in Elasticsearch with a ‘drop’ processor and drop the record before ingest.
Might make a good feature request though! Can you give some more specifics or examples (a PCAP is helpful) on what you would exclude?
Thanks for the reply. That makes sense. I just didn’t know if the collector itself could do filtering like that yet.
One use case I have is our network is segmented across a bunch of VLANs, like most corporate environments. We have separate VLANs for data, voice, printers, cameras, staff WiFi, guest WiFi, etc. This spans 40+ remote sites, all with the same layout.
Let’s say for example that each site has a camera VLAN and we follow a consistent subnet scheme where the 2nd octet represents the site and the 3rd octet is the VLAN. So something like 10.0.30.0/24, 10.1.30.0/24, 10.2.30.0/24, etc.
In this case, I don’t care about the camera traffic and would want to just exclude those ranges altogether from ever being ingested. Something like what METADATA_USERDEF_PATH does with the enrichment:
I’ll look into doing it with an ingest pipeline for now, but yeah, being able to drop traffic like this right at the collector would be a great feature add.
Use metadata enrichment to add the field ingest_action to the record.
192.0.2.0/24:
metadata:
ingest_action: 'DROP'
Create an ingest pipeline that will drop records for which the value of ingest_action is DROP.
PUT _ingest/pipeline/elastiflow_drop
{
"description": "ElastiFlow - drop records with an ingest_action of DROP",
"processors": [
{
"drop": {
"if": "ctx['ingest_action'] == 'DROP'"
}
}
]
}
Configure the collector’s Elasticsearch output to add the ingest pipeline to index template for ElastiFlow indices.