Can I exclude/filter IP ranges from being ingested in ElastiFlow 7.x?

Network Flows
1

I am running ElastiFlow 7.x using the Docker setup and trying to find a way to exclude specific IP ranges from being ingested or at least filtered before they make it to Elasticsearch.

I know about the EF_PROCESSOR_ENRICH_IPADDR_METADATA_USERDEF_PATH option and updating the ipaddrs.yml, but that only handles enrichment. It does not actually stop the data from being stored.

I also know I can filter things out in Kibana, but that still means the data is ingested and stored, which is what I am trying to avoid.

The issue is that vendors like Palo Alto do not let you filter the NetFlow export itself, so everything just gets sent to the collector. I want to avoid storing flows for internal traffic or other noisy sources.

Is there any way in version 7 to drop or ignore flows based on source or destination IP before they get indexed?

Appreciate any help or ideas.

2

You can drop certain “types” of records at the collector using EF_OUTPUT_ELASTICSEARCH_ALLOWED_RECORD_TYPES, but there is no mechanism to exclude certain IPs or address ranges.

I would probably look into creating an ingest pipeline in Elasticsearch with a ‘drop’ processor and drop the record before ingest.

Might make a good feature request though! Can you give some more specifics or examples (a PCAP is helpful) on what you would exclude?

3

Thanks for the reply. That makes sense. I just didn’t know if the collector itself could do filtering like that yet.

One use case I have is our network is segmented across a bunch of VLANs, like most corporate environments. We have separate VLANs for data, voice, printers, cameras, staff WiFi, guest WiFi, etc. This spans 40+ remote sites, all with the same layout.

Let’s say for example that each site has a camera VLAN and we follow a consistent subnet scheme where the 2nd octet represents the site and the 3rd octet is the VLAN. So something like 10.0.30.0/24, 10.1.30.0/24, 10.2.30.0/24, etc.

In this case, I don’t care about the camera traffic and would want to just exclude those ranges altogether from ever being ingested. Something like what METADATA_USERDEF_PATH does with the enrichment:

exclude:
cidr:
- 10.0.30.0/24
- 10.1.30.0/24
- 10.2.30.0/24

Or even with regex:

exclude:
regex:
- “^10\.(25[0-5]|2[0-4][0-9]|1?[0-9]{1,2})\.30\.0/24$”

I’ll look into doing it with an ingest pipeline for now, but yeah, being able to drop traffic like this right at the collector would be a great feature add.

4

You can achieve this using a combination of the collectors User-Defined Metadata enrichment for IP addresses, and an Elasticsearch/OpenSearch ingest pipeline. The basic steps are…

  1. Use metadata enrichment to add the field ingest_action to the record.
192.0.2.0/24:
  metadata:
    ingest_action: 'DROP'
  1. Create an ingest pipeline that will drop records for which the value of ingest_action is DROP.
PUT _ingest/pipeline/elastiflow_drop
{
	"description": "ElastiFlow - drop records with an ingest_action of DROP",
	"processors": [
        {
            "drop": {
                "if": "ctx['ingest_action'] == 'DROP'"
            }
        }
	]
}
  1. Configure the collector’s Elasticsearch output to add the ingest pipeline to index template for ElastiFlow indices.
EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_PIPELINE_FINAL: 'elastiflow_drop'

I will be creating a blog that explains this in more detail.

1 Like
5

Awesome, this got me exactly what I needed.