Hi, I want to pipe the flow data to be consumed to a file, separate from the “status logs” displaying errors etc. Any way to accomplish this? I imagine it has something to do with stdout enabled and in json format.
Want to do this to use filebeat to consume the metrics so we have backup of the raw output which is consumed to elasticsearch.
EDIT: Using the rhel linux systemd installation.
Hi,
It’s not entirely clear to me what you are asking, but I think you want to send flow records to Filebeat AND Elasticsearch. Is that correct?
Also, is the “status log” you are referring to the log for the collector itself, usually written to /var/log/elastiflow/flowcoll/flowcoll.log? If so, that log would not contain any flow records.
The supported output options for ElastiFlow NetObserv flow are listed here in the product documentation. If the goal is to send data directly to Elasticsearch and to Filebeat, I would suggest setting up the Filebeat HTTP Endpoint and then using the Generic HTTP Output for the flow collector to send the data. You can enable both outputs in the flowcoll.yml.
EF_OUTPUT_GENERIC_HTTP_ENABLE
EF_OUTPUT_ELASTICSEARCH_ENABLE
Let us know if this helps.
Thanks,
Dexter
Hi, not quite. At it’s core, I just want to write the raw flow records to a file on my local filesystem in json.
Then after that we use filebeat to read those files on the server. The goal is to not directly use the EF_OUTPUT_ELASTICSEARCH setup.
Yes, referrering to “status log” is the log for the collector itself, written to “/var/log/elastiflow/flowcoll/flowcoll.log”.
I would want to write to e.g. /var/log/elastiflow/flows.json or equivalent.
Think that is possible in any smart way?
We have a ‘standard out’ output that you could re-direct to a file, but it’s only recommended for testing and low flow volumes.
Everyone that I know that uses Filebeat, uses the Filebeat http_endpoint. You could then set up Logstash to write to both Elastic and a local file.
I’m curious to know what the value is of having a local file with raw flow records. You would most likely need to set up some sort of file rotation scheme to avoid filling the disk, and the volume and sometimes density of flow records make parsing them from file difficult. This is why we have outputs like Elasticsearch that can handle the high volume and density much more efficiently.
This may just be a temporary solution if the flow records are too heavy. For now, the focus is the same, to create a file by piping systemd or equivalent.
Though I cannot seem to get it to work. Specifying
EF_OUTPUT_STDOUT_ALLOWED_RECORD_TYPES: 'as_path_hop,flow_option,flow,ifa_hop,telemetry,metric,log'
EF_OUTPUT_STDOUT_ENABLE: "true"
EF_OUTPUT_STDOUT_FORMAT: json
And updating /etc/systemd/system/flowcoll.service to include
StandardOutput=/var/log/elastiflow/records.log
Doesn’t seem to return anything. Am I misunderstanding something? flowcoll is ran through systemctl.
A process only has one stdout stream, so the flow records and collector logs will be mixed. I don’t know of anyway to avoid that outside of an external script or something. This is why we specifically state that the STDOUT output is for testing only.
Your ‘StandardOutput’ setting in systemd should be done in an override file ( to prevent getting overwritten with upgrades ) and the syntax is likely something like StandardOutput=append:/var/log/elastiflow/records.log. You will also need to do a systemctl daemon-reload
Regards,
Dexter