Problem description: After installing and configuring ElasticFlow and ElasticSearch/Kibana, data not seen in ElasticSearch Dashboard.
Please let me know if I am missing steps or configuration as I am lost.
- Followed configuration guide for installing ElastiFlow for RHEL 8
- Applied base license according to General Configuration | ElastiFlow
- Configured Netflow on Cisco ASA, confirmed flow-export packets are sent and confirmed flows are arriving at the Elatiflow collector with tcpdump.
[root@ipr-ost-netflow ~]# tcpdump -i ens192 port 9995
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
20:28:15.530137 IP _gateway.18526 > ipr-ost-netflow.domain.com.palace-4: UDP, length 1404
20:28:15.569445 IP _gateway.18526 > ipr-ost-netflow.domain.com.palace-4: UDP, length 1388
20:28:15.603115 IP _gateway.18526 > ipr-ost-netflow.domain.com.palace-4: UDP, length 1388
20:28:15.614045 IP _gateway.18526 > ipr-ost-netflow.domain.com.palace-4: UDP, length 1404
20:28:15.614235 IP _gateway.18526 > ipr-ost-netflow.domain.com.palace-4: UDP, length 1396
20:28:15.614347 IP _gateway.18526 > ipr-ost-netflow.domain.com.palace-4: UDP, length 1420
20:28:15.632153 IP _gateway.18526 > ipr-ost-netflow.domain.com.palace-4: UDP, length 1420
20:28:15.666376 IP _gateway.18526 > ipr-ost-netflow.domain.com.palace-4: UDP, length 1396
20:28:15.693831 IP _gateway.18526 > ipr-ost-netflow.domain.com.palace-4: UDP, length 1388
20:28:15.736514 IP _gateway.18526 > ipr-ost-netflow.domain.com.palace-4: UDP, length 1388
20:28:15.774794 IP _gateway.18526 > ipr-ost-netflow.domain.com.palace-4: UDP, length 1440
20:28:15.805470 IP _gateway.18526 > ipr-ost-netflow.domain.com.palace-4: UDP, length 1448
20:28:15.815164 IP _gateway.18526 > ipr-ost-netflow.domain.com.palace-4: UDP, length 1388
-
Followed install guide for ElastiSearch and Kibana and made all config changes as noted in RHEL/AlmaLinux | ElastiFlow
-
Set the following in /etc/elastiflow/flowcoll.yml
EF_ACCOUNT_ID: "6778c2cd4df43ce24e271785"
EF_FLOW_LICENSED_UNITS: 1
EF_FLOW_LICENSE_KEY: "My license key from email"
EF_LICENSE_ACCEPTED: "true"
EF_FLOW_DATA_PATH: /netflow/var/lib/elastiflow/flowcoll
EF_LOGGER_FILE_LOG_ENABLE: "true"
EF_LOGGER_FILE_LOG_FILENAME: /netflow/var/log/elastiflow/flowcoll/flowcoll.log
EF_LOGGER_LEVEL: debug
EF_FLOW_SERVER_UDP_IP: 192.168.46.69
EF_FLOW_SERVER_UDP_PORT: 2055,4739,6343,9995
EF_PROCESSOR_DECODE_NETFLOW9_ENABLE: "true"
EF_OUTPUT_ELASTICSEARCH_ADDRESSES: 192.168.46.69:9200
EF_OUTPUT_ELASTICSEARCH_ECS_ENABLE: "true"
EF_OUTPUT_ELASTICSEARCH_ENABLE: "true"
EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD: rollover
EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_REPLICAS: 1
EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_SHARDS: 0
#EF_OUTPUT_ELASTICSEARCH_MAX_RETRIES: 3
EF_OUTPUT_ELASTICSEARCH_PASSWORD: <omittted>
EF_OUTPUT_ELASTICSEARCH_TLS_CA_CERT_FILEPATH: "/etc/elastiflow/ca/ca.crt"
EF_OUTPUT_ELASTICSEARCH_TLS_ENABLE: "true"
EF_OUTPUT_ELASTICSEARCH_USERNAME: elastic
- Performing a http get at 192.168.46.69:9200 shows:
name "ipr-ost-netflow"
cluster_name "elasticsearch"
cluster_uuid "5ZP1PFYCRje_jsTEps41yA"
version
number "8.17.0"
build_flavor "default"
build_type "rpm"
build_hash "2b6a7fed44faa321997703718f07ee0420804b41"
build_date "2024-12-11T12:08:05.663969764Z"
build_snapshot false
lucene_version "9.12.0"
minimum_wire_compatibility_version "7.17.0"
minimum_index_compatibility_version "7.0.0"
tagline "You Know, for Search"
In /netflow/var/log/elastiflow/flowcoll/flowcoll.log I am seeing panic errors.
{"level":"panic","ts":"2025-01-07T20:47:20.529-0500","logger":"flowcoll","caller":"elasticsearch/instance_registration.go:35","msg":"failed to instantiate config","code":"elasticsearch/conf-error","reason":"ENV: 'EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_SHARDS' Value: '0' Error: failed 'required_if' validation","stacktrace":"github.com/elastiflow/flowcoll/pkg/outputs/elasticsearch.NewInstantiatorRegistration\n\t/tmp/collectors/pkg/outputs/elasticsearch/instance_registration.go:35\ngithub.com/elastiflow/flowcoll/pkg/apps/unified_flowcoll.newInstantiator\n\t/tmp/collectors/pkg/apps/unified_flowcoll/instantiator.go:109\ngithub.com/elastiflow/flowcoll/pkg/apps/unified_flowcoll.NewApp\n\t/tmp/collectors/pkg/apps/unified_flowcoll/app.go:80\nmain.main\n\t/tmp/collectors/cmd/flowcoll/main.go:93\nruntime.main\n\t/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.5.linux-amd64/src/runtime/proc.go:271"}
{"level":"error","ts":"2025-01-07T20:47:20.529-0500","logger":"flowcoll","caller":"panic/panic.go:7","msg":"Recovered from panic","error":"failed to instantiate config","stacktrace":"github.com/elastiflow/flowcoll/internal/panic.RecoverLogger\n\t/tmp/collectors/internal/panic/panic.go:7\nruntime.gopanic\n\t/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.5.linux-amd64/src/runtime/panic.go:770\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite\n\t/root/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:196\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write\n\t/root/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:262\ngo.uber.org/zap.(*Logger).Panic\n\t/root/go/pkg/mod/go.uber.org/zap@v1.27.0/logger.go:285\ngithub.com/elastiflow/flowcoll/pkg/outputs/elasticsearch.NewInstantiatorRegistration\n\t/tmp/collectors/pkg/outputs/elasticsearch/instance_registration.go:35\ngithub.com/elastiflow/flowcoll/pkg/apps/unified_flowcoll.newInstantiator\n\t/tmp/collectors/pkg/apps/unified_flowcoll/instantiator.go:109\ngithub.com/elastiflow/flowcoll/pkg/apps/unified_flowcoll.NewApp\n\t/tmp/collectors/pkg/apps/unified_flowcoll/app.go:80\nmain.main\n\t/tmp/collectors/cmd/flowcoll/main.go:93\nruntime.main\n\t/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.5.linux-amd64/src/runtime/proc.go:271"}