Flowcoll is downloading 70 till 80 Mbyte each hour

ElastiFlow Community
1

Dear All,
I realised that the flowcoll process is downloading each hour between 70 MB and 80 MB of data. This happens in exactly 60 minutes intervals after the flowcoll process is started and needs less than a minute in my case. Download is done with https from 2606:4700:10::6816:2dc2 most of the time.

In my opinion the system is running amok. I can’t believe that’s necessary to download 70 MB each hour. Are there any logs to dig deeper into this issue ? Or maybe to switch off this behaviour ?
And there is the question, what is it downloading ?

Kind regards
Hans

2

This is likely from downloading the NetIntel threat database, though I’m not sure how often this occurs by default.

If you do not want the threat or app identification you can turn it off in the configuration file. Please see ElastiFlow NetIntel | ElastiFlow

3

Hi dxturner,
many thanks for your swift answer. Following the documentation I couldn’t find a setup for EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_ENABLE
The file /etc/elastiflow/flowcoll.yml has only comment lines. So it seems all values are default. All the changes are done in /etc/systemd/system/flowcoll.service.d/flowcoll.conf
But in none of this 2 files is something which could be similar as what is listed in this link.
Any other ideas how I could fix this behaviour ?
But general you are right. In flowcoll.log I can see exactly at that time the following entries:

2025-02-03T23:00:14.205+0100 info ipaddr_enricher.netintel_threats netintel/enricher.go:262 fetching threat type collection and cidr tree

2025-02-03T23:00:15.163+0100 info ipaddr_enricher.netintel_threats netintel/enricher.go:313 Threat Type size: 53230 bytes

2025-02-03T23:00:28.868+0100 info flowcoll.metrics_provider metrics/provider.go:120 gathering metrics

2025-02-03T23:00:30.414+0100 error ipaddr_enricher.netintel_threats netintel/enricher.go:251 error initializing cidr tree {“error”: “context deadline exceeded (Client.Timeout or context cancellation while reading body)”}

github.com/elastiflow/go-enrich-ipaddr/netintel.(*NetIntel).run.func1

/go/pkg/mod/github.com/elastiflow/go-enrich-ipaddr@v1.0.10/netintel/enricher.go:251

Kind regards
Hans

4

EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_ENABLE is enabled by default and there is no specific entry in the configuration file. If you want to disable it you will need to specifically set it to false by adding an entry to the file.

If you are using /etc/systemd/system/flowcoll.service.d/flowcoll.conf for configuration use the “Environment=” syntax in that file to set the value.

As of v6 ElastiFlow started supporting the YAML file configuration in /etc/elastiflow/flowcoll.yml. Though you can have configuration settings in both files, I would recommend using one or the other to avoid conflicts. I prefer flowcoll.yml because I do not have to do sudo systemctl daemon-reload for each change. I only have to do sudo systemctl restart flowcoll.

Hope this helps.

5

Hi,
many thanks for this hint. This issue stoped now. Stays the question, why happens this error ? Probably I will upgrade. I am still on version 7.3, I see 7.7 is now available. But if I understand correctly I would need a license in any case, which I don’t have.

Kind regards
Hans

6

We have a default timeout on the download. After getting several reports of people seeing the timeout error, we have increased the default timeout setting.

You can get a “free” basic license that’s good for year by registering here:

7

Hi, it seems you fixed this issue by increasing the timeout. I upgraded to flowcoll 7.7.0 and enabled NetIntel threat database again. I see in the logs in one hour intervalls activieties but the big amount of data download doesn’t happen anymore.
There is only one error which comes periodically.

2025-02-06T10:33:19.036+0100 error ipaddr_enricher.netintel_threats netintel/enricher.go:238 error refreshing ipdb & threat collection {“error”: “error initializing ip trie: failed to unmarshal ipdb bytes: proto: cannot parse invalid wire-format data”}
github.com/elastiflow/go-enrich-ipaddr/netintel.(*NetIntel).Run.func1
/go/pkg/mod/github.com/elastiflow/go-enrich-ipaddr@v1.1.1/netintel/enricher.go:238

Which type of enrichment is that, what do I see more if this is enabled ?
Is there anywhere a link describing it ?

Many thanks for your support.

Kind regards
Hans

8

I have not been able to replicate this error. Please let me know some more details about your implementation … native linux? if so, what OS / pkg was used? docker?

It would also be helpful to have your flowcoll.yml file (comment out any passwords/sensitive information) so that I test an identical setup. You can post here or email me with just “dexter” and “elastiflow.com.” If email, the log would be helpful, too.

Thanks,
Dexter

9

Hi,
I’m getting a similar error message on Ubuntu (no docker) with flowcoll 7.7.0.
Result of tail -f /var/log/elastiflow/flowcoll/flowcoll.log | grep netintel while the service is restarting :

{"level":"info","ts":"2025-02-07T19:46:57.376+0100","logger":"flowcoll.processor[default]","caller":"envconf/logger.go:49","msg":"EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_API_ADDR=https://query.netintel.elastiflow.com"}
{"level":"info","ts":"2025-02-07T19:46:59.915+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:283","msg":"initializing threat type collection"}
{"level":"info","ts":"2025-02-07T19:46:59.915+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:365","msg":"fetching resource from file: /var/lib/elastiflow/flowcoll/threat_collection.pb"}
{"level":"info","ts":"2025-02-07T19:46:59.918+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:288","msg":"Threat Type size: 53230 bytes"}
{"level":"info","ts":"2025-02-07T19:46:59.922+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:262","msg":"initializing ipdb"}
{"level":"info","ts":"2025-02-07T19:46:59.923+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:349","msg":"fetching resource from server: RESOURCE_IP_DB"}
{"level":"error","ts":"2025-02-07T19:48:30.445+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:223","msg":"error initializing ipdb & threat collection","error":"error initializing ip trie: failed to initialize ipdb: error fetching resource from server: context deadline exceeded (Client.Timeout or context cancellation while reading body)","stacktrace":"github.com/elastiflow/go-enrich-ipaddr/netintel.(*NetIntel).Run\n\t/go/pkg/mod/github.com/elastiflow/go-enrich-ipaddr@v1.1.1/netintel/enricher.go:223\ngithub.com/elastiflow/go-enrich-ipaddr/enrichipaddr.New\n\t/go/pkg/mod/github.com/elastiflow/go-enrich-ipaddr@v1.1.1/enrichipaddr/enrichipaddr.go:144\ngithub.com/elastiflow/flowcoll/pkg/processors/flowprocessor/cached.(*Components).generateEnrichIPAddrCache\n\t/tmp/collectors/pkg/processors/flowprocessor/cached/cached.go:257\ngithub.com/elastiflow/flowcoll/pkg/processors/flowprocessor/cached.(*Components).buildCaches\n\t/tmp/collectors/pkg/processors/flowprocessor/cached/cached.go:85\ngithub.com/elastiflow/flowcoll/pkg/processors/flowprocessor/cached.NewCachedComponents\n\t/tmp/collectors/pkg/processors/flowprocessor/cached/cached.go:55\ngithub.com/elastiflow/flowcoll/pkg/processors/flowprocessor.NewInstantiatorRegistration.newCreateInstanceFunc.func1\n\t/tmp/collectors/pkg/processors/flowprocessor/instance_registration.go:150\ngithub.com/elastiflow/go-env-conf/instantiator.(*Instantiator).Run\n\t/go/pkg/mod/github.com/elastiflow/go-env-conf@v0.8.7/instantiator/instantiator.go:78\ngithub.com/elastiflow/flowcoll/pkg/apps/unified_flowcoll.(*App).Run\n\t/tmp/collectors/pkg/apps/unified_flowcoll/app.go:157\nmain.main\n\t/tmp/collectors/cmd/flowcoll/main.go:106\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}
{"level":"info","ts":"2025-02-07T19:48:30.446+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:245","msg":"running netintel"}

Here is the content of my flowcoll.yml :

EF_ACCOUNT_ID: "XYZ"
EF_LICENSE_ACCEPTED: "true"
EF_FLOW_LICENSE_KEY: "XYZ"
EF_LOGGER_FILE_LOG_ENABLE: "true"
EF_LOGGER_LEVEL: info
EF_FLOW_SERVER_UDP_IP: 0.0.0.0
EF_FLOW_SERVER_UDP_PORT: 2055
EF_FLOW_SERVER_UDP_READ_BUFFER_MAX_SIZE: 134217728
EF_AWS_VPC_FLOW_LOG_S3_ENABLE: "false"
EF_PROCESSOR_ENRICH_APP_ID_ENABLE: "true"
EF_PROCESSOR_ENRICH_APP_ID_PATH: "/etc/elastiflow/app/appid.yml"
EF_PROCESSOR_ENRICH_APP_IPPORT_ENABLE: "true"
EF_PROCESSOR_ENRICH_APP_IPPORT_PATH: "/etc/elastiflow/settings/apps_user_defined.yml"
EF_PROCESSOR_ENRICH_APP_IPPORT_PRIVATE: "true"
EF_PROCESSOR_ENRICH_APP_IPPORT_PUBLIC: "true"
EF_PROCESSOR_ENRICH_IPADDR_DNS_ENABLE: "true"
EF_PROCESSOR_ENRICH_IPADDR_DNS_NAMESERVER_IP: "192.168.90.254"
EF_PROCESSOR_ENRICH_IPADDR_DNS_NAMESERVER_TIMEOUT: 3000
EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_ASN_ENABLE: "true"
EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_ENABLE: "true"
EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_INCLEXCL_PATH: "/etc/elastiflow/maxmind/incl_excl.yml"
EF_PROCESSOR_ENRICH_IPADDR_METADATA_ENABLE: "true"
EF_PROCESSOR_ENRICH_IPADDR_METADATA_REFRESH_RATE: 15
EF_PROCESSOR_ENRICH_IPADDR_METADATA_USERDEF_PATH: "/etc/elastiflow/metadata/ipaddrs.yml"
EF_PROCESSOR_ENRICH_NETIF_FLOW_OPTIONS_ENABLE: "true"
EF_PROCESSOR_ENRICH_NETIF_METADATA_ENABLE: "true"
EF_PROCESSOR_ENRICH_NETIF_METADATA_USERDEF_PATH: "/etc/elastiflow/metadata/netifs.yml"
EF_PROCESSOR_ENRICH_NETIF_SNMP_COMMUNITIES: public
EF_PROCESSOR_ENRICH_NETIF_SNMP_ENABLE: "true"
EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_ENABLE: "true"
EF_OUTPUT_ELASTICSEARCH_ADDRESSES: 192.168.90.251:9200
EF_OUTPUT_ELASTICSEARCH_ECS_ENABLE: "true"
EF_OUTPUT_ELASTICSEARCH_ENABLE: "true"
EF_OUTPUT_ELASTICSEARCH_INDEX_PERIOD: rollover
EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_ILM_LIFECYCLE: rollover
EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_REPLICAS: 0
EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_SHARDS: 1
EF_OUTPUT_ELASTICSEARCH_PASSWORD: "XYZ"
EF_OUTPUT_ELASTICSEARCH_TIMESTAMP_SOURCE: end
EF_OUTPUT_ELASTICSEARCH_TLS_CA_CERT_FILEPATH: "/etc/elastiflow/ca/ca.crt"
EF_OUTPUT_ELASTICSEARCH_TLS_ENABLE: "true"
EF_OUTPUT_ELASTICSEARCH_TLS_SKIP_VERIFICATION: "false"
EF_OUTPUT_ELASTICSEARCH_USERNAME: "XYZ"
EF_OUTPUT_OPENSEARCH_ECS_ENABLE: "false"
EF_OUTPUT_OPENSEARCH_ENABLE: "false"

Thanks for your help.

10

Hi Dexter,
many thanks coming back to my issue.
environment is:
native Debian GNU/Linux 12 (bookworm) last patch level
ELK stack 8.17.1
flow-collector_7.7.0_linux_amd64
flowcoll.yml is empty I have the config in …/flowcoll.service.d/flowcoll.conf
This is it. I removed all comment lines


[Service]

Environment="EF_LICENSE_ACCEPTED=true"

Environment="EF_ACCOUNT_ID=XXXXXXXXXXXXXXXXXXXXXXXX"

Environment="EF_FLOW_LICENSE_KEY=XXXXXXXXXXXXXXXXXXXXXXXX"

Environment="EF_FLOW_LICENSED_UNITS=1"

Environment="EF_API_PORT=16080"

Environment="EF_LOGGER_LEVEL=info"

Environment="EF_LOGGER_ENCODING=console"

Environment="EF_LOGGER_FILE_LOG_ENABLE=true"

Environment="EF_LOGGER_FILE_LOG_FILENAME=/var/log/elastiflow/flowcoll/flowcoll.log"

Environment="EF_FLOW_SERVER_UDP_IP=0.0.0.0"

Environment="EF_FLOW_SERVER_UDP_PORT=9995"

Environment="EF_AWS_VPC_FLOW_LOG_ENABLE=false"

Environment="EF_PROCESSOR_DECODE_IPFIX_ENABLE=true"

Environment="EF_PROCESSOR_DECODE_NETFLOW9_ENABLE=true"

Environment="EF_PROCESSOR_DECODE_SFLOW5_ENABLE=true"

Environment="EF_PROCESSOR_DECODE_SFLOW_FLOWS_ENABLE=true"

Environment="EF_PROCESSOR_DECODE_SFLOW_FLOWS_KEEP_SAMPLES=false"

Environment="EF_PROCESSOR_DECODE_SFLOW_COUNTERS_ENABLE=true"

Environment="EF_PROCESSOR_ENRICH_APP_ID_ENABLE=false"

Environment="EF_PROCESSOR_ENRICH_APP_IPPORT_ENABLE=false"

Environment="EF_PROCESSOR_ENRICH_IPADDR_METADATA_ENABLE=false"

Environment="EF_PROCESSOR_ENRICH_IPADDR_DNS_ENABLE=false"

Environment="EF_PROCESSOR_ENRICH_IPADDR_DNS_NAMESERVER_IP="

Environment="EF_PROCESSOR_ENRICH_IPADDR_DNS_NAMESERVER_TIMEOUT=3000"

Environment="EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_ASN_ENABLE=true"

Environment="EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_ASN_PATH=/usr/share/GeoIP/GeoLite2-ASN.mmdb"

Environment="EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_ENABLE=true"

Environment="EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_PATH=/usr/share/GeoIP/GeoLite2-City.mmdb"

Environment="EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_VALUES=city,country,country_code,location,timezone"

Environment="EF_PROCESSOR_ENRICH_IPADDR_MAXMIND_GEOIP_LANG=en"

Environment="EF_PROCESSOR_ENRICH_IPADDR_RISKIQ_ASN_ENABLE=false"

Environment="EF_PROCESSOR_ENRICH_IPADDR_RISKIQ_ASN_ENDPOINT=https://api.passivetotal.org/v2/netflow/as/download"

Environment="EF_PROCESSOR_ENRICH_IPADDR_RISKIQ_ASN_REFRESH_INTERVAL=1440"

Environment="EF_PROCESSOR_ENRICH_IPADDR_RISKIQ_THREAT_ENABLE=true"

Environment="EF_PROCESSOR_ENRICH_IPADDR_RISKIQ_THREAT_ENDPOINT=https://api.passivetotal.org/v2/netflow/blocklist/download"

Environment="EF_PROCESSOR_ENRICH_IPADDR_RISKIQ_THREAT_REFRESH_INTERVAL=240"

Environment="EF_PROCESSOR_ENRICH_IPADDR_RISKIQ_API_USER=#################"

Environment="EF_PROCESSOR_ENRICH_IPADDR_RISKIQ_API_KEY=#################"

Environment="EF_PROCESSOR_ENRICH_IPADDR_RISKIQ_API_TIMEOUT=180"

Environment="EF_PROCESSOR_ENRICH_NETIF_METADATA_ENABLE=false"

Environment="EF_PROCESSOR_ENRICH_NETIF_FLOW_OPTIONS_ENABLE=true"

Environment="EF_PROCESSOR_ENRICH_NETIF_SNMP_ENABLE=true"

Environment="EF_PROCESSOR_ENRICH_NETIF_SNMP_PORT=161"

Environment="EF_PROCESSOR_ENRICH_NETIF_SNMP_VERSION=2"

Environment="EF_PROCESSOR_ENRICH_NETIF_SNMP_COMMUNITIES=XXXXXXXXX"

Environment="EF_PROCESSOR_ENRICH_NETIF_SNMP_TIMEOUT=2"

Environment="EF_PROCESSOR_ENRICH_NETIF_SNMP_RETRIES=1"

Environment="EF_PROCESSOR_ENRICH_JOIN_ASN=true"

Environment="EF_PROCESSOR_ENRICH_JOIN_GEOIP=true"

Environment="EF_OUTPUT_ELASTICSEARCH_ENABLE=true"

Environment="EF_OUTPUT_ELASTICSEARCH_ECS_ENABLE=false"

Environment="EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_SHARDS=1"

Environment="EF_OUTPUT_ELASTICSEARCH_INDEX_TEMPLATE_REPLICAS=0"

Environment="EF_OUTPUT_ELASTICSEARCH_ADDRESSES=###################:9200"

Environment="EF_OUTPUT_ELASTICSEARCH_USERNAME=########"

Environment="EF_OUTPUT_ELASTICSEARCH_PASSWORD=########"

Environment="EF_OUTPUT_ELASTICSEARCH_TLS_ENABLE=true"

Environment="EF_OUTPUT_ELASTICSEARCH_TLS_SKIP_VERIFICATION=false"

Environment="EF_OUTPUT_ELASTICSEARCH_TLS_CA_CERT_FILEPATH=/etc/elastiflow/elasticsearch-ca.pem"

Environment="EF_OUTPUT_OPENSEARCH_ENABLE=false"

Environment="EF_OUTPUT_OPENSEARCH_ECS_ENABLE=false"

Environment="EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_SHARDS=1"

Environment="EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REPLICAS=0"

Environment="EF_OUTPUT_OPENSEARCH_ADDRESSES=127.0.0.1:9200"

Environment="EF_OUTPUT_OPENSEARCH_USERNAME=#######"

Environment="EF_OUTPUT_OPENSEARCH_PASSWORD=#######"

Environment="EF_OUTPUT_OPENSEARCH_TLS_ENABLE=false"

Environment="EF_OUTPUT_OPENSEARCH_TLS_SKIP_VERIFICATION=false"

Environment="EF_OUTPUT_OPENSEARCH_TLS_CA_CERT_FILEPATH="

Environment="EF_OUTPUT_SPLUNK_HEC_ENABLE=false"

Environment="EF_OUTPUT_SPLUNK_HEC_ADDRESSES=127.0.0.1:8088"

Environment="EF_OUTPUT_SPLUNK_HEC_TOKEN="

Environment="EF_OUTPUT_KAFKA_ENABLE=false"

Environment="EF_OUTPUT_KAFKA_BROKERS="

Environment="EF_OUTPUT_KAFKA_SASL_ENABLE=false"

Environment="EF_OUTPUT_CRIBL_ENABLE=false"

Environment="EF_OUTPUT_CRIBL_ADDRESSES=127.0.0.1:10080"

Environment="EF_OUTPUT_CRIBL_TOKEN="

Environment="EF_OUTPUT_GENERIC_HTTP_ENABLE=false"

Environment="EF_OUTPUT_GENERIC_HTTP_ECS_ENABLE=false"

Environment="EF_OUTPUT_GENERIC_HTTP_ADDRESSES=127.0.0.1:8888"

Environment="EF_OUTPUT_RISKIQ_ENABLE=true"

Environment="EF_OUTPUT_RISKIQ_HOST=flow.riskiq.net"

Environment="EF_OUTPUT_RISKIQ_PORT=20000"

Environment="EF_OUTPUT_RISKIQ_CUSTOMER_UUID=##############"

Environment="EF_OUTPUT_RISKIQ_CUSTOMER_ENCRYPTION_KEY=##############=="

Environment="EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_ENABLE=true"
11

The client timeout is the same error that Hans reported earlier. Upgrading to 7.7.0 fixed the timeout for him, since the timeout was extended in 7.5.2 to address that problem. I believe the default setting for EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_TIMEOUT is now 90 seconds (it was 15). You may want to bump that up to 120 and see if that helps.

12

Haven’t had a chance to test with your config, but please note that RiskIQ was EOL last June so we are no longer supporting/using that integration. See this Changelog note.

13

Yes, it worked !

{"level":"info","ts":"2025-02-07T22:18:58.904+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:283","msg":"initializing threat type collection"}
{"level":"info","ts":"2025-02-07T22:18:58.905+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:365","msg":"fetching resource from file: /var/lib/elastiflow/flowcoll/threat_collection.pb"}
{"level":"info","ts":"2025-02-07T22:18:58.907+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:288","msg":"Threat Type size: 53230 bytes"}
{"level":"info","ts":"2025-02-07T22:18:58.910+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:262","msg":"initializing ipdb"}
{"level":"info","ts":"2025-02-07T22:18:58.911+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:349","msg":"fetching resource from server: RESOURCE_IP_DB"}
{"level":"info","ts":"2025-02-07T22:27:49.915+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:267","msg":"ipdb size: 245862598 bytes"}
{"level":"info","ts":"2025-02-07T22:28:12.792+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:225","msg":"resources successfully initialized"}
{"level":"info","ts":"2025-02-07T22:28:12.793+0100","logger":"ipaddr_enricher.netintel_threats","caller":"netintel/enricher.go:245","msg":"running netintel"}

As you can see for me the download took more than 10 minutes (4G connection; I have about 10Mbps downlink). I’ve set EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_TIMEOUT to 1800, think I’ll be fine now.
Thanks !

14

Hi,

… took more than 10 minutes … 10Mbps downlink

I am wondering if these are really 10 Mbps
I have less than 40 Mbps downlink and it takes 25 seconds
Or it’s not 80 Mbyte what you are downloading ??

// Hans

15

I removed these settings now.

// Hans

16

Hi Dexter,
since I removed all RiskIQ settings in the config there are no errors or warnings in my flowcoll.log.
BUT
Obviously I didn’t look close enough or lets say not long enough back into history.
These downloads happens now each 5 hours instead of each hour as it was earlier. And the amount of data is now about 250 MByte.
Is it really necessary to download so much data each time ?

Kind regards
Hans

17

Hans,

The size of the NetIntel data has increased in recent releases because we are now including AS information. In the next release we will be improving the compression of the data so the download size should be much lower.

The reason you are now seeing every 5 hours instead of every hour is because we knew the size would be increasing and have set up a default persist time of 4 hours to help manage the downloads. We check the time of the NetIntel data files and if they are older than 4 hours (default persist rate of 240 minutes) then we refresh the file in 1 hour (default refresh rate of 60 minutes).

You can adjust the persist and refresh rates with the following settings:

EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_PERSISTENCE_INTERVAL
EF_PROCESSOR_ENRICH_IPADDR_NETINTEL_REFRESH_RATE

So, for example, you could set the ‘persistence’ interval to 1380 (1380 minutes = 23 hours) and leave the ‘refresh’ rate at the default of 60, which would result in downloading once per day, instead of every 5 hours.

I hope this helps manage the download volume for your situation, and let us know if you need any other information.

And thanks for the question! This was educational for me and I hope everyone else!

Regards,
Dexter

18

Hi Dexter,
many thanks for this update. These are great news you are giving to us.
And also many thanks for your effort and work you are putting into this great product.

Kind regards
Hans