Dear Elastiflow Team ,
I have installed Elastiflow version 7.3.2 and configured my Output as Opensearch cluster having 3 nodes each version 2.15.0 ,
When my opensearch is down for 2 hours, and when it comes up after the 2 hours time gap , I was able to see the data that my elastiflow has received during the Outage.
I clearly checked the logs , which suggests that elastiflow has an in memory buffer where it stores the messages that were failed to be sent to opensearch.
#EF_OUTPUT_OPENSEARCH_ADDRESSES: 127.0.0.1:9200
#EF_OUTPUT_OPENSEARCH_ALLOWED_RECORD_TYPES: as_path_hop,flow_option,flow,ifa_hop,telemetry,metric
#EF_OUTPUT_OPENSEARCH_AWS_ACCESS_KEY: “”
#EF_OUTPUT_OPENSEARCH_AWS_REGION: “”
#EF_OUTPUT_OPENSEARCH_AWS_SECRET_KEY: “”
#EF_OUTPUT_OPENSEARCH_BATCH_DEADLINE: 2000
#EF_OUTPUT_OPENSEARCH_BATCH_MAX_BYTES: 8388608
#EF_OUTPUT_OPENSEARCH_CLIENT_CA_CERT_FILEPATH: “”
#EF_OUTPUT_OPENSEARCH_CLIENT_CERT_FILEPATH: “”
#EF_OUTPUT_OPENSEARCH_CLIENT_KEY_FILEPATH: “”
#EF_OUTPUT_OPENSEARCH_DROP_FIELDS: “”
#EF_OUTPUT_OPENSEARCH_ECS_ENABLE: “false”
#EF_OUTPUT_OPENSEARCH_ENABLE: “false”
#EF_OUTPUT_OPENSEARCH_INDEX_PERIOD: daily
#EF_OUTPUT_OPENSEARCH_INDEX_SUFFIX: “”
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_CODEC: best_compression
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ENABLE: “true”
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_ISM_POLICY: elastiflow
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_OVERWRITE: “true”
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_DEFAULT: _none
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_PIPELINE_FINAL: _none
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REFRESH_INTERVAL: 10s
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_REPLICAS: 1
#EF_OUTPUT_OPENSEARCH_INDEX_TEMPLATE_SHARDS: 3
#EF_OUTPUT_OPENSEARCH_MAX_RETRIES: 3
#EF_OUTPUT_OPENSEARCH_PASSWORD: admin
#EF_OUTPUT_OPENSEARCH_RETRY_BACKOFF: 1000
#EF_OUTPUT_OPENSEARCH_RETRY_ENABLE: “true”
#EF_OUTPUT_OPENSEARCH_RETRY_ON_TIMEOUT_ENABLE: “true”
#EF_OUTPUT_OPENSEARCH_TIMESTAMP_SOURCE: collect
#EF_OUTPUT_OPENSEARCH_TLS_CA_CERT_FILEPATH: “”
#EF_OUTPUT_OPENSEARCH_TLS_ENABLE: “false”
#EF_OUTPUT_OPENSEARCH_TLS_SKIP_VERIFICATION: “false”
#EF_OUTPUT_OPENSEARCH_USERNAME: admin
It is cleary mentioned in the documentation that Elastiflow retires the timed out bulk index requests which is ie. “EF_OUTPUT_OPENSEARCH_MAX_RETRIES: 3” configured to 3, and retry back off time is configured to 1second ,
Note: the elastiflow never stops even though the retry limit has crossed the Max retry limit.
With the above configuration , ideally the data that was recieved by elastiflow during the outage should have been discarded , since it should retried max number of times and opensearch was unavailable.
I have few questions
- How do i control the in memory buffer settings
- Why is the retry mechanism not working properly, or is my configuration wrong?
Please do help me out , Thanks.