Data Prepper enhancements for monitoring and horizontal scalabilityby: Rajiv Taori · on:
We are pleased to announce a beta version (0.8.0-beta) of Data Prepper is available for download and includes enhancements for new monitoring metrics and horizontal scalability. Data Prepper receives trace data from the OpenTelemetry collector, and aggregates, transforms, and normalizes it for analysis and visualization in the Trace Analytics Kibana plugin. We have previously announced alpha availability of Trace Analytics on Open Distro for Elasticsearch. You can get the latest version from the Getting Started & Downloads page.
Metrics state for Data Prepper instances can be accessed via an API in a format that is compatible with Prometheus monitoring. Metrics API is accessible at
/metrics/prometheus, and includes metrics for monitoring the prepper, sink, and buffer components. These metrics can be used to make decisions on horizontal scaling of Data Prepper, which may be required when processing higher throughput Trace Analytics workloads. To enable horizontal scaling, you can run Data Prepper instances in a cluster with requests distributed by a load balancer. The creation of service maps in Trace Analytics requires certain span information to be held in memory on Data Prepper nodes, and therefore requires instances in a Data Prepper cluster to forward requests to other specific instances within the cluster. A peer forwarder capability in Data Prepper has been added to support this.
If you’re already part of our community of users and contributors to Trace Analytics and OpenTelemetry support, a hearty ‘thank you’ from the entire team goes to you. We’re glad you’ve joined us on this journey and we deeply hope that this release solves a few problems and delights you along the way.