Performance dilemma: Java Agent Log Collection vs. Filelog Receiver in high-concurrency production

[weipengd]

Hi Team, I am evaluating log collection for a high-concurrency Java environment. We are already using OTel SDK for Metrics and Traces and we love it. However, I am hesitant about using the SDK/Agent for Logs.

My concerns:

Will intercepting logs and exporting OTLP within the same JVM significantly impact business performance (GC pressure, CPU spikes) compared to just writing to a file?

Is it common industry practice to keep Metrics/Traces in the SDK but use a Filelog Receiver for Logs to ensure resource isolation?

I'd love to hear about production experiences or benchmarks regarding the overhead of the Java Agent's log appender.

[jack-berg]

This is a loaded question! But some thoughts:

• In opentelemetry-java, we take performance seriously and have done a fair bit of work to minimize the memory impact of our SDKs and OTLP exporters. This is true for logs as well as metrics and traces.
• Not all disks are created equal. Applications with low perf disks may be impacted more by a slow disk than a slow network. I think a lot of us have this intuition that writing to a disk is the safest and lowest overhead thing you can do with a log but I don't think that's always true.
• Volume matters. A high throughput app with low level debug logging is different than a high throughput app with a higher severity threshold, or a low throughput app with low level debug logging. Networks and the opentelemetry SDK can become constrained with high enough log volume, but then again so can disks.
• Why are logs different than traces and metrics? If you're already ok with sending traces and metrics to a network location directly from the app, are logs really that different? In some cases, probably yes. But tracing can already have some pretty heavy load, so if tracing isn't a problem, what makes logs different?
• Ultimately, emitting logs directly from an app over OTLP has a number of advantages, and so is my default recommendation:
  • Its more convenient than setting up / managing a separate channel just for logs.
  • Its the best way to preserve shared context. In otel, metrics logs and traces from the same app share the same resource attributes, which assists with correlation. Metrics are tied to traces through exemplars. Logs are tied to traces through embedding trace context in logs. Its possible to write logs to files and scrape those files all in such a way that resource and trace context is preserved, but its definitely more tricky / error prone.

See also https://opentelemetry.io/docs/specs/otel/logs/#legacy-first-party-applications-logs for a discussion of the difference between these two approaches.

I'd love to hear about production experiences or benchmarks regarding the overhead of the Java Agent's log appender.

I don't have those, but the otel java agent emits logs over OTLP by default and I can't recall hearing people complaining about perf or asking to turn it off. That's a data point in itself.

[weipengd]

Hi @jack-berg,

Thank you for the detailed insights! I truly appreciate the perspective of the OTel maintainers. However, I’d like to push back a bit from the perspective of enterprise-grade production stability.

In my specific case, since I am using the OTel SDK to manually inject trace_id and span_id into the logs (via Logback MDC), the correlation issue you mentioned is already solved. The logs are as "trace-aware" as they would be via OTLP.

Given that, my primary concern is Resource Isolation. In a high-concurrency, mission-critical environment:

Memory & GC Risk: If we use the OTLP exporter, we are essentially buffering logs in the JVM heap. If the network jitters or the Collector becomes slow, this buffer will grow, increasing GC pressure or even risking OOM.

Backpressure: Does the OTel Java SDK provide a guaranteed circuit breaker or a disk-spooling fallback? Without these, the observability tool becomes a potential "noisy neighbor" to the business logic.

The "Safety Net" of Files: Writing to a local disk (with an async appender) acts as a high-throughput, physical buffer. It decouples the application’s survival from the health of the observability pipeline.

My question to the team is: For a Tier-0 production service, can the OTel Java Agent/SDK guarantee that log collection will never impact the application's p99 latency or stability during a telemetry backend slowdown?

In many enterprise architectures, "Stability > Convenience." If the SDK cannot provide strict resource limits or disk-fallback, the Filelog Receiver approach remains the only "safe" choice for us.

I'd love to hear how the OTel team views these reliability trade-offs for extreme scale.