You may be interested in this: https://github.com/open-telemetry/otel-arrow#benchmark-summa...

Those are interesting results! I'm not surprised it works a lot better for metrics than logs and traces. Something I'd really love to have for logs/traces processing is the ability to query clp[1][2] with a dataframe interface (e.g. datafusion [3]). While I'm on that subject, I'd also prefer that interface for metrics processing. I don't need real-time streaming metrics graphs, it's perfectly fine to compute one on-demand.

I suspect something like clp is the way to go for logs-like data, that is, low entropy text with a lot of numerical content.

[1] https://www.uber.com/blog/reducing-logging-cost-by-two-order... [2] https://www.uber.com/blog/modernizing-logging-with-clp-ii/ [3] https://github.com/apache/datafusion

In my experience, Immutables really helps address some of Java's verbosity problem. Introducing it into my project cut 2,500 LOC (~10% of total code base), https://github.com/glowroot/glowroot/commit/7bdd3ff