Bounded GenAI metrics from Otel traces with O raw prompts

Hi friends,

Author here. Like to present a project I've been working on - Otelcol-GenAI-Sketches, a Otel connector that works on genai/agent traces, and summarizes them (via sketches), to produce:

() exact request counts/totals for tokens; () distinct users/prompts (estimates) (*) 'heavy' items in terms of tokens

Why does this exist: I'm a data scientist by profession, and needed to look at Otel traces, but soon ran into cardinality and portability issues, along with privacy issues (raw prompts being transported elsewhere, like for analysis in a notebook env is a no-no). I wrote a sketchlib first (llm-sketchkit) and then the connector for Otel collectors.

To try it out, just:

(i) git clone https://github.com/llm-measurement/otelcol-genai-sketches.gi... (ii) cd otelcol-genai-sketches (iii) export GENAI_SKETCH_SECRET="$(openssl rand -hex 32)" (iv) make example-up (v) profit!

This would start a demo app that includes a Otel collector, prometheus server/db and a grafana dashboard.

The repos contain benchmarks, accuracy results and some hopefully helpful answers to the obvious question -- why not just use Datasketches?!

Either way, will be grateful if you can try it out, specially on your workloads, and give me any/all feedback.

Thanks for reading!