Problem or motivation
We should test and document how packaging our examples into Docker containers and pushing their traces agentevals trough an OTel collector works.
Proposed solution
We have an in-flight PR packaging agentevals up for Kubernetes here: #66.
This far, we've validated that when running agentevals in Kubernetes then examples can push directly to agentevals when its OTLP endpoint is port-forwarded. Based on this an intermediate OTel collector should work, but needs to be tested.
The documentation can cover scenarios such as adding example processors (filtering, sampling) to avoid overloading agentevals backend as it's not yet designed for continuous evaluations.
Alternatives considered
No response
Additional context
We can also publish example agents as Docker containers and have automated testing for them as a follow-up.
Human confirmation
Problem or motivation
We should test and document how packaging our examples into Docker containers and pushing their traces agentevals trough an OTel collector works.
Proposed solution
We have an in-flight PR packaging agentevals up for Kubernetes here: #66.
This far, we've validated that when running agentevals in Kubernetes then examples can push directly to agentevals when its OTLP endpoint is port-forwarded. Based on this an intermediate OTel collector should work, but needs to be tested.
The documentation can cover scenarios such as adding example processors (filtering, sampling) to avoid overloading agentevals backend as it's not yet designed for continuous evaluations.
Alternatives considered
No response
Additional context
We can also publish example agents as Docker containers and have automated testing for them as a follow-up.
Human confirmation