However, all of our internal services use LogNet’s awesome SpringBoot GRPC library to communicate but there’s no native Micrometer support. GRPC itself does have internal metrics but they aren’t yet exposed to Spring in that GRPC library. Since we are a tiny startup with limited resources, we did some simple things to get Micrometer hooked up to our GRPC services for some basic metrics.
Our Micrometer setup was to include the dependency in our service’s build file:
And since these are internal services, we exposed everything:
Then for every service, we have the HTTP endpoints
$HOST:$PORT/actuator/prometheus available for use.
We run things in Kubernetes, so we first add the following annotations to our service pods to make them discoverable by Prometheus.
And we add the following job to Prometheus Server’s prometheus.yml to discover and scrape pods.
This job is already included by default with the Prometheus Helm chart.
We went with the standard Spring/Micrometer generic method timing approach for this. The upside was that it was trivial to implement, but the downside is that we have to remember to annotate each GRPC method.
@Configuration class, we added a
And then for every GRPC call, we throw on a
This adds then adds the GRPC method metrics to the Prometheus actuator under the
With that getting pulled into Prometheus, we can then do things like get the average length per GRPC call using PromQL like so:
For this, we decided to hook in a Micrometer registry counter into our existing generic GRPC exception handler, which lives in an internal shared library that all GRPC services automatically pull in via our common Gradle platform.
All we did here was to add the
MeterRegistry to the constructor, so it gets set by the Spring context. Then we use that
MeterRegistry instance to increment a counter with the full class name as a
Tag in the catch block.
Then each service gets the context’s MeterRegistry autowired into a config constructor and just sets it on the exception handler bean:
With those in place, the
/actuator/prometheus endpoint now has a new counter with the full class name of the exception as a tag:
Which in PromQL then lets you do stuff like: