Observability

As discussed in the Configuration chapter, ProSA uses the Opentelemetry stack to provide metrics, traces and logs. All required settings are described in this chapter.

This section explains how you can use observability features for your own purposes. When you create an adaptor, you may want to generate custom metrics, traces or logs from relevant data you are processing.

It is important to understand how to implement these features within ProSA, as ProSA handles much of the integration for you.

Metrics

Metrics in ProSA are managed using the OpenTelemetry Meter. With the meter, you can declare counters, gauges, and more.

A meter is created from the main task or from processors. You create your metrics using this meter object.

fn create_metrics<M>(proc_param: prosa::core::proc::ProcParam<M>)
where
  M: Sized + Clone + Tvf,
{
  // Get a meter to create your metrics
  let meter = proc_param.meter("prosa_metric_name");

  // Create a gauge for your metric
  let gauge_meter = meter
    .u64_gauge("prosa_gauge_metric_name")
    .with_description("Custom ProSA gauge metric")
    .init();

  // Record your value for the gauge with custom keys
  gauge_meter.record(
    42u64,
    &[
        KeyValue::new("prosa_name", "MyAwesomeProSA"),
        KeyValue::new("type", "custom"),
    ],
  );
}

If you want to create asynchronous metrics with regular updates, for example triggered by messages, you can do:

fn create_async_metrics<M>(proc_param: prosa::core::proc::ProcParam<M>) -> tokio::sync::watch::Sender<u64>
where
  M: Sized + Clone + Tvf,
{
  // Get a meter to create your metrics
  let meter = proc_param.meter("prosa_async_metric_name");

  let (value, watch_value) = tokio::sync::watch::channel(0u64);
  let _observable_gauge = meter
    .u64_observable_gauge("prosa_gauge_async_metric_name")
    .with_description("Custom ProSA gauge async metric")
    .with_callback(move |observer| {
      let value = *watch_value.borrow();
      observer.observe(
        value,
        &[
          KeyValue::new("prosa_name", "MyAwesomeProSA"),
          KeyValue::new("type", "custom"),
        ],
      );

      // You can call `observe()` multiple time if you have metrics with different labels
    })
    .init();

  value
}

fn push_metric(metric_sender: tokio::sync::watch::Sender<u64>) {
  metric_sender.send(42);
  // Alternatively, use `send_modify()` if you need to modify the value in place
}

Traces

If you package ProSA with cargo-prosa, traces are automatically configured. If you want to set it up manually in your Observability setting, use the tracing_init() method.

Once tracing is configured, you can use the Tracing crate anywhere in your code. Tracing create spans and send them automatically to the configured tracing endpoint or to stdout.

Traces is also deeply integrated into ProSA's internal messaging. ProSA messages (which inplement the prosa::core::msg::Msg trait) have an internal span that represents the flow of a message through ProSA services.

fn process_prosa_msg<M>(msg: prosa::core::msg::Msg<M>)
where
  M: Sized + Clone + Tvf,
{
  // Enter the span: record the begin time
  // When it drop at the end of function, it end the span.
  let _enter = msg.enter_span();

  tracing::info!("Add an info with the message to the entered span: {msg:?}");

  let msg_span = msg.get_span();
  // You can also retrieve the span of the message if you want to link it to something else.
}

Logs

With traces, standalone logs are often less useful, since events are better attached to spans, making it easier to know which transaction produced a given log message.

However, if you want to log messages, you can use the log crate. Like tracing, logging is provisioned automatically with ProSA.

log::info!("Generate an info log (will not be attached to a trace)");

If you prefer not to use log, you also have access to the OpenTelemetry Logger from the main task and processors. You can use this logger for full control over your observability logs.