Practical architecture and implementation guide using Spring Boot 4.1.1, Micrometer, Prometheus, and Grafana

Audience: Senior software engineers, tech leads, engineering managers, platform engineers, and SREs.
Technical baseline: Java 21, Spring Boot 4.1.1, Maven, Docker Compose. Metrics and operational thresholds are illustrative; calibrate them against your workloads and SLOs.

1. Executive Summary: The Business Imperative

Distributed systems don't fail loudly. They just degrade. Connection pool saturates, Kafka consumer falls behind, or one dependency adds 300 milliseconds to a critical request. The application keeps returning some successful responses, liveness checks stay green, and the first clear incident report comes from a customer.

Observability should narrow the gap between when degradation begins and when a team can explain and mitigate it. Metrics are often the first useful clue: they compress system behavior into measurable time series. When paired with logs and distributed traces, they make incidents less mysterious and system design more evidence-driven.

For an e-commerce checkout flow, rising payment-authorization p99 might produce retries, thread exhaustion, timeouts, and abandoned purchases. The commercial impact does not wait for a full outage. An observability strategy should detect partial failure, surface the customer-facing effect, and guide responders toward the component creating it.

The business case rests on several measurable outcomes:

Outcome How well-designed metrics contribute
Lower mean time to detect (MTTD) Alert on meaningful changes to error ratio, latency, traffic, and business outcomes.
Lower mean time to recover/restore (MTTR) Narrow the investigation to the failing region, workload, resource, or dependency.
SLA/SLO compliance Quantify reliability through agreed service-level indicators (SLIs) and error budgets.
Capacity planning Understand utilization, growth trends, queueing, and headroom before demand peaks.
Fraud and anomaly detection Highlight unusual authorization declines, transaction spikes, or changes in rejection patterns.
Revenue protection Identify degrading customer journeys before prolonged service disruption.
Engineering efficiency Replace repeated manual log searches with dependable, reusable operational views.

Metrics are not a substitute for product analytics or a fraud-detection pipeline. Aggregation is exactly what makes metrics affordable to query, but it removes the per-event detail those systems need. Use each data source for the problem it solves.

1.1 Simple ROI model

Consider a hypothetical checkout service experiencing six important incidents annually, each affecting a business flow worth an estimated $8,000 per minute. If clearer telemetry helps shorten average recovery by 15 minutes, estimated annual revenue exposure avoided is:

6 incidents × 15 minutes avoided × $8,000/minute = $720,000

This is an illustration, not a universal ROI formula or promise. Real analysis should incorporate incident severity, traffic variability, mitigation effectiveness, the cost of telemetry storage, on-call labor, and engineering implementation. The objective is to move from reactive firefighting to proactive reliability engineering: review SLO trends during planning, treat capacity as a forecastable constraint, and improve instrumentation after every meaningful incident.

2. Foundations: Observability and the Role of Metrics

2.1 Monitoring versus observability

Traditional monitoring asks whether predefined conditions are healthy: is the process running, are requests failing, and is CPU above a threshold ? Observability means collecting enough signals to investigate questions that were not necessarily anticipated when the service was deployed. Why is latency high only for one endpoint and one region ? Which dependency caused the delay? Did the last rollout change queue processing behavior?

Monitoring is an outcome of observability, not its enemy. Reliable system needs both known-condition alerts and investigative telemetry.

2.2 Metrics, logs, and traces

flowchart LR
    A[Distributed services] --> M[Metrics: rates, ratios, durations]
    A --> L[Logs: discrete events and context]
    A --> T[Traces: end-to-end request paths]
    M --> O[Detect impact]
    L --> O
    T --> O
    O --> R[Diagnose, mitigate, verify]

Metrics are aggregated numerical observations indexed by time and labels. Logs preserve individual events, ideally with structured attributes and correlation identifiers. Traces join operations across services and help identify where time was spent. Useful investigation commonly starts with a latency panel, pivots to a trace of a slow request, and then checks logs at the failing span.

Prometheus counter might look like:

checkout_attempts_total{application="checkout-service",region="us-east-1",result="success"} 15280

This tells us how many observed successes occurred since the process's counter began, not which customers placed orders. To derive a rate, query the counter over a window; raw counters reset when a process restarts.

2.3 Types of metrics

Instrument Behavior Good examples
Counter Increases as events occur; can reset on restart Requests, failures, completed checkouts
Gauge Current value can rise or fall In-flight operations, queue depth, connections
Timer Measures number and duration of operations Request handling and checkout processing
Distribution summary Records distributions of non-time values Payload bytes, batch sizes, cart item counts
Histogram Represents counts in value buckets Aggregatable request-latency distribution

An average is rarely enough for latency-sensitive workflows. A small fraction of extremely slow requests can be damaging even when mean response time is acceptable. Prefer bucket-based histograms when you need fleet-wide p95/p99 estimates. Remember that quantiles derived from histograms are estimates whose accuracy depends on bucket boundaries and traffic volume.

2.4 Golden Signals, RED, and USE

Google's Four Golden Signals are Latency, Traffic, Errors, and Saturation. They work well as a compact summary of service health. RED-Rate, Errors, Duration-organizes request-oriented metrics for API services. USE-Utilization, Saturation, Errors-shifts attention to the underlying resources: CPU, thread pools, disk, network, database connections, and message consumers.

These are complementary frameworks. RED can reveal a dramatic rise in API latency, while USE identifies the exhausted database pool behind it. Neither framework can tell you whether an order was actually fulfilled, which is why domain metrics matter: successful checkouts, payment authorization failures, order-creation rate, reservation rejections, processing volume, or Kafka event age.

Define domain metric semantics precisely. Rejected card due to insufficient funds may be a business outcome but not a service-availability failure. Payment gateway timeout normally is a service failure even if your HTTP API wraps it in a 200 response. Alerting quality is limited by classification quality.

3. Architecture: From Instrumentation to Action

3.1 Reference telemetry pipeline

flowchart TB
    subgraph Services[Application instances]
      A[Checkout / Spring Boot 4]
      B[Payment / Spring Boot 4]
      C[Inventory / Other runtime]
    end
    A --> I[Micrometer and framework instrumentation]
    B --> I
    C --> O[OpenTelemetry instrumentation]
    I -->|HTTP scrape: /actuator/prometheus| P[Prometheus collection]
    O -->|OTLP push, optional| K[OpenTelemetry Collector]
    K --> V[Compatible backend or Prometheus receiver]
    P --> S[Prometheus TSDB]
    P -. remote write .-> V
    S --> G[Grafana dashboards]
    V --> G
    S --> Q[Prometheus alert rules]
    Q --> AM[Alertmanager]
    AM --> ON[On-call routing]

The logical stages are: instrumentation → scrape or push collection → time-series storage → visualization and alerting. In a real fleet, Prometheus should ordinarily scrape each application instance, not randomly sample replicas through a load-balanced address. The network, authentication, service discovery, and storage topology must match deployment boundaries.

3.2 Micrometer and OpenTelemetry

Micrometer offers a convenient Java MeterRegistry facade, common tags, timers, counters, gauges, and integration with Spring Boot Actuator. OpenTelemetry provides cross-language telemetry conventions and SDKs for metrics, traces, and logs, often with an OpenTelemetry Collector for processing and routing.

For this Spring Boot 4 guide, Micrometer is the primary metrics API. Spring Boot can export Micrometer metrics through a Prometheus registry or through OTLP. Do not assume Spring Boot's Micrometer meters automatically flow through the OpenTelemetry Java SDK's SdkMeterProvider. Spring's documented integration treats those as distinct paths. Choose a path deliberately so you do not duplicate telemetry or silently omit a registry.

3.3 Backends and operational layers

  • Prometheus: straightforward pull-style scraping, local TSDB, PromQL, and alert rule evaluation; an effective starting point.
  • VictoriaMetrics: an alternative metrics storage platform that can support longer retention or scaled ingestion, depending on deployment architecture.
  • M3 / M3DB: distributed time-series infrastructure for larger workloads where the team can support the corresponding operational complexity.
  • Grafana: dashboards and investigation workflows across compatible sources.
  • Alertmanager: grouping, routing, inhibition, and notification of alerts evaluated by Prometheus.

Choose based on measured ingestion rate, active series, query load, availability objectives, retention, tenant isolation, and team capability.

3.4 Pull vs. push

Pull: Prometheus polls an endpoint at a defined interval. Target reachability and scrape failures are visible, and applications need not maintain exporter connections. It requires network access from collector to application.

Push: Application or collector exports telemetry outward, often using OTLP. This is useful across network boundaries or with a common collector-based pipeline. The Prometheus Pushgateway is specialized for certain short-lived batch jobs; it is not a general replacement for scrape-based instrumentation in long-running services..

3.5 Cardinality, retention, and overhead

Metric cardinality is the product of distinct combinations of labels. A metric labelled by 20 services, 3 regions, and 2 outcomes can produce up to 120 series. Add 500,000 unique customer IDs and the theoretical maximum becomes 60 million series. Not all combinations will occur, but the explosion is real. Avoid request IDs, transaction IDs, emails, raw exception messages, or arbitrary URL paths as metric tags.

Scrape frequency and retention also matter. At 100,000 active series scraped every 15 seconds, the pipeline collects about 576 million samples/day. Compressed values alone do not describe storage use; indexing, churn, exemplars, WAL, retention, and replication add costs. Start with a scrape interval around 15-30 seconds for general service telemetry, measure ingestion and query pressure, and adjust based on the system's detection needs. Consider short high-resolution retention plus long-term aggregation where required.

4. Hands-On Implementation: Spring Boot 4.1.1 + Micrometer

We will build a minimal checkout API, expose its Actuator endpoint using authentication, scrape it with Prometheus, and explore the output in Grafana. The sample is designed to be adaptable, not a complete checkout implementation. It uses simulated processing and development-only credentials.

4.1 Prerequisites and project structure

Install Java 21, Maven 3.9+, Docker Engine, and Docker Compose. Use the following layout:

observability-demo/
├── pom.xml
├── Dockerfile
├── docker-compose.yml
├── prometheus/
│   ├── prometheus.yml
│   └── alerts.yml               # Optional
├── grafana/provisioning/datasources/datasource.yml
└── src/main/
    ├── java/com/example/observability/
    │   ├── ObservabilityApplication.java
    │   ├── CheckoutController.java
    │   ├── CheckoutService.java
    │   └── SecurityConfig.java
    └── resources/application.yml

Spring Boot 4.x migration note: Use spring-boot-starter-webmvc rather than the deprecated spring-boot-starter-web for a new servlet MVC service. Actuator and Micrometer still work together, but some auto-configuration classes and security package names changed under Spring Boot 4's modularization. This guide uses the Spring Boot 4.1.1 API package for EndpointRequest.

4.2 Maven dependencies

Create pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example</groupId>
  <artifactId>observability-demo</artifactId>
  <version>1.0.0</version>
  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>4.1.1</version>
    <relativePath/>
  </parent>
  <properties><java.version>21</java.version></properties>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-webmvc</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-security</artifactId>
    </dependency>
  </dependencies>
  <build><plugins><plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
  </plugin></plugins></build>
</project>

Do not manually select a Micrometer version unless you have a tested dependency-management reason. Let the Spring Boot parent manage compatible Micrometer and Spring component versions.

4.3 Application configuration

Create src/main/resources/application.yml:

spring:
  application:
    name: checkout-service

server:
  port: 8080

management:
  endpoints:
    web:
      exposure:
        include: health,prometheus
  endpoint:
    health:
      show-details: never
  metrics:
    tags:
      application: ${spring.application.name}
      region: ${REGION:us-east-1}
      env: ${APP_ENV:local}
    distribution:
      percentiles-histogram:
        http.server.requests: true

observability:
  scrape:
    username: ${METRICS_USERNAME}
    password: ${METRICS_PASSWORD}

Only expose the endpoints that are actually required. Tags such as application, region, and environment are stable operational dimensions; a customer ID is not. Spring MVC HTTP request instrumentation is automatically provided when the appropriate dependencies and auto-configuration are present. The Micrometer meter http.server.requests is exported in Prometheus format, commonly as http_server_requests_seconds_* series when appropriate histogram settings are enabled.

4.4 Application entry point

Create ObservabilityApplication.java:

package com.example.observability;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class ObservabilityApplication {
    public static void main(String[] args) {
        SpringApplication.run(ObservabilityApplication.class, args);
    }
}

4.5 Secure the metrics endpoint

Create SecurityConfig.java:

package com.example.observability;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.security.autoconfigure.actuate.web.servlet.EndpointRequest;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.annotation.Order;
import org.springframework.security.config.Customizer;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.core.userdetails.User;
import org.springframework.security.core.userdetails.UserDetailsService;
import org.springframework.security.crypto.bcrypt.BCryptPasswordEncoder;
import org.springframework.security.crypto.password.PasswordEncoder;
import org.springframework.security.provisioning.InMemoryUserDetailsManager;
import org.springframework.security.web.SecurityFilterChain;

@Configuration
public class SecurityConfig {

    @Bean
    PasswordEncoder passwordEncoder() {
        return new BCryptPasswordEncoder();
    }

    @Bean
    UserDetailsService metricsUsers(
            @Value("${observability.scrape.username}") String username,
            @Value("${observability.scrape.password}") String password,
            PasswordEncoder encoder) {
        return new InMemoryUserDetailsManager(
                User.withUsername(username)
                    .password(encoder.encode(password))
                    .roles("METRICS")
                    .build());
    }

    @Bean
    @Order(1)
    SecurityFilterChain actuatorChain(HttpSecurity http) throws Exception {
        http.securityMatcher(EndpointRequest.toAnyEndpoint())
            .authorizeHttpRequests(auth -> auth
                .requestMatchers(EndpointRequest.to("health")).permitAll()
                .requestMatchers(EndpointRequest.to("prometheus")).hasRole("METRICS")
                .anyRequest().denyAll())
            .httpBasic(Customizer.withDefaults());
        return http.build();
    }

    @Bean
    @Order(2)
    SecurityFilterChain demoChain(HttpSecurity http) throws Exception {
        http.authorizeHttpRequests(auth -> auth
                .requestMatchers("/demo/**").permitAll()
                .anyRequest().denyAll());
        return http.build();
    }
}

The EndpointRequest import above is intentionally Spring Boot 4.x-specific. The publicly accessible demo route is not suitable for a real payment API. HTTP Basic should run behind TLS and private network access in production; inject credentials through a secret manager, not a committed file. A separate management network or port is often preferable. For real business operations, enforce authorization, validation, and idempotency.

4.6 Custom Counter, Timer, and Gauge

Create CheckoutService.java:

package com.example.observability;

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import org.springframework.stereotype.Service;

import java.time.Duration;
import java.util.concurrent.atomic.AtomicInteger;

@Service
public class CheckoutService {
    private final Counter success;
    private final Counter failure;
    private final Timer checkoutLatency;
    private final AtomicInteger inFlight = new AtomicInteger();

    public CheckoutService(MeterRegistry registry) {
        success = Counter.builder("checkout.attempts")
            .description("Completed successful checkout attempts")
            .tag("result", "success")
            .register(registry);

        failure = Counter.builder("checkout.attempts")
            .description("Completed failed checkout attempts")
            .tag("result", "failure")
            .register(registry);

        checkoutLatency = Timer.builder("checkout.processing")
            .description("Checkout operation processing time")
            .publishPercentileHistogram()
            .serviceLevelObjectives(
                Duration.ofMillis(100),
                Duration.ofMillis(250),
                Duration.ofMillis(500),
                Duration.ofSeconds(1))
            .minimumExpectedValue(Duration.ofMillis(1))
            .maximumExpectedValue(Duration.ofSeconds(5))
            .register(registry);

        Gauge.builder("checkout.inflight", inFlight, AtomicInteger::get)
            .description("Currently active checkout operations")
            .register(registry);
    }

    public boolean process(boolean simulatedFailure) {
        inFlight.incrementAndGet();
        try {
            return checkoutLatency.record(() -> {
                simulateWork();
                if (simulatedFailure) {
                    failure.increment();
                    return false;
                }
                success.increment();
                return true;
            });
        } finally {
            inFlight.decrementAndGet();
        }
    }

    private void simulateWork() {
        try {
            Thread.sleep(80);
        } catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
            throw new IllegalStateException("Processing interrupted", ex);
        }
    }
}

This code makes several intentional trade-offs. It registers meters once at startup; that avoids per-request meter creation. Result labels are a bounded two-value set. The AtomicInteger stays strongly referenced, which is important because Micrometer gauges may hold weak references to observed objects.finally block prevents the in-flight count from getting stuck after an exception. The timer publishes an aggregatable histogram with boundaries relevant to the illustrative latency objective.

Production refinement: model every terminal outcome, including unexpected exceptions, cancellation, and timeouts. This particular demo counts only simulated successes and failures; interruptions raise an exception without incrementing either counter. That is intentionally simpler than a production-grade outcome classification. A real SLI must have an explicit denominator and documented exclusions.

4.7 Demo HTTP controller

Create CheckoutController.java:

package com.example.observability;

import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/demo")
public class CheckoutController {
    private final CheckoutService service;

    public CheckoutController(CheckoutService service) {
        this.service = service;
    }

    @GetMapping("/checkouts")
    public ResponseEntity<String> checkout(
            @RequestParam(defaultValue = "false") boolean fail) {
        if (service.process(fail)) {
            return ResponseEntity.ok("Checkout succeeded");
        }
        return ResponseEntity.unprocessableEntity()
            .body("Simulated checkout failure");
    }
}

Using GET for a state-changing checkout is not good API design; here it is simply a harmless demonstration endpoint. Real checkout should normally use POST with authorization and idempotency. Also notice the business failure returns 422, not 5xx. An alert based solely on server errors will miss business processing failures unless you define appropriate domain indicators.

4.8 Dockerfile

Create Dockerfile:

FROM maven:3.9-eclipse-temurin-21 AS build
WORKDIR /workspace
COPY pom.xml .
RUN mvn -B dependency:go-offline
COPY src ./src
RUN mvn -B clean package -DskipTests

FROM eclipse-temurin:21-jre
WORKDIR /app
RUN groupadd --system appgroup && useradd --system --gid appgroup appuser
COPY --from=build /workspace/target/observability-demo-1.0.0.jar app.jar
USER appuser
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "/app/app.jar"]

This multi-stage build keeps build tools out of the runtime image and runs the app as a non-root account. In production, additionally apply image scanning, tested version updates, resource constraints, and a dependency security policy.

4.9 Prometheus and Grafana configuration

Create prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: checkout-service
    metrics_path: /actuator/prometheus
    scrape_interval: 15s
    scrape_timeout: 10s
    basic_auth:
      username: metrics-reader
      password: local-demo-password
    static_configs:
      - targets: ["checkout-service:8080"]

Create grafana/provisioning/datasources/datasource.yml:

apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

Create docker-compose.yml:

services:
  checkout-service:
    build: .
    environment:
      REGION: us-east-1
      APP_ENV: local
      METRICS_USERNAME: metrics-reader
      METRICS_PASSWORD: local-demo-password
    ports:
      - "127.0.0.1:8080:8080"
    networks: [observability]

  prometheus:
    image: prom/prometheus:v3.5.0
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=15d"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"
    depends_on: [checkout-service]
    networks: [observability]

  grafana:
    image: grafana/grafana:12.1.0
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: local-grafana-password
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    ports:
      - "127.0.0.1:3000:3000"
    depends_on: [prometheus]
    networks: [observability]

networks:
  observability:
    driver: bridge
volumes:
  prometheus-data:
  grafana-data:

These image tags are illustrative pinned examples, not a claim that they are the latest or security-approved images. Prefer image digests and your organization's approved versions for production. The demo credentials are intentionally insecure and only appropriate for an isolated local environment. depends_on supplies startup order, not a readiness guarantee.

4.10 Run and verify

From the project directory:

docker compose up --build -d
docker compose ps
curl http://localhost:8080/actuator/health

Expect a health response indicating UP. The unauthenticated metrics endpoint should return 401:

curl -i http://localhost:8080/actuator/prometheus

With the local scrape credentials it should expose metrics:

curl -u metrics-reader:local-demo-password \
  http://localhost:8080/actuator/prometheus

Generate requests:

for i in $(seq 1 50); do
  curl -s http://localhost:8080/demo/checkouts > /dev/null
done
for i in $(seq 1 10); do
  curl -s 'http://localhost:8080/demo/checkouts?fail=true' > /dev/null
done

Then inspect relevant metrics:

curl -s -u metrics-reader:local-demo-password \
  http://localhost:8080/actuator/prometheus | grep checkout_

You should find series resembling checkout_attempts_total, checkout_processing_seconds_count, checkout_processing_seconds_sum, checkout_processing_seconds_bucket, and checkout_inflight. In Prometheus at http://localhost:9090/targets, verify the checkout target is UP. In Grafana at http://localhost:3000, sign in with the Compose demo credentials and query the provisioned Prometheus source in Explore.

5. From Raw Metrics to Useful Dashboards

An effective observability dashboard should provide a clear, concise view of service health while enabling engineers to identify potential problems before they escalate into major incidents. The objective is not to visualize every metric available in the system, but to present the most relevant information in a way that supports faster troubleshooting and informed operational decisions.

Start with a service-level overview built around the key indicators of application reliability and performance:

  • Request Rate: Monitor the number of incoming requests per second or minute to understand traffic patterns, identify unexpected spikes, and recognize sudden drops in demand. A significant decrease in request volume may indicate an upstream failure, routing issue, or service disruption.
  • Technical Error Ratio: Track the percentage of requests resulting in technical failures, such as HTTP 5xx responses, unhandled exceptions, and timeouts. This helps engineers understand whether a service is successfully handling incoming traffic and whether failures are increasing over time.
  • p95 and p99 Latency: Measure tail latency rather than relying solely on average response times. The 95th and 99th percentile measurements reveal performance degradation affecting slower requests, which can be hidden by otherwise healthy averages. Increasing tail latency may indicate dependency bottlenecks, thread contention, garbage collection pauses, or resource exhaustion.
  • Resource Saturation: Monitor critical resources such as CPU, JVM memory, database connection pools, thread pools, and message queue backlogs. These measurements help engineers determine whether performance degradation is caused by capacity limitations or infrastructure pressure.

Once the service-level overview is established, introduce a dedicated business metrics section that connects application behavior to actual business outcomes.

For an e-commerce checkout service, this could include

  • Checkout Attempt Rate: The number of checkout attempts initiated or completed within a given time window, depending on the metric's definition.
  • Checkout Success Rate: The proportion of completed checkout attempts that result in successful transactions.
  • Checkout Failure Ratio: The percentage of checkout attempts that fail because of payment processing errors, inventory issues, timeouts, or other business and technical conditions.
  • Checkout Processing Duration: The time required to process a checkout operation, helping identify slow business transactions even when the underlying API appears healthy.

These business metrics provide context that infrastructure and HTTP metrics alone cannot offer. For example, a checkout endpoint may consistently return HTTP 200 responses while payment authorizations are being declined or downstream order creation is failing. From an infrastructure perspective, the service appears healthy. From a business perspective, customers may be unable to complete purchases.

Well-designed dashboard should make this discrepancy immediately visible.

Organize dashboards using progressive levels of detail. The first level should answer whether the service is healthy and whether customers are affected. The second should expose the specific symptoms, such as increased checkout failures or elevated p99 latency. The third should provide drill-down views into dependencies, database connections, JVM behavior, and other relevant components.

During an incident, an engineer should be able to move naturally from identifying the symptom to narrowing down the probable cause, without navigating through dozens of unrelated charts.

Dashboards should also provide consistent filtering by service, environment, region, and deployment version where these dimensions are available and have controlled cardinality. This makes it easier to compare behavior across instances, isolate regional problems, and correlate performance degradation with recent releases.

Finally, avoid treating dashboards as static reporting artifacts. Review them after incidents, remove panels that offer little diagnostic value, and add measurements that would have shortened the investigation.

Successful observability dashboard is not the one displaying the largest number of metrics. It is the one that helps engineers quickly understand what is failing, assess customer impact, and determine where to investigate next.

Begin with a concise service health overview that highlights essential performance and reliability indicators, including request throughput, technical error rates, p95 and p99 response times, and resource saturation. Together, these metrics provide a clear picture of how the service is performing and help identify early signs of degradation or potential bottlenecks. Add a business panel showing checkout attempts and failure ratios. Dashboard should help an engineer decide where to look next, not merely showcase every collected data point..

5.1 Request rate

sum(rate(http_server_requests_seconds_count{application="checkout-service"}[5m]))

5.2 HTTP server error ratio

sum(rate(http_server_requests_seconds_count{application="checkout-service",status=~"5.."}[5m]))
/
clamp_min(sum(rate(http_server_requests_seconds_count{application="checkout-service"}[5m])), 1e-9)

Service returning all 422 failures would still show a near-zero server-error ratio. Combine this with a business success indicator rather than assuming HTTP 5xx covers all critical outcomes.

5.3 Checkout failures and success throughput

sum(rate(checkout_attempts_total{result="failure"}[5m]))
/
clamp_min(sum(rate(checkout_attempts_total[5m])), 1e-9)

sum(rate(checkout_attempts_total{result="success"}[5m]))

Treat result="failure" as a demonstration classification. In a real system, track technical failures separately from legitimate business declines and never interpret absent requests as guaranteed health.

5.4 p95 latency from a classic histogram

histogram_quantile(
  0.95,
  sum by (le) (rate(checkout_processing_seconds_bucket[5m]))
)

To inspect regional variation:

histogram_quantile(
  0.95,
  sum by (region, le) (rate(checkout_processing_seconds_bucket[5m]))
)

The le label must be retained for a classic histogram. Do not average p95 values from individual instances: quantiles are not generally aggregatable. Aggregate buckets first, then estimate the percentile. This processing timer measures time inside the checkout service, not network, gateway, retry, or browser latency. Select an end-to-end observation point for customer experience SLOs.

5.5 Proportion of processing operations under 500 ms

sum(rate(checkout_processing_seconds_bucket{le="0.5"}[5m]))
/
clamp_min(sum(rate(checkout_processing_seconds_count[5m])), 1e-9)

This ratio relies on the configured 500 ms bucket and correctly observed terminal operations. It is not automatically a complete customer latency SLI. Timeout handling, traffic eligibility, and failed operations must be defined explicitly before you use it for an error budget.

6. Anti-Patterns and Better Engineering Choices

6.1 High-cardinality dimensions

Never tag each counter with a customer or transaction identifier:

// Anti-pattern: unbounded, high-churn labels.
Counter.builder("checkout.attempts")
    .tag("customer_id", customerId)
    .tag("transaction_id", transactionId)
    .register(registry)
    .increment();

Every distinct label combination can become a separate time series. This inflates memory, index size, ingestion cost, and query latency. Use bounded operational categories such as payment_method=card|wallet or result=success|technical_failure|declined, with explicit validation. Carry individual transaction IDs through structured logs and traces instead. Removing meters after every request does not solve the backend churn from unbounded series.

6.2 Poor naming and inconsistent semantics

Prefer semantic Micrometer names such as checkout.attempts, checkout.processing, and checkout.inflight. Let the registry create Prometheus-compatible unit and type suffixes; for example, a timer may export checkout_processing_seconds_bucket. Define whether checkout.attempts counts starts, completions, or outcomes and do not silently change its meaning after dashboards and alerts depend on it.

6.3 Alerting on causes rather than customer symptoms

Paging whenever CPU exceeds 80% creates noise. CPU is a useful diagnostic, but an elevated value may be normal during peak traffic. Page primarily on actionable customer-visible SLO violations and use CPU, thread counts, pool saturation, or Kafka lag as supporting evidence. Infrastructure alarms still have a place when they represent imminent danger: for example, rapid disk exhaustion or loss of redundancy.

6.4 An SLO and burn-rate example

Suppose checkout availability has a 99.9% request-based SLO over 30 days. Its allowed failure fraction is 0.001. A burn rate of 14.4 means the service is spending budget at 14.4 times the nominal rate. A simplified two-window alert requires both the 5-minute and 1-hour error ratios to exceed 14.4 × 0.001 = 0.0144 (1.44%).

groups:
  - name: checkout-slo
    rules:
      - alert: CheckoutFastErrorBudgetBurn
        expr: |
          (
            sum(rate(checkout_attempts_total{result="failure"}[5m]))
            /
            clamp_min(sum(rate(checkout_attempts_total[5m])), 1e-9)
          ) > 0.0144
          and
          (
            sum(rate(checkout_attempts_total{result="failure"}[1h]))
            /
            clamp_min(sum(rate(checkout_attempts_total[1h])), 1e-9)
          ) > 0.0144
        labels:
          severity: critical
          service: checkout-service
        annotations:
          summary: "Checkout error budget is burning rapidly"
          description: "Check dependency latency, recent releases, and technical checkout failures."

For Prometheus to load this optional rule, add rule_files: ["/etc/prometheus/alerts.yml"] to prometheus.yml and mount the file into the Prometheus container. A firing Prometheus rule does not automatically page anyone: configure Alertmanager and its notification routes separately. For production, use a full fast/slow burn-rate strategy, minimum traffic expectations where appropriate, and recording rules for reusable SLIs. Most importantly, define which failures actually count against the SLO; customer-declined payments usually do not.

6.5 Blind spots beyond HTTP

  • Thread pools: watch active threads, queue depth, rejected tasks, and request concurrency.
  • Garbage collection: inspect pause durations and allocation pressure, not only heap usage.
  • Database connection pools: check active, idle, pending, and acquisition-timeout metrics, including HikariCP where instrumented.
  • Kafka: monitor consumer lag alongside processing rate, retry count, rebalance events, and the age of the oldest unprocessed message.
  • External dependencies: track call duration, timeouts, circuit-breaker state, and retry amplification.

A RED dashboard may show that customers are waiting. USE and dependency metrics tell engineers why, while traces and logs explain the individual slow paths.

7. Troubleshooting and Operational Runbook

When a graph goes blank, first determine whether the service failed or the telemetry pipeline failed. A missing time series should not automatically be treated as a healthy zero.

Symptom First check Likely direction
/actuator/prometheus returns 404 Endpoint exposure and registry dependency Incorrect path or missing actuator/registry
Endpoint returns 401 Prometheus basic_auth and application credentials No or bad authentication
Endpoint returns 403 Spring Security role and network gateway policy Authenticated but forbidden
Prometheus target reports DOWN /targets last scrape error Network, auth, TLS, timeout, or path
Prometheus has data, Grafana blank Data source, time picker, metric and label filters Query or visualization mismatch
Some custom metrics absent Generate instrumented traffic; inspect meter names Lazy registration, filter, or naming mismatch
Scrape duration increases scrape_duration_seconds, payload size, cardinality Slow endpoint or large exposition

7.1 Check endpoint and credentials

curl -i http://localhost:8080/actuator/health
curl -i http://localhost:8080/actuator/prometheus
curl -i -u metrics-reader:local-demo-password \
  http://localhost:8080/actuator/prometheus

For this sample, health should return UP, unauthenticated metrics should return 401, and authenticated metrics should return 200 with the exposition format. If local access works but Prometheus cannot scrape, investigate connectivity from Prometheus's network, not just from your laptop. Within a container, localhost means that container, not the application service. The Compose target correctly uses checkout-service:8080.

7.2 Inspect Prometheus scrape health

Open http://localhost:9090/targets and query:

up{job="checkout-service"}

1 means the most recent scrape succeeded; 0 means it failed; no series may mean the target was not discovered or the data is absent. Distinguish missing from zero in alerts where telemetry continuity matters. Inspect scrape duration and samples:

scrape_duration_seconds{job="checkout-service"}

scrape_samples_scraped{job="checkout-service"}

If scrape time approaches the configured timeout, check endpoint CPU pressure, network latency, response payload growth, excess histogram buckets, unbounded tags, and gauges that perform expensive work on observation. Avoid gauges that execute a database count query synchronously on every scrape; a collection endpoint must not amplify an incident.

7.3 Check Docker and firewall routing

docker compose logs checkout-service
docker compose logs prometheus
docker compose ps

On Kubernetes, check Service and EndpointSlice selection, NetworkPolicies, DNS, service discovery configuration, and any ingress or service mesh that intercepts requests. Prometheus scraping is server-to-server HTTP; browser CORS generally does not apply. A 401/403, TLS error, connection refusal, DNS failure, or request timeout is a better initial hypothesis than CORS.

7.4 Runbook sequence for real degradation

flowchart TD
  A["SLO or latency alert"] --> B{"Telemetry available?"}
  B -->|No| C["Check scrape target and collection path"]
  B -->|Yes| D["Assess RED and business outcomes"]
  C --> D
  D --> E["Identify impacted endpoints and regions"]
  E --> F["Inspect USE, pools, queues, and dependencies"]
  F --> G["Correlate with traces, logs, and deploy events"]
  G --> H["Mitigate: rollback, scale, fail over, shed load"]
  H --> I["Verify SLI and business recovery"]
  I --> J["Write incident follow-up and improve signals"]

Before changing infrastructure, establish the customer impact and affected scope. Check whether the degradation began after a rollout, load change, or dependency incident. Mitigation should follow evidence: rollback a regression, scale a saturated tier, limit retry storms, or reroute traffic when safe. Verify recovery through SLO and business metrics, not solely a green health endpoint. Finish with a blameless review of missing signals, misleading alerts, and architectural contributors.

8. Production Blueprint: Beyond the Local Demo

Service discovery. Use instance-aware discovery rather than static Compose addresses. Kubernetes service discovery or an operator-managed setup can target replicas individually and preserve instance-level scrape health.

Availability and durability. A single Prometheus server is a monitoring single point of failure. For important systems, consider redundant collectors, carefully planned storage, and remote write or a suitable long-term backend. Ensure dashboards and alert rules query the intended data sources and account for duplicates where relevant.

Secrets and network boundaries. Restrict actuator exposure; use TLS, private networking, secret management, and least-privilege credentials. A publicly reachable Prometheus endpoint can reveal internal architecture and usage patterns.

Infrastructure as code. Version-control dashboards, recording rules, alert definitions, scrape configurations, and service-level objectives. Check Prometheus rule syntax using promtool, and use CI checks to catch accidental label changes and invalid expressions. Deploy observability configuration with a review process comparable to application configuration.

Metric contracts and owners. For each important meter, document its meaning, type, unit, accepted tag values, expected cardinality, team owner, and associated SLI. Agree on these details before several teams build alerts from the same name.

Cost governance. Review active series, churn, histograms, retention, and query patterns regularly. Start with service-critical RED and USE signals; add metrics when they answer a recurring question or support a defined operational decision. Delete unused dimensions rather than endlessly expanding your collection footprint.

Incremental adoption. First standardize baseline JVM and HTTP dashboards. Then instrument critical business journeys. Next define SLOs and error budgets. Add tracing, structured logs, and dependency metrics for diagnosis. Finally, evaluate retention, high availability, and cross-team governance. Building platform complexity before operational demand is another form of overengineering.

9. Conclusion

Reliable observability is not measured by the number of metrics you collect. It is measured by how reliably engineers can answer: Are customers affected? What changed? Where is the bottleneck? What should we do? Did the mitigation work?

Spring Boot 4 and Micrometer provide a low-friction way to instrument request behavior and domain operations. Prometheus converts that instrumentation into queryable time series. Grafana makes patterns visible, and well-designed alerts translate SLO violations into actions. OpenTelemetry and collector-based pipelines extend the architecture where cross-language consistency and routing requirements justify them.

Start with meaningful service indicators, guard against unbounded labels, and validate the collection pipeline before declaring success. Instrument the business path rather than assuming that infrastructure health means customer success. Then make observability part of design reviews, incident response, capacity planning, and continuous improvement.

The goal is not more dashboards. It is fewer surprises, shorter incidents, and more dependable services.

10. Official References and Suggested Reading

  1. Spring Boot 4.1.1 release announcement
  2. Spring Boot 4.0 migration guide
  3. Spring Boot Actuator metrics
  4. Spring Boot Actuator endpoints and security
  5. Spring Boot observability and OpenTelemetry integration
  6. Spring Boot 4.1.1 EndpointRequest API
  7. Micrometer histograms and percentiles
  8. Prometheus documentation
  9. Google SRE — Monitoring Distributed Systems
  10. Google SRE Workbook — Alerting on SLOs
  11. OpenTelemetry documentation
  12. Grafana documentation

Publication note: This is a self-contained technical guide with runnable-style sample configuration, not an assertion that the exact project has been compiled and integration-tested in the reader's environment. Verify and adapt authentication, metric semantics, dependency/image versions, readiness, and operational SLO definitions before deploying to production.