Standardized metrics format
Every microservice should expose at least the following endpoints:
- /metrics - returning product metrics in Prometheus format
- /metrics/requests - returning requests metrics in Prometheus format
These endpoints should be exposed for external usage, along with the application (for example, the statutory metrics can be accessed through /services/oms-statutory/api/metrics).
Every microservice should store its request metrics as counters (distributed by at least HTTP method, endpoint and the response status).
All the metrics names should be prepended with the microservice name and the underscore (for example, the metric for total requests in the statutory module should be called statutory_requests_total).
The following metrics would be scraped periodically by Prometheus and can be visualised in Grafana in the future.
The logic behind this:
1. The prometheus instance to scrape data can (and will in production) be located on the different machine than the machine this application is running on, therefore the metrics need to be exposed externally and not only be accessible within the docker/host network.
2. We need at least 2 different sets of metrics to scrape then with the different interval (also the product metrics can be quite resource-intensive), and that's impossible with over endpoint exposing all the metrics.
3. We need to distinguish the metrics by the name to understand what service is this metric relates to, so every microservice should have the unique set of metric, that's why the prefix for each MS should be unique per body.
- /metrics - returning product metrics in Prometheus format
- /metrics/requests - returning requests metrics in Prometheus format
These endpoints should be exposed for external usage, along with the application (for example, the statutory metrics can be accessed through /services/oms-statutory/api/metrics).
Every microservice should store its request metrics as counters (distributed by at least HTTP method, endpoint and the response status).
All the metrics names should be prepended with the microservice name and the underscore (for example, the metric for total requests in the statutory module should be called statutory_requests_total).
The following metrics would be scraped periodically by Prometheus and can be visualised in Grafana in the future.
The logic behind this:
1. The prometheus instance to scrape data can (and will in production) be located on the different machine than the machine this application is running on, therefore the metrics need to be exposed externally and not only be accessible within the docker/host network.
2. We need at least 2 different sets of metrics to scrape then with the different interval (also the product metrics can be quite resource-intensive), and that's impossible with over endpoint exposing all the metrics.
3. We need to distinguish the metrics by the name to understand what service is this metric relates to, so every microservice should have the unique set of metric, that's why the prefix for each MS should be unique per body.