Categorica

Native Prometheus integration with NGINX Today we are officially open-sourcing our own Prometheus statistics exporter module for NGINX! Introduction NGINX has been the de-facto choice for a reverse proxy server for quite some time now. At Categorica, we too use NGINX for this purpose, along with more traditional web-content serving. Similarly, we rely on Prometheus to collect and Grafana to aggregate and display statistics on our active deployments. Metrics collection As it stands, there’s only really one choice for statistics & metrics collection with NGINX: NGINX Prometheus Exporter, a fully featured Prometheus exporter. When it came to setting up our own metrics collection, there was something that just felt slightly off: Why does the metrics endpoint itself run its own web-server? In practise, this isn’t a big problem. Whilst it’s possible to deploy it as a systemd service (and hence just a regular binary), it is more typical to deploy NGINX Prometheus Exporter via Docker’s side-car containers; an additional complexity that doesn’t feel necessary. It does however mean that metrics collection itself evades metrics collection, à la “who watches the watchmen”. In short, this idealogical indignation did not survive the free time we had on evenings and weekends, and the resulting NGINX plugin quickly became our preferred mechanism for statistics collection. As of today, we’ve decided to share this. Usage You’ll need to be comfortable with the command line, and having a basic understanding of the GNU “configure, make, make install” dance won’t hurt, but it’s not strictly necessary. Building NGINX from source is recommended, to ensure the required features are compiled in (as your package manager’s idea of NGINX may vary from ours), although again, this is not required. Prerequisites Even if you’re not building NGINX from source, you’ll need the NGINX sources to compile the module. The supplied Makefile can do this step for you. If you’re not building NGINX from source, NGINX must have been compiled with --with-compat and --with-http_stub_status_module. A C99 compiler (e.g. Clang). PCRE, OpenSSL, and zlib libraries & headers. Additionally, these steps presume that you are building from within a Unix-like environment. Building NGINX Skip this if you already have an appropriate version of NGINX installed. From a terminal cd path-to-nginx-source; # we'll use the path to the NGINX source whilst building the plugin. export NGINX_SRC=`pwd`; ./configure --with-compat --with-http_stub_status_module; make -j; sudo make install; This will build & install NGINX, typically with the /usr/local prefix. For more information on building NGINX, you can check out their documentation. Building the plugin If you’ve already got an appropriate version of NGINX installed, then the supplied Makefile will determine what version it is (by invoking nginx -v), and download the appropriate NGINX sources required to build the plugin. If this behaviour is not desired, then you can set the path to the NGINX sources with the environment variable - NGINX_SRC - which will skip the version check & source download. Afterward, building the plugin is done with make module sudo make install This will build the plugin, and install it to the appropriate plugin location. More information can be found on our Github repository’s README.md. Running NGINX with the plugin Once the plugin has been built, some configuration is needed to describe the route which statistics should be made available on. load_module modules/ngx_http_stats_prom_module.so; http { # Optional: set shared memory size (default 1m) # prometheus_metrics_shm_size 2m; server { # standard NGINX server directives apply here listen 9100; location /metrics { prometheus_metrics; } } } This configuration exports statistics suitable for Prometheus ingestion on http://hostname:9100/metrics. Presuming that the nginx instance is running locally, you should be able to query the metrics endpoint, via curl: curl http://localhost:9100/metrics with sample response: # HELP nginx_up Whether the nginx process is up # TYPE nginx_up gauge nginx_up 1 # HELP nginx_process_pid PID of the worker process (serving metrics) # TYPE nginx_process_pid gauge nginx_process_pid 2985885 # HELP nginx_http_requests_total Total number of HTTP requests # TYPE nginx_http_requests_total counter nginx_http_requests_total 434488 # HELP nginx_http_responses_total HTTP responses by status class # TYPE nginx_http_responses_total counter nginx_http_responses_total{status="1xx"} 0 nginx_http_responses_total{status="2xx"} 42598 nginx_http_responses_total{status="3xx"} 771 nginx_http_responses_total{status="4xx"} 77 nginx_http_responses_total{status="5xx"} 1 # etc. Exported statistics Once running, you should be able to view metrics for NGINX on the route specified in your configuration. Once started, the available metrics are: MetricTypeLabelsnginx_upgaugenginx_process_pidgaugenginx_http_requests_totalcounternginx_http_responses_totalcounterstatus (1xx–5xx)nginx_http_requests_by_method_totalcountermethod (GET, HEAD, POST, PUT, DELETE, OPTIONS, PATCH, other)nginx_http_request_bytes_totalcounternginx_http_response_bytes_totalcounternginx_http_version_totalcounterversion (1.0, 1.1, 2.0, 3.0)nginx_http_request_duration_secondshistogram(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10)nginx_connections_activegaugenginx_connections_readinggaugenginx_connections_writinggaugenginx_connections_waitinggaugenginx_connections_accepted_totalcounternginx_connections_handled_totalcounter These are available as soon as the process launches, and are measured from process start. In addition, there are basic metrics for each server or upstream directive defined in your nginx.conf file. These will appear after the first received request on that prefix. MetricTypeLabels<prefix>_requests_totalcounterserver / upstream<prefix>_response_totalcounterserver / upstream, status (1xx–5xx)<prefix>_request_bytes_totalcounterserver / upstream<prefix>_response_bytes_totalcounterserver / upstream It is worth noting that, at most, 512 server or upstream directives are supported. Prefix lengths greater than 255 characters are ignored. Finally, only 1 megabyte of memory is allocated for statistics. If this is exhausted, statistics collection will be silently skipped. This can be configured in your NGINX configuration with prometheus_metrics_shm_size. Limitations nginx-stats-prometheus is not a drop-in replacement for nginx-prometheus-exporter. The key differences are: Significantly fewer statistics are collected. The plugin is tied to the version of NGINX that it is built against. Updating NGINX typically requires that the plugin be re-built. Significantly less battle tested. Obviously our stats collection is far less mature and used. Licence & Contributing We’ve decided to release nginx-stats-prometheus as MIT licenced. We are open to pull requests, so feel free to submit one. We only ask that a human has at least reviewed the patch-set before submitting to us. We currently offer no support for nginx-stats-prometheus and are unable to guarantee help with any issues that may arise from using it. Conclusion Perhaps you want to run NGINX without containerisation, are looking to simplify your deployment’s configuration, or just prefer the idea of running statistics collection from a plugin, give nginx-stats-prometheus a try!