Checking Software Status

Monitoring the monitors

Checking Software Status

systemctl

The quick way to check the status of all of the supporting software is to run systemctl status on all of the services.

sudo systemctl status prometheus grafana-server node_exporter json_exporter eth-metrics

You will see output like the following.

Prometheus and Grafana status

node_exporter and json_exporter status

Ethereum Metrics Exporter status

If you would like to see the logs individually for the supporting software, use the following commands to see the last 100 lines of log files.

sudo journalctl -u prometheus -n 100
sudo journalctl -u grafana-server -n 100
sudo journalctl -u node_exporter -n 100
sudo journalctl -u json_exporter -n 100
sudo journalctl -u eth-metrics -n 100

Replace "100" in the commands above with any number of lines you would like to see.

To see a never-ending stream of logs, use these commands, and use CTRL-C to exit.

sudo journalctl -fu prometheus
sudo journalctl -fu grafana-server
sudo journalctl -fu node_exporter
sudo journalctl -fu json_exporter
sudo journalctl -fu eth-metrics

Healthy Startups

Here is what a healthy start looks like in the logs for the supporting software.

Logs

Prometheus

Logs for a healthy Prometheus start

Grafana

Logs for a healthy Grafana start

node_exporter

Logs for a healthy node_exporter start

json_exporter

Logs for a healthy json_exporter start

Ethereum Metrics Exporter

Logs for a healthy Ethereum Metrics Exporter start

Prometheus Targets

One way to check if your supporting software is running is to see whether Prometheus is able to scrape the software.

Go to http://127.0.0.1:9090, replacing 127.0.0.1 with the IP address at which Prometheus is running.

At the top of the screen you should see a Status menu. Hover over that menu and click on the Targets link.

Find the Targets link under the Status menu

The Targets screen will appear listing all of the Prometheus targets we previously set up.

You may need click Clear Selection in the targets filter in the upper-left corner of the screen.

Clear the selection to see all Prometheus targets

Each line of blue or red text represents a single Prometheus job serving one or more targets.

• Blue means Prometheus has successfully scraped data recently

• Red means Prometheus has not successfully scraped data recently from one

The numbers in parentheses show how many of the targets within each group Prometheus has successfully scraped recently.

The Targets screen showing the status of various Prometheus jobs

If Prometheus has not had any data from a target within the last 15 minutes, it will show that target in the color red on this screen. Two of the jobs will often show a red color on this screen:

• consensus_beacon_genesis - This job runs every few hours, because the genesis information never changes. Therefore, Prometheus often shows this job in red.

• github_latest_release - This job runs every few hours, because GitHub has a low rate limit. Therefore, Prometheus often shows this job in red. Additionally, it may take a couple hours after a new release for that release to be reflected on the Prometheus Targets screen.

The last job that may be red is blackbox_exporter. That job is not presently covered in these instructions.