# Stake Local Dashboard A multi-client Grafana dashboard for Ethereum staking ## Introduction The Stake Local Dashboard is a Grafana dashboard and software configuration that provides access to Ethereum staking data from nine different Ethereum clients. Clients may be configured in one or more staking groups that are running on one or more hosts. The dashboard relies on data collected from client metrics pages and client API calls to populate the customizable dashboard. Third-party APIs are also used to collect the current ETH price ([CoinGecko](https://www.coingecko.com/)) and the latest client version information ([GitHub](https://github.com/)). The result is a powerful tool for viewing the status of staking operations. This really should be called a dashboard library, because it is meant to be customized to meet your staking needs. The panels within may not all be relevant to your clients or your monitoring area of focus. They may not be organized according to your priorities. Therefore, remove or reorganize panels, hide columns or queries, and make the dashboard your own. The following screenshot shows a sample dashboard using a mix of panels that I selected for an at-a-glance view of testnet staking operations. This is just one way that the library of panels can be personalized to your staking operations, and the dashboard depicted below is only using one quarter of the panels available.

Sample Dashboard for Prysm/Geth on Goerli

Additional screenshots of the sample dashboard for other Ethereum clients are in the [Sample Dashboard](#sample-dashboard) section below. ## Key Features * Consensus clients: [Lighthouse](https://github.com/sigp/lighthouse), [Lodestar](https://github.com/ChainSafe/lodestar), [Nimbus](https://github.com/status-im/nimbus-eth2), [Prysm](https://github.com/prysmaticlabs/prysm), and [Teku](https://github.com/ConsenSys/teku) * Execution clients: [Besu](https://github.com/hyperledger/besu), [Erigon](https://github.com/ledgerwatch/erigon), [Geth](https://github.com/ethereum/go-ethereum), and [Nethermind](https://github.com/NethermindEth/nethermind) * 84+ primary multi-client dashboard panels with significant support across clients * 28 additional library panels with limited support across clients * Group consensus and execution clients into pairs and easily switch between groups * Displays multiple staking groups at a time * Track the status of multiple hosts and easily switch between hosts * Displays multiple hosts at a time * Displays and calculates validator and address balances and rates of change over time * Supports for multiple currencies for fiat values, via [CoinGecko](https://www.coingecko.com/). Includes: USD, EUR, JPY, GBP, AUD, CAD, CHF, CNY, HKD, NZD, ILS, KRW, MXN, RUB, SGD, TWD, and INR * Displays client current version and latest release from [GitHub](https://github.com/), with links to release notes * Links to [Beaconcha.in](https://beaconcha.in/) for slot, block, epoch and validator details * Links to [Etherscan.io](https://etherscan.io/) for Ethereum address details * Explorer links can be customized to the network served. Links for Goerli nodes will go to Goerli explorers, for example. * Provides some status of supporting software: [Grafana](https://github.com/grafana/grafana), [Prometheus](https://github.com/prometheus/prometheus), [json\_exporter](https://github.com/prometheus-community/json_exporter), [node\_exporter](https://github.com/prometheus/node_exporter), and [Ethereum Metrics Exporter](https://github.com/ethpandaops/ethereum-metrics-exporter) If you want to get started without reading more about the dashboard, begin the installation and configuration process with [Dashboard Installation](/dashboard-installation/installation-overview). {% content-ref url="/pages/g3PkY6foUD5LtLNab6bl" %} [Dashboard Installation](/dashboard-installation/installation-overview) {% endcontent-ref %} ## What is Stake Local? Stake Local is the new name under which I will be publishing my contributions to the Ethereum staking community. I wanted to distinguish these efforts independent of my username, and I wanted the new name to have an obvious connection to staking. Stake Local is not just a name, though. It is also an imperative for decentralized and independent staking. Stake Local!

## Screenshots The screenshots in the sections that follow show each primary dashboard section for each of the five different Goerli staking groups listed in the image below. Switching between groups or hosts can be done using the dashboard variable drop-downs at the top of the screen.

### Clients Section The Clients section displays panels related to the operation of the clients, including: * Sync status * Peer count * Peer count by client type * RPC received per second * Execution layer transaction pools

{% hint style="info" %} Some data is presented on multiple panels in different forms. For example, peer types are shown in the Clients section as a pie chart of current peer counts and also as as bar chart showing peer counts over time. You can choose the panels you would like to keep or move around within the dashboard. {% endhint %} ### Network Section The Network section covers the status of the Ethereum network (i.e. Mainnet, Goerli, Sepolia). This information is independent from the clients being monitored and data should be similar from any staking group on the same network. * Participation Rate * Network liveness * Validator counts * Current slot, epoch, and block numbers * Base fee per gas * Gas used * Epochs since finalization * Expected time between proposals * Head block information * Beacon reorgs

### Software Section The Software section shows software information for both the selected group and the selected host. Software is displayed with self-reported current version information, version number of the latest GitHub release, software uptime, and database size. The Latest Release link will take the user to the release notes for the latest release on GitHub.

{% hint style="info" %} Not all clients report current version information or database size. {% endhint %} ### Local Validators Section The Local Validators section is intended to show status information for locally-run validators, but it can be used just as well for monitoring any other validator on the network. Data includes: * Individual Ethereum address balance * Individual validator balance and status * Total validators balance * Total Ethereum addresses balance * Balance changes over time and annualized rates of change, for validators, addresses, and the sum of both * Ethereum price in one of over a dozen supported currencies, including: USD, EUR, JPY, GBP, AUD, CAD, CHF, CNY, HKD, NZD, ILS, KRW, MXN, RUB, SGD, TWD, and INR * Total number of monitored validators {% hint style="info" %} Prysm users have an advantage with access to proposal, attestation, and aggregation slot numbers provided directly by the client. Other consensus clients can still monitor validators through client API calls, but less detail is available. {% endhint %}

{% hint style="info" %} The only limit to the number of validators and Ethereum addresses you can monitor is the number of queries your Ethereum clients can handle. {% endhint %} {% hint style="danger" %} See [known issues](#known-issues) for a discussion about validator and address balance changes and how accurate "earnings" calculations are difficult with the tools used. {% endhint %} ### Host Section The Host section displays panels related to the system or host on which one or more of the consensus/execution groups are running. * Memory use * CPU use and temperature * Network traffic * Disk use * Open file descriptors * I/O operations per second.

{% hint style="warning" %} Not every data point/series available is represented by a panel, and some panels show the same data in different ways. Additional panels may be added in the future to cover any important gaps. {% endhint %} ## Library Sections See the [Library Sections](#library-sections) page for screenshots of additional dashboard panels with more limited client support. ## Feature Screenshots ### Links The dashboard provides links to external resources in a few sections. * The Local Validators section provides links to Beaconcha.in for validators by index and address, and slots by slot number. * The Local Validators section provides links to Etherscan.io for Ethereum addresses by public key. * The Network section provides links to Beaconcha.in for current slot, head slot, current epoch, justified epoch, and finalized epoch. * The Network section provides links to Etherscan.io for head block and finalized block. * The Software section provides links to GitHub for latest release notes for supported products.

### Multiple Group Selection Display multiple staking groups at a time by selecting more than one group in the Group variable drop-down at the top of the dashboard. Most sections will repeat for each selected staking group. The Host section repeats for each selected host. The Sample Dashboard section does not presently work with multiple selections.

The Clients Section with Multiple Groups Selected

### Currency Support The dashboard converts validator and address balances to fiat values using data provided by the [CoinGecko API](https://www.coingecko.com/api). Supported currencies include: USD, EUR, JPY, GBP, AUD, CAD, CHF, CNY, HKD, NZD, ILS, KRW, MXN, RUB, SGD, TWD, and INR. A few of the currencies are shown below.

## Sample Dashboard Users are encouraged to rearrange, remove, and modify the dashboard panels to meet their specific needs. Not all stakers use all clients and not all clients provide the same data. Stakers using only a single client pair may want to remove all of the panels not supported by their clients. Stakers focused on client internal metrics may want to remove or hide financial performance metrics to meet their needs. A Sample Dashboard is provided as an example of a reorganization and prioritization of the panels from other dashboard sections.

{% hint style="info" %} All panels in the Sample Dashboard section are copies of the panels from the primary sections and can be safely removed or modified without affecting the originals in the primary sections. Some of the panels in the Sample Dashboard have had columns from the original panels hidden in order to reduce the space required and make room for additional panels. {% endhint %} ## Supporting Software Because this dashboard was intended to be an upgrade to the dashboards I previously created, I started with the same Grafana/Prometheus/json\_exporter/node\_exporter software stack used for the original dashboards. Investigating alternative software solutions (e.g. Loki) was something for a later phase. However, I later added [Ethereum Metrics Exporter](https://github.com/ethpandaops/ethereum-metrics-exporter) to the software stack after I realized that it gave me access to some additional API calls inaccessible to json\_exporter. The diagram below shows the communication paths configured between the various software in use.

Communication Paths Between Software Supporting the Stake Local Dashboard Library

{% hint style="warning" %} Metrics for stand-alone validator clients are included for Lighthouse, Prysm, and Nimbus. Support for other stand-alone validators and remote key managers is intended for the future. {% endhint %} ## Known Issues ### Validator and Address Balances After withdrawals are enabled, calculations of accurate earnings and rates of earning over longer time periods on the dashboard are practically impossible with the software in use. Prometheus cannot distinguish validator/address deposits from withdrawals. It cannot tell the difference between a bad week of earnings or a good week of earnings where everything has been withdrawn. It only sees a net balance change. {% hint style="info" %} Prometheus is not intended for transactional type data, such as deposits and withdrawals. Using a different type of database could solve this problem, but that was out of scope for this project. {% endhint %} Instead, the dashboard displays the raw balance change and calculates annualized rates of change based on that raw balance change over a period of time. Withdrawals will skew annualized rates of change downward for as long as the withdrawal is still within the lookback period for the calculation. For example: A withdrawal made within the last hour will negatively skew the annualized rate of return for the one-hour annualized rate of change. After one hour has passed, the one-hour annualized rate of change will again reflect an accurate annualized rate of earnings. Due to the expectation that earnings will be skimmed from validators more often than once a week, anything greater than a one-day annualized rate of change will likely be inaccurate after withdrawals are enabled. ### Queries There is limited consistency to the names or meanings of client-provided metrics. Dashboard panels must query for many different metric names to cover all of the possible clients. For example, the panel that displays client version information queries Prometheus for metrics with the following names: * *lighthouse\_info* * *prysm\_version* * *beacon\_teku\_version* * *execution\_version* * *prometheus\_build\_info* * *lodestar\_version* * *blackbox\_exporter\_build\_info* * *node\_exporter\_build\_info* * *dirk\_release* * *vouch\_release* * *process\_start\_time\_seconds* * *eth\_exe\_web3\_client\_version* * *eth\_con\_node\_version* All of these metric names appear in a single query to Prometheus and Prometheus spends processing power looking for each, even if you are running a small subset of the clients. Differing units, subtle semantic differences, and special display needs may require that some clients have completely separate queries for their data. Similar to above, if the client to which that query applies is not in use, the query is still executed and some processing power is wasted. In practice, running all of the supporting software on the same system as a single monitored client has negligible impact to system resources. See the screenshots in the [Hosts section](#host-section) for examples of resource use under the Stake Local dashboard software stack. Only nuc06 is not running all supporting software at this time. ### Installation and Configuration Installation and configuration of the supporting software and dashboard involves many steps. None of steps are particularly challenging, but 5 software products must be installed and configured, and a dozen Prometheus targets must be configured and labeled to get up and running with a single consensus/execution client pair. An archive containing template Prometheus target configuration files is provided for quick installation. Reasonable default values have been provided in these files, but you may want or need to make changes to customize your installation. ## Notes {% hint style="info" %} The dashboard is not a substitute for third-party monitoring services and explorers. * Third-party monitoring services and explorers have more historical data and more advanced querying capabilities than Prometheus provides. * Third-party monitoring services and explorers are more easily accessible from anywhere. Special configurations and precautions are required to access Grafana from anywhere. * Third party monitoring services can alert you when your Internet connection is down, while Grafana would not likely have a way to communicate with you. * If your Internet connection is down, third-party monitoring services can still monitor/alert on your staking status, but will not have access to host data. * Some Prometheus metrics reset to zero after your client has restarted. This will not happen with third party explorers. Grafana does have some benefits. * Grafana dashboards are highly customizable. * Large variety of client and system performance metrics available. * Nearly all data on the dashboard is from local sources. * Grafana is still locally accessible when your Internet is down. {% endhint %} {% hint style="info" %} Though one dashboard panel may support multiple clients, the type of data displayed is not always equivalent or at the same level of detail. Examples: * The RPC Received per Second panel shows RPC calls broken out by specific statuses for Lighthouse but only simple successes and failures for Geth. * The Validator Count panel shows pending/exiting queue lengths for Prysm but only the active validator count for all other clients. {% endhint %} ### Support Stake Local stakelocal.eth ## Installation and Configuration To install the Stake Local Dashboard Library and all supporting software, visit the [Dashboard Installation](https://github.com/stakelocal/stakelocal-dashboard/blob/docs/docs/broken-reference/README.md) section. # Library Sections Panels with support from fewer clients The following additional dashboard sections include panels with support from a smaller subset of clients. These panels have been moved to separate sections named after supported clients. ## Erigon, Geth, Lighthouse

## Erigon, Geth, Lighthouse, Lodestar, Prysm

## Erigon, Geth, Nethermind

## Lighthouse

## Lighthouse, Lodestar, Teku

## Lighthouse, Nimbus

## Lighthouse, Prysm

## Lighthouse, Teku

## Lodestar

## Prysm

# Installation Overview Installation of the Stake Local Dashboard ## Installation Overview The overall process of installing and configuring the Stake Local Dashboard is as follows: * [Install Supporting Software](/dashboard-installation/install-supporting-software) - Install Prometheus, Grafana, node\_exporter, json\_exporter, and Ethereum Metrics Exporter. * [Configure Ethereum Clients](https://github.com/stakelocal/stakelocal-dashboard/blob/docs/docs/dashboard-installation/broken-reference/README.md) - Update Ethereum client configurations to allow supporting software access to metrics and API data sources. * [Configure Supporting Software](/dashboard-installation/configure-supporting-software) - Base configuration of Prometheus, node\_exporter, json\_exporter and Ethereum Metrics Exporter. * [Prometheus Jobs & Targets Configuration](/dashboard-installation/prometheus-jobs-and-targets) - Configuration of Prometheus scraping jobs, scraping targets, and processing rules. * [Optional: Alternative Dashboard Compatibility](/dashboard-installation/optional-alternative-dashboard-compatibility) - Configuration changes to ensure compatibility with prior Ethereum client dashboards. * [Enable & Start/Restart Services](/dashboard-installation/enable-and-start-restart-services) - Start/restart software to enable all configurations. * [Install Dashboard](/dashboard-installation/install-dashboard) - Install Grafana dashboard. {% hint style="info" %} You may skip the relevant [Install Supporting Software](/dashboard-installation/install-supporting-software) steps if you already have Prometheus, Grafana, node\_exporter, json\_exporter, or Ethereum Metrics Exporter installed. Configuration is handled in a separate section. {% endhint %} {% hint style="danger" %} For ease of installation, the present guide instructs you to overwrite any existing `prometheus.yml` , and `json_exporter.yml` configuration files with entirely new files. You may need to manually merge your current files with the new files in order to keep some preexisting jobs functioning. Instructions are provided to ensure that over a dozen other Ethereum client dashboards continue to work with the Stake Local `prometheus.yml` configuration file, but not all of your current Prometheus jobs may be covered by these modifications. {% endhint %} # Install Supporting Software Install the supporting software required for the Stake Local Dashboard For full functionality, the Stake Local Dashboard requires the following software packages: * [Prometheus ](https://github.com/prometheus/prometheus)- Collects data from Ethereum clients and the data exporters * [Grafana ](https://github.com/grafana/grafana)- Queries Prometheus for data and displays dashboards * [node\_exporter ](https://github.com/prometheus/node_exporter)- Exports host status and performance metrics in the Prometheus format. * [json\_exporter ](https://github.com/prometheus-community/json_exporter)- Converts JSON-formatted API responses to the Prometheus format. Used for API calls to consensus clients, execution clients, and third-party APIs. * [Ethereum Metrics Exporter](https://github.com/ethpandaops/ethereum-metrics-exporter) - Exports consensus and execution client API responses in the Prometheus format. Covers some API calls inaccessible to json\_exporter. {% hint style="warning" %} The three "exporters" in the list above are technically optional, but highly recommended. Many dashboard panels will be partially or completely disabled without one or more of these exporters. {% endhint %} {% hint style="info" %} Skip the installation process for any products already installed on your system. Configuration of the software is handled later. {% endhint %} ### Instructions for Ubuntu Server 22.04 LTS The following sections cover installation of all supporting software on a clean Ubuntu Server 22.04 LTS. You may need to adapt these instructions to your specific operating system. {% content-ref url="/pages/sINVEe2p0dWTwrvEwDyi" %} [Install Prerequisites](/dashboard-installation/install-supporting-software/install-prerequisites) {% endcontent-ref %} {% content-ref url="/pages/4mYW3DwmiuodrsY21GWz" %} [Install Prometheus](/dashboard-installation/install-supporting-software/install-prometheus) {% endcontent-ref %} {% content-ref url="/pages/Hkbbq2Rc2b9ryAXUrnoo" %} [Install Grafana](/dashboard-installation/install-supporting-software/install-grafana) {% endcontent-ref %} {% content-ref url="/pages/ORAPGZyKfCMkAkupbh6N" %} [Install node\_exporter](/dashboard-installation/install-supporting-software/install-node_exporter) {% endcontent-ref %} {% content-ref url="/pages/88FpqdXzATjRxYSNICEw" %} [Install json\_exporter](/dashboard-installation/install-supporting-software/install-json_exporter) {% endcontent-ref %} {% content-ref url="/pages/aep0kNmU5k9vYheQ7G5L" %} [Install Ethereum Metrics Exporter](/dashboard-installation/install-supporting-software/install-ethereum-metrics-exporter) {% endcontent-ref %} # Install Prerequisites Supporting software for supporting software ## Install Prerequisites The following tools may not yet be available on your system, but are necessary for installing our supporting software. ## make `make` is a utility to assist in building software. ```bash sudo apt-get update sudo apt-get install make ```

Results of `apt-get install make` command

Check to see if `make` was correctly installed. ```bash make --version ```

### Go Go is a programming language and the `go` compiler is needed to build software written in Go. The instructions below are for version 1.20. [Check for newer versions of go](https://go.dev/dl/), and adapt the instructions below. ```bash cd wget https://go.dev/dl/go1.20.linux-amd64.tar.gz sudo tar -C /usr/local -xzf go1.20.linux-amd64.tar.gz sudo ln -s /usr/local/go/bin/go /usr/bin/go ```

Results of the `wget`, `tar`, and `ln` commands

Check to see if `go` was installed correctly. ```bash go version ```

Optionally, clean up installation files. ```bash rm go1.20.linux-amd64.tar.gz ``` ### Stake Local Dashboard Repository Clone the Stake Local Dashboard repository. ``` mkdir ~/git cd ~/git git clone https://github.com/stakelocal/stakelocal-dashboard.git ``` # Install Prometheus Install Prometheus to collect and serve data ## Prometheus Prometheus handles data collection, storage, and queries. Data sources are configured as targets from which Prometheus will request metrics data. Prometheus stores this data and responds to Grafana queries against the data. ### Create System Account Create a system account under which Prometheus can run.

sudo adduser --system prometheus --group --no-create-home

### Install Prometheus The instructions below are for version 2.42.0. [Check for newer versions of Prometheus](https://github.com/prometheus/prometheus/releases), and adapt the instructions below. Download the latest release. ```bash cd wget https://github.com/prometheus/prometheus/releases/download/v2.42.0/prometheus-2.42.0.linux-amd64.tar.gz ```

Unpack the release. ```bash tar xzvf prometheus-2.42.0.linux-amd64.tar.gz ```

Copy multiple executable files to `/usr/local/bin`. ```bash cd prometheus-2.42.0.linux-amd64 sudo cp -t /usr/local/bin/ prometheus promtool ``` Create additional required directories. ```bash sudo mkdir -p /etc/prometheus/console_libraries \ /etc/prometheus/consoles \ /etc/prometheus/files_sd \ /etc/prometheus/rules \ /etc/prometheus/rules.d \ /var/lib/prometheus ``` Change ownership on the `/var/lib/prometheus` directory so the prometheus service can access it. ``` sudo chown prometheus:prometheus /var/lib/prometheus ``` ### Configure Startup Create a systemd service file to configure automatic startup of the Prometheus service. ```bash sudo nano /etc/systemd/system/prometheus.service ``` Insert the following into the Prometheus systemd service file. ``` [Unit] Description=Prometheus Wants=network-online.target After=network-online.target [Service] User=prometheus Group=prometheus Type=simple Restart=always RestartSec=5 ExecStart=/usr/local/bin/prometheus \ --config.file /etc/prometheus/prometheus.yml \ --storage.tsdb.path /var/lib/prometheus/ \ --web.console.templates=/etc/prometheus/consoles \ --web.console.libraries=/etc/prometheus/console_libraries \ --storage.tsdb.retention.time=32d ExecReload=kill -HUP $MAINPID [Install] WantedBy=multi-user.target ``` {% hint style="info" %} Data retention is set to 32 days in the default configuration. The number of days can be increased by changing the value assigned to the `storage.tsdb.retention.time` flag in the last line of the `ExecStart` configuration above. {% endhint %} Save the file and exit the editor. ### Optional: Clean Up Installation Files Remove the Prometheus archive and installation directory. ```bash cd rm prometheus-2.42.0.linux-amd64.tar.gz rm -rf prometheus-2.42.0.linux-amd64 ``` ### Notes {% hint style="info" %} Prometheus will listen on port 9090. This is the port for the Prometheus admin UI. {% endhint %} {% hint style="warning" %} We have not yet configured nor started Prometheus. {% endhint %} # Install Grafana Install Grafana to serve dashboards ## Grafana Grafana provides all dashboard functionality. Dashboards are displayed using Grafana's user interface, and query for data from Prometheus. ### Install Supporting Software Install the supporting software we may need. ```bash sudo apt-get update sudo apt-get install apt-transport-https sudo apt-get install software-properties-common wget ``` ### Install Grafana Install the key for authenticating Grafana packages. ```bash wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add - ``` This command may generate a warning about `apt-key` being deprecated. That is not a problem.

Add the Grafana repository to the list of external repositories and update the package list. ```bash sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main" ``` The `add-apt-repository` command may also generate a warning about `apt-key` being deprecated.

Update the package list and install Grafana. ```bash sudo apt-get update sudo apt-get install grafana ```

Results of `sudo apt-get install grafana` command

### Notes {% hint style="info" %} Grafana will listen on port 3000. This is the port for the Grafana user interface. {% endhint %} {% hint style="warning" %} We have not yet configured nor started Grafana. {% endhint %} # Install node\_exporter Install node\_exporter for system status and performance data ## node\_exporter node\_exporter exports metrics related to system status and performance. ### Create System Account Create a system account under which node\_exporter can run. ``` sudo adduser --system node_exporter --group --no-create-home ```

### Install node\_exporter The instructions below are for version 1.5.0. [Check for newer versions of node\_exporter](https://github.com/prometheus/node_exporter/releases), and adapt the instructions below. Download the latest release. ```bash cd wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz ```

Unpack the release. ```bash tar xzvf node_exporter-1.5.0.linux-amd64.tar.gz ```

Copy the executable file to `/usr/local/bin`. ```bash sudo cp node_exporter-1.5.0.linux-amd64/node_exporter /usr/local/bin/ ``` ### Configure Startup Create a systemd service file to configure automatic startup of the node\_exporter service. ```bash sudo nano /etc/systemd/system/node_exporter.service ``` Insert the following into the node\_exporter systemd service file.

[Unit]
Description=Node Exporter

[Service]
Type=simple
Restart=always
RestartSec=5
User=node_exporter
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Save the file and exit the editor. ### Optional: Clean Up Installation Files Remove the node\_exporter archive and installation directory. ```bash cd rm node_exporter-1.5.0.linux-amd64.tar.gz rm -rf node_exporter-1.5.0.linux-amd64 ``` ### Notes {% hint style="info" %} node\_exporter will listen on port 9100. This is the port from which Prometheus will query for metrics. {% endhint %} {% hint style="warning" %} We have not yet configured nor started node\_exporter. {% endhint %} # Install json\_exporter Install json\_exporter for access to local and third-party API data ## json\_exporter json\_exporter queries endpoints providing JSON-formatted data and converts that data into the Prometheus metrics format. It is used to query consensus and execution client JSON APIs, and to query external APIs for price and software release data. ### Create System Account Create a user account under which json\_exporter can run. ```bash sudo adduser --system json_exporter --group --no-create-home ```

### Build the json\_exporter Executable The instructions below are for version 0.5.0. [Check for newer versions of json\_exporter](https://github.com/prometheus-community/json_exporter/releases), and adapt the instructions below. Clone the latest release. ```bash mkdir ~/git cd ~/git git clone --depth 1 --branch v0.5.0 https://github.com/prometheus-community/json_exporter.git ```

Build json\_exporter from source code. ```bash cd json_exporter make build ```

### Install json\_exporter Copy the json\_exporter executable to the `/usr/local/bin` directory. ```bash sudo cp json_exporter /usr/local/bin/ ``` Check to see if `json_exporter` was installed correctly. ```bash /usr/local/bin/json_exporter --version --help ```

Create a directory for the configuration file. ```bash sudo mkdir /etc/json_exporter ``` ### Configure Startup Create a systemd service file to configure automatic startup of the json\_exporter service. ```bash sudo nano /etc/systemd/system/json_exporter.service ``` Insert the following into the json\_exporter systemd service file. ``` [Unit] Description=JSON Exporter [Service] Type=simple Restart=always RestartSec=5 User=json_exporter ExecStart=/usr/local/bin/json_exporter --config.file /etc/json_exporter/json_exporter.yml [Install] WantedBy=multi-user.target ``` Save the file and exit the editor. ### Optional: Clean Up Installation Files Remove the json\_exporter git repository. ```bash cd rm -rf ~/git/json_exporter ``` ### Notes {% hint style="info" %} json\_exporter will listen on port 7979. This is the port from which Prometheus will query for metrics. {% endhint %} {% hint style="warning" %} We have not yet configured nor started json\_exporter. {% endhint %} # Install Ethereum Metrics Exporter Install Ethereum Metrics Exporter for access to additional API data ## Ethereum Metrics Exporter Ethereum Metrics Exporter exports queries consensus and execution client APIs and exports that data in the Prometheus metrics format. There is some overlap between the client API endpoints accessed by Ethereum Metrics Exporter and json\_exporter, but both exporters offer access to data in unique ways. {% hint style="warning" %} A single instance of Ethereum Metrics Exporter can be configured to query a single consensus/execution client group. Multiple instances of Ethereum Metrics Exporter are required to monitor multiple consensus/execution groups. {% endhint %} ### Create System Account Create a user account under which Ethereum Metrics Exporter can run. ```bash sudo adduser --system eth-metrics --group --no-create-home ```

### Install Ethereum Metrics Exporter Download and build the Ethereum Metrics Exporter executable. ```bash go install github.com/ethpandaops/ethereum-metrics-exporter@latest ```

Copy the executable to the `/usr/local/bin` directory. ```bash sudo cp ~/go/bin/ethereum-metrics-exporter /usr/local/bin ``` ### Create Supporting Directories Create additional required directories. ```bash sudo mkdir -p /var/local/lib/eth-metrics /etc/eth-metrics ``` ### Configure Startup Create a systemd service file to configure automatic startup of the Ethereum Metrics Exporter service for a single consensus/execution client pair. ```bash sudo nano /etc/systemd/system/eth-metrics.service ``` Insert the following into the Ethereum Metrics Exporter systemd service file. ``` [Unit] Description=Ethereum Metrics Exporter Wants=network-online.target After=network-online.target [Service] User=eth-metrics Group=eth-metrics Type=simple Restart=always RestartSec=5 ExecStart=/usr/local/bin/ethereum-metrics-exporter \ --config /etc/eth-metrics/eth-metrics.yml \ --metrics-port 9095 [Install] WantedBy=multi-user.target ``` {% hint style="warning" %} The default listening port for the Ethereum Metrics Exporter is 9090, which also happens to be the default listening port for Prometheus. In the service file above, the Ethereum Metrics Exporter port is re-assigned to 9095 through the `metrics port` flag. {% endhint %} {% hint style="info" %} Each instance of Ethereum Metrics Exporter can only monitor a single consensus/execution client pair. If you need to monitor multiple consensus/execution client pairs using Ethereum Metrics Exporter, there are two options: 1. If each client pair is running on a separate host, it is easiest to follow the instructions above to install Ethereum Metrics Exporter on each host and configure it to query the local clients. 2. If multiple client pairs are running on the same host, or if you want to run all monitoring software from the same host, you will need to configure multiple instances of Ethereum Metrics Exporter on the same system. For each new client pair, copy the eth-metrics.service file above to a new unique name, and modify the configuration file and port specified in the `ExecStart` line to something unique across instances. {% endhint %} ### Notes {% hint style="info" %} Ethereum Metrics Exporter will listen on port 9095. This is the port from which Prometheus will query for metrics. {% endhint %} {% hint style="warning" %} We have not yet configured nor started Ethereum Metrics Exporter. {% endhint %} # Configure Ethereum Clients Configuration consensus and execution clients to serve metrics and API data Each consensus and execution client must have the appropriate metrics services and APIs enabled and accessible to the supporting software. Find your clients below and ensure that the required command-line flags/configuration values are present in your startup scripts, configuration files, or service files. {% hint style="info" %} Refer to the guides you originally used to set up your clients for help determining where the following options should be set. {% endhint %} ## Consensus Clients {% tabs %} {% tab title="Lighthouse" %} In addition to other command-line flags, the following Lighthouse flags may be required to support the dashboard. **Required** * `--metrics` - Enables the Prometheus metrics service **Optional** * `--metrics-address 0.0.0.0` - Authorizes Lighthouse to answer metrics requests from other IP addresses. This is only required if you want to run Prometheus on a separate system {% hint style="info" %} The default metrics port for Lighthouse is 5054. The default REST API port for Lighthouse is 5052. {% endhint %} {% hint style="warning" %} If you would like other IP addresses to have access to Lighthouse's metrics, do not expose Lighthouse's metrics port to the Internet. Keep the metrics port closed at your router. {% endhint %} {% endtab %} {% tab title="Lodestar" %} In addition to other command-line flags, the following flags are required to support the dashboard. **Required** * `--metrics true` - Enables the Prometheus metrics service * `--rest` - Enables the HTTP API **Optional** * `--metrics.address 0.0.0.0` - Authorizes Lodestar to answer metrics requests from other IP addresses. Only required if you want to run Prometheus on a separate system * `--rest.address '0.0.0.0'` - Authorizes Lodestar to answer HTTP API requests from other IP addresses. Only required if you want to run Prometheus on a separate system {% hint style="info" %} The default metrics port for Lodestar is 8008. The default REST API port for Lodestar is 9596. {% endhint %} {% hint style="warning" %} If you would like other IP addresses to have access to Lodestar's metrics, do not expose Lodestar's port to the Internet. Keep the metrics port closed at your router. {% endhint %} {% endtab %} {% tab title="Nimbus" %} In addition to other command-line flags, the following Nimbus flags may be required to support the dashboard. **Required** * `--metrics` - Enables the Prometheus metrics service * `--rest` - Enables the REST API **Optional** * `--metrics-address 0.0.0.0` - Authorizes Lighthouse to answer metrics requests from other IP addresses. Only required if you want to run Prometheus on a separate system. * `--rest-address 0.0.0.0` - Authorizes Lighthouse to answer API requests from other IP addresses. Only required if you want to run Prometheus on a separate system. {% hint style="info" %} The default metrics port for Nimbus is 8008. The default REST API port for Nimbus is 5052. {% endhint %} {% hint style="warning" %} If you would like other IP addresses to have access to Nimbus' metrics, do not expose Nimbus' port to the Internet. Keep the metrics port closed at your router. {% endhint %} {% endtab %} {% tab title="Prysm" %} In addition to other command-line flags, the following flags are required to support the dashboard. **Required** * `--monitoring-host 0.0.0.0` - Authorizes Lighthouse to answer metrics requests from other IP addresses. Only required if you want to run Prometheus on a separate system. **Optional** * `--grpc-gateway-host '0.0.0.0'` - Authorizes Prysm to answer API requests from other IP addresses. Only required if you want to run Ethereum Metrics Exporter or json\_exporter on systems separate from Prysm. {% hint style="info" %} The default metrics port for the Prysm beacon chain is 8080. The default metrics port for the Prysm validator is 8081. The default REST API port for the Prysm beacon chain is 3500. {% endhint %} {% hint style="warning" %} If you would like other IP addresses to have access to Prysm's metrics, do not expose Prysm's port to the Internet. Keep the metrics port closed at your router {% endhint %} {% endtab %} {% tab title="Teku" %} In addition to other command-line flags, the following Teku flags may be required to support the dashboard. **Required** * `--metrics-enabled` - Enables the Prometheus metrics service * `--rest-api-enabled` - Enables the REST API * `--metrics-categories=JVM,PROCESS,BEACON,DISCOVERY,EVENTBUS,EXECUTOR,LIBP2P,NETWORK,STORAGE,STORAGE_HOT_DB,STORAGE_FINALIZED_DB,REMOTE_VALIDATOR,VALIDATOR,VALIDATOR_PERFORMANCE` - Turns on all metrics **Optional** * `--metrics-interface 0.0.0.0` - Authorizes Teku to answer metrics requests from other IP addresses. Only required if you want to run Prometheus on a separate system. * `--rest-api-interface 0.0.0.0` - Authorizes Teku to answer API requests from other IP addresses. Only required if you want to run Prometheus on a separate system. * `--rest-api-host-allowlist 0.0.0.0` - Authorizes Teku to answer API requests from other IP addresses. Only required if you want to run Prometheus on a separate system. * `--metrics-host-allowlist 0.0.0.0` - Authorizes Teku to answer metrics requests from other IP addresses. Only required if you want to run Prometheus on a separate system. {% hint style="info" %} The default metrics port for Teku is 8008. The default REST API port for Teku is 5051. {% endhint %} {% hint style="warning" %} If you would like other IP addresses to have access to Teku's metrics, do not expose Teku's port to the Internet. Keep the metrics port closed at your router {% endhint %} {% endtab %} {% endtabs %} ## Execution Clients {% tabs %} {% tab title="Besu" %} In addition to other command-line flags, the following Besu flags may be required to support the dashboard. **Required** * `--metrics-enabled` - Enables the Prometheus metrics service * `--rpc-http-enabled` - Enables the HTTP API * `--rpc-http-api=ETH,NET,WEB3,TXPOOL` - These API namespaces are required, in addition to any others you already use. * `--metrics-protocol=PROMETHEUS` - Configures Besu to return metrics in Prometheus format * `--metrics-category=BLOCKCHAIN,ETHEREUM,EXECUTORS,NETWORK,PEERS,PERMISSIONING,PRUNER,RPC,SYNCHRONIZER,TRANSACTION_POOL,STRATUM,JVM,PROCESS` - Turns on all metrics **Optional** * `--metrics-host=0.0.0.0` - Authorizes Besu to answer metrics requests from other IP addresses. Only required if you want to run Prometheus on a separate system. * `--rpc-http-host=0.0.0.0` - Authorizes Besu to answer API requests from other IP addresses. Only required if you want to run Prometheus on a separate system. * `--host-allowlist="*"` - TBD {% hint style="info" %} The default metrics port for Besu is 9545. The default RPC port for Besu is 8545. {% endhint %} {% hint style="warning" %} If you would like other IP addresses to have access to Besu's metrics, do not expose Besu's port to the Internet. Keep the metrics port closed at your router. {% endhint %} {% endtab %} {% tab title="Erigon" %} In addition to other command-line flags, the following Erigon flags may be required to support the dashboard. **Required** * `--metrics` - Enables the Prometheus metrics service * `--http` - Enables the HTTP API * `--http.api=eth,net,txpool,web3` - These API namespaces are required, in addition to any others you already use. * `--private.api.addr=0.0.0.0:9091` - Required to change the private API port to from its default of 9090 to 9091. It conflicts with Prometheus' use of port 9090. **Optional** * `--metrics.addr 0.0.0.0` - Authorizes Erigon to answer metrics requests from other IP addresses. Only required if you want to run Prometheus on a separate system. * `--http.addr 0.0.0.0` - Authorizes Erigon to answer API requests from other IP addresses. Only required if you want to run Prometheus on a separate system. {% hint style="info" %} The default metrics port for Erigon is 6060. The default RPC port for Erigon is 8545. {% endhint %} {% hint style="warning" %} If you would like other IP addresses to have access to Erigon's metrics, do not expose Erigon's port to the Internet. Keep the metrics port closed at your router. {% endhint %} {% endtab %} {% tab title="Geth" %} In addition to other command-line flags, the following Geth flags may be required to support the dashboard. **Required** * `--metrics` - Enables the Prometheus metrics service * `--http` - Enables the HTTP API * `--http.api net,web3,eth,txpool` - These API namespaces are required, in addition to any others you already use. **Optional** * `--metrics.addr=0.0.0.0` - Authorizes Geth to answer metrics requests from other IP addresses. Only required if you want to run Prometheus on a separate system. * `--http.addr=0.0.0.0` - Authorizes Geth to answer API requests from other IP addresses. Only required if you want to run Prometheus on a separate system. {% hint style="info" %} The default metric port for Geth is 6060. The default RPC port for Geth is 8545. {% endhint %} {% hint style="warning" %} If you would like other IP addresses to have access to Geth's metrics, do not expose Geth's port to the Internet. Keep the metrics port closed at your router. {% endhint %} {% endtab %} {% tab title="Nethermind" %} In addition to other command-line flags, the following Nethermind flags may be required to support the dashboard. **Required** * `--Metrics.Enabled true` - Enables the Prometheus metrics service * `--Metrics.ExposePort 8080` - Required to open the metrics port **Optional** * `--JsonRpc.Host 0.0.0.0` - Authorizes Nethermind to answer API requests from other IP addresses. Only required if you want to run Prometheus on a separate system. {% hint style="info" %} The default metrics port for Nethermind is 8080. The default RPC port for Nethermind is 8545. {% endhint %} {% hint style="warning" %} If you would like other IP addresses to have access to Nethermind's metrics, do not expose Nethermind's port to the Internet. Keep the metrics port closed at your router. {% endhint %} {% endtab %} {% endtabs %} {% hint style="warning" %} Restart all clients for which the configuration changed. Refer to the guides you originally used to set up your clients for help determining how to restart the clients and whether any configuration changes have taken effect. {% endhint %} # Configure Supporting Software System independent software configurations With all supporting software installed, we can begin configuring the software. The instructions in this section will install configurations that are not specific to the clients you are monitoring nor the systems on which they run. 1. [Configure json\_exporter](/dashboard-installation/configure-supporting-software/configure-json_exporter) - Add a configuration file that defines how to call various APIs and translate the responses into Prometheus metrics. 2. [Configure Prometheus Rules](/dashboard-installation/configure-supporting-software/configure-prometheus-rules) - Add a configuration file that defines queries for Prometheus to pre-process for efficiency. 3. [Configure Ethereum Metrics Exporter](/dashboard-installation/configure-supporting-software/configure-ethereum-metrics-exporter) - Add a configuration file that defines the consensus and execution clients to which Ethereum Metrics Exporter will connect. # Configure json\_exporter Configure how json\_exporter communicates with APIs ## Configure json\_exporter {% hint style="danger" %} The instructions below will overwrite your current `json_exporter.yml` file with a version that supports the Stake Local Dashboard. A backup will be made of your existing file, but it is up to you to merge the two files, if necessary. {% endhint %} {% hint style="warning" %} This configuration requires json\_exporter v0.5.0 or later. {% endhint %} Back up your current `json_exporter.yml` file to your home directory, if you have one. ```bash cp -v --backup=numbered /etc/json_exporter/json_exporter.yml ~/ ```

### Install Rules from Repository Copy the `json_exporter.yml` rules configuration file from the Stake Local Dashboard repository to `/etc/json_exporter/`. ```bash cd ~/git/stakelocal-dashboard/config/json_exporter sudo cp json_exporter.yml /etc/json_exporter ``` # Configure Prometheus Rules Configure how Prometheus preprocesses data for efficiency ## Configuring Prometheus Rules Prometheus rules improve query response times by precomputing expensive query expressions. ### Install Rules from Repository Copy the `stakelocal_rules.yml` rules configuration file from the Stake Local Dashboard repository to `/etc/prometheus/rules.d/`. ```bash cd ~/git/stakelocal-dashboard/config/prometheus/rules.d sudo cp stakelocal_rules.yml /etc/prometheus/rules.d ``` # Configure Ethereum Metrics Exporter Configure the consensus and execution clients to which Ethereum Metrics Exporter will listen ## Configure Ethereum Metrics Exporter Edit the `eth-metrics.yml` file. ```bash sudo nano /etc/eth-metrics/eth-metrics.yml ``` Insert the following text into the `eth-metrics.yml` file. ```yaml consensus: enabled: true url: "" name: "" execution: enabled: true url: "" name: "" modules: - "eth" - "net" - "web3" - "txpool" pair: enabled: true ``` Make the following customizations to the text inserted: 1. Replace `` with the URL to your consensus client API. It should look similar to `http://localhost:3500/`. You can replace `localhost` with the IP address of your consensus client, if it is not running on the local host. Use the correct port for your consensus client. \ **Default Consensus Client API Ports** 1. Lighthouse: 5052 2. Lodestar: 9596 3. Nimbus: 5052 4. Prysm: 3500 5. Teku: 5051 2. Replace `` with one of the following execution client names: `Lighthouse Beacon`, `Lodestar Beacon`, `Nimbus`, `Prysm Beacon`, `Teku`. 3. Replace `` with the URL to your execution client API. It should look similar to `http://localhost:8545`. You can replace `localhost` with the IP address of your execution client, if it is not running on the local host. All execution clients use the same port. 4. Replace `` with one of the following execution client names: `Besu`, `Erigon`, `Geth`, `Nethermind`. When you are done making changes, you final `eth-metrics.yml` file might look something like the following. ```yaml consensus: enabled: true url: "http://127.0.0.1:3500/" name: "Prysm Beacon" execution: enabled: true url: "http://127.0.0.1:8545/" name: "Geth" modules: - "eth" - "net" - "web3" - "txpool" pair: enabled: true ``` Save the file and exit the editor. # Prometheus Jobs & Targets Configure the client data sources from which Prometheus will scrape data ## Prometheus Jobs & Targets A Prometheus `job` represents a set of rules for processing client metrics from similar data sources/targets. A Prometheus `target` is a single data source from which Prometheus should collect and process data using the rules of a specific job. Prometheus jobs are collections of targets for which the same rules are applied. The `consensus_node_version` job is one example. This job is configured to build a URL to a consensus API endpoint and pass that URL to json\_exporter to parse the results. This job may be configured for multiple consensus client targets, allowing monitoring of Lighthouse, Lodestar, Nimbus, Prysm, and Teku with the same rules simultaneously. In this section we will set up the Prometheus jobs required for the dashboard. Then we will move on to configuring the various Prometheus targets your specific Ethereum clients. # Prometheus Jobs Configure the Prometheus jobs to which targets will be assigned ## Prometheus Jobs ### Configure prometheus.yml {% hint style="danger" %} The instructions below will overwrite your current `prometheus.yml` file with a version that supports the Stake Local Dashboard. A backup will be made of your existing file, but it is up to you to merge the two files, if necessary. Many existing Ethereum client dashboards will work with this `prometheus.yml` file as-is or with minimal changes. See [Optional: Alternative Dashboard Compatibility](/dashboard-installation/optional-alternative-dashboard-compatibility) for a guide to maintaining compatibility with other dashboards. {% endhint %} Back up your current `prometheus.yml` file to your home directory, if you have one. ```bash cp -v --backup=numbered /etc/prometheus/prometheus.yml ~/ ```

Copy the `prometheus.yml` rules configuration file from the Stake Local Dashboard repository to `/etc/prometheus/`. ```bash cd ~/git/stakelocal-dashboard/config/prometheus/ sudo cp prometheus.yml /etc/prometheus ``` ### Check Configuration Check the configuration with `promtool`. ```bash sudo promtool check config /etc/prometheus/prometheus.yml ``` {% hint style="info" %} You will get many warnings about files that do not exist. These are the target files that have not yet been configured. This is not a problem. {% endhint %} Look for "SUCCESS" messages similar to the following for both the configuration file and the rules file. ``` SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax SUCCESS: 46 rules found ```

If you see the output above, then you have successfully configured the Prometheus jobs. # About Stake Local Labels How Stake Local uses Prometheus Labels ## About Stake Local Labels In Prometheus, `labels` are used to tag data as it is collected. Labels can be defined by the data source, by Prometheus directly, by the Prometheus target configuration, and by the Prometheus job configuration. The Stake Local Dashboard uses labels extensively for categorization, filtering, configuration, and interoperability between data source. These labels are defined in the Prometheus target configuration files configured in the next few sections. As you configure the Prometheus targets, update the labels as needed for your staking architecture. {% hint style="warning" %} Nearly all of the labels required by the dashboard have reasonable default values that will work as-is for monitoring a mainnet consensus/execution group. You can leave the default values or update them later to customize your dashboard. **Note:** Changing the labels at a later time may cause the colors on some dashboard panels to change at the point in time where the label values changed. {% endhint %} ### network The `network` label is used to tag the Ethereum network that applies to the data source. These labels are used to display the network name in some dashboard panels, and can be used for creating new panels that compare different networks. Example values: * Mainnet * Goerli * Sepolia Default value: Mainnet {% hint style="info" %} Some of the Prometheus targets will not be associated with a specific network. For example, the GitHub target is not specific to a particular Ethereum network. {% endhint %} ### host The `host` label is used to tag the host/system on which clients/services are running. This is primarily used for filtering node\_exporter data to specific hosts, but it is also used to compare metrics across all software running on the same host. Example values: * nuc01 * Prysm/Geth Mainnet System * sir-stake-a-lot Default value: Default Host ### service The `service` label is used to define the display name of the service being run. This is the name of the service as it will show up on the dashboard panels. A unique name should be given to each client instance to ensure that clients can be distinguished from one another, particularly if you are running multiple instances of a client. Consensus clients with separate beacon and validator metrics should be distinguished in their service label. Example values: * Prysm Beacon * Teku * Nimbus * Lighthouse Beacon * Lodestar Beacon * Geth * Erigon * Nethermind * Besu * Prysm Validator * Lighthouse Validator * Prysm Mainnet * Lighthouse Goerli * Geth Sepolia * Internet Testnet 02 Default value: Client specific ### client The `client` label is used to match GitHub release information to the current running client. Values appear similar to the default service label values, these values should not be modified. Example values: * Prysm * Geth * Lighthouse * Lodestar * Erigon * Besu * Teku * Nimbus * Nethermind Default value: Client specific {% hint style="warning" %} Do not modify the client labels from their default values. This may negatively affect GitHub integration. {% endhint %} ### group The `group` label is primarily used to name consensus/execution client pairs and their supporting software. This allows the dashboard to display results for only those services supporting a single client pair. To work correctly, the group label must be spelled identically for all software in the same group. Example values: * Prysm/Geth Mainnet * Teku/Besu Goerli * Mainnet Default value: Default Group {% hint style="info" %} Some of the Prometheus targets will not be associated with a specific group. For example, the GitHub target is not specific to a particular consensus/execution client group. {% endhint %} ### explorer The `explorer` label configures the domain name of the third-party consensus or execution layer explorer for external links in dashboard panels. The explorer label should be customized for the Ethereum network and layer (consensus/execution) of the data source. {% hint style="info" %} By default, consensus clients are configured to go to `beaconcha.in` and execution clients are configured to go to `etherscan.io`. Other explorers can be used if their URL paths are the same as those for the current explorers. {% endhint %} Example values: * beaconcha.in * etherscan.io * goerli.beaconcha.in * goerli.etherscan.io * sepolia.beaconcha.in * sepolia.etherscan.io Default values: beaconcha.in or etherscan.io ### json\_exporter The `json_exporter` label is used to define the instance of json\_exporter that will be handling API requests for the Prometheus target. {% hint style="info" %} Keep the default value unless you are running json\_exporter on a different host than Prometheus. {% endhint %} Example values: * 127.0.0.1:7979 * 192.168.0.10:7979 * localhost:7979 Default value: 127.0.0.1:7979 ### instance The `instance` label defines the address and port of the client API from which Ethereum address and validator statuses/balances are collected. Ethereum address Prometheus targets must have the address and port of an execution client defined, while validator Prometheus targets must have the address and port of a consensus client defined. Example values: * 127.0.0.1:5052 * localhost:5051 * 192.168.0.10:8545 Execution client default value: 127.0.0.1:8545 {% hint style="warning" %} A port is not provided for the consensus client default value, because consensus clients do not all use the same port for their API. This value must be updated during the configuration process. {% endhint %} # Base Targets Installation Install default Prometheus targets that we can edit later Download and unpack the archive of default target configuration files. This will install configuration files for all supported clients. We can [clean up the unused target](/dashboard-installation/prometheus-jobs-and-targets/clean-up-unused-targets) files later. ```bash cd ~/git/stakelocal-dashboard sudo tar xzvf stakelocal_prometheus_targets.tgz -C /etc/prometheus/files_sd ```

Clean up the archive file now that we have unpacked it. ```bash rm stakelocal_prometheus_targets.tgz ``` Most of the targets unpacked will have default IP addresses, ports, and label values suitable to get the dashboard up and running quickly without modification. Specifically, the default target files assume the following. * All clients and supporting software are accessible at 127.0.0.1 from any other client. * All clients and supporting software are using the ports defined in this guide, nearly all of which are the client default ports. And the default labels are assigned as follows. * Where used, the `network` label is set to "Mainnet". * Where used, the `host` label is set to "Default Host". * Where used, the `group` label is set to "Default Group". * Where used, the `explorer` label is set to "beaconcha.in" or "etherscan.io". If these assumptions and defaults are acceptable, you can jump ahead to [Ethereum Address Targets](/dashboard-installation/prometheus-jobs-and-targets/ethereum-address-targets) and continue from there. Changes can always be made later, and Prometheus will automatically reload these configuration files without running any restart/reload commands. {% hint style="info" %} You may want to review and update the label values now. Label values that are changed in the future may cause historical data to appear as a different data series with separate colors/legends for as long as the historical data is within view. {% endhint %} If your clients or supporting software are not accessible at `127.0.0.1` or are running on non-standard ports, you must continue to the next sections. The following sections will guide you through a review of the Prometheus targets we just unpacked. Making changes as necessary to IP addresses, ports, or labels. # Consensus Client Targets Customize the default consensus client targets ## Consensus Client Targets Consensus client data is collected by Prometheus from three different sources: 1. Directly from the client metrics page 2. Indirectly from the client API via json\_exporter 3. Indirectly from the client API via Ethereum Metrics Exporter We will review the configuration files for the first two here. Ethereum Metrics Exporter is covered in a later section. {% hint style="info" %} If you would like to monitor more than one instance of the same client, please see the [Additional Modifications](/additional-modifications) section for details. {% endhint %} ### Consensus Client Metrics Configuration To configure the Prometheus target for client metrics, follow the instructions for your client below. {% tabs %} {% tab title="Lighthouse" %} Edit the Lighthouse configuration file in the `/etc/prometheus/files_sd/stakelocal/consensus_metrics` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/consensus_metrics/lighthouse.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:5054'] labels: network: 'Mainnet' host: 'Default Host' service: 'Lighthouse Beacon' group: 'Default Group' explorer: 'beaconcha.in' ``` If this instance of Lighthouse is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% tab title="Lodestar" %} Edit the Lodestar configuration file in the `/etc/prometheus/files_sd/stakelocal/consensus_metrics` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/consensus_metrics/lodestar.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:8008'] labels: network: 'Mainnet' host: 'Default Host' service: 'Lodestar Beacon' group: 'Default Group' explorer: 'beaconcha.in' ``` If this instance of Lodestar is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% tab title="Nimbus" %} Edit the Nimbus configuration file in the `/etc/prometheus/files_sd/stakelocal/consensus_metrics` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/consensus_metrics/nimbus.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:8008'] labels: network: 'Mainnet' host: 'Default Host' service: 'Nimbus' group: 'Default Group' explorer: 'beaconcha.in' ``` If this instance of Nimbus is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% tab title="Prysm" %} Edit the Prysm configuration file in the `/etc/prometheus/files_sd/stakelocal/consensus_metrics` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/consensus_metrics/prysm.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:8080'] labels: network: 'Mainnet' host: 'Default Host' service: 'Prysm Beacon' group: 'Default Group' explorer: 'beaconcha.in' ``` If this instance of Prysm is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% tab title="Teku" %} Edit the Teku configuration file in the `/etc/prometheus/files_sd/stakelocal/consensus_metrics` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/consensus_metrics/teku.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:8008'] labels: network: 'Mainnet' host: 'Default Host' service: 'Teku' group: 'Default Group' explorer: 'beaconcha.in' ``` If this instance of Teku is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% endtabs %} An example consensus client metrics target file for the Goerli network could look like the following. ``` - targets: ['192.168.0.42:8080'] labels: network: 'Goerli' host: 'server12' service: 'Prysm Beacon' group: 'Goerli Prysm/Geth' explorer: 'goerli.beaconcha.in' ``` ### Consensus Client API Configuration To configure the Prometheus target for client APIs, follow the instructions for your client below. {% tabs %} {% tab title="Lighthouse" %} Edit the Lighthouse configuration file in the `/etc/prometheus/files_sd/stakelocal/consensus_apis` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/consensus_apis/lighthouse.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:5052'] labels: network: 'Mainnet' host: 'Default Host' service: 'Lighthouse Beacon' group: 'Default Group' explorer: 'beaconcha.in' ``` If this instance of Lighthouse is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% tab title="Lodestar" %} Edit the Lodestar configuration file in the `/etc/prometheus/files_sd/stakelocal/consensus_apis` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/consensus_apis/lodestar.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:9596'] labels: network: 'Mainnet' host: 'Default Host' service: 'Lodestar Beacon' group: 'Default Group' explorer: 'beaconcha.in' ``` If this instance of Lodestar is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% tab title="Nimbus" %} Edit the Nimbus configuration file in the `/etc/prometheus/files_sd/stakelocal/consensus_apis` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/consensus_apis/nimbus.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:5052'] labels: network: 'Mainnet' host: 'Default Host' service: 'Nimbus' group: 'Default Group' explorer: 'beaconcha.in' ``` If this instance of Nimbus is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% tab title="Prysm" %} Edit the Prysm configuration file in the `/etc/prometheus/files_sd/stakelocal/consensus_apis` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/consensus_apis/prysm.yml ``` It should look like the following.

    - targets: ['127.0.0.1:3500']
      labels:
        network: 'Mainnet'
        host: 'Default Host'
        service: 'Prysm Beacon'
        group: 'Default Group'
        explorer: 'beaconcha.in'

If this instance of Prysm is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% tab title="Teku" %} Edit the Teku configuration file in the `/etc/prometheus/files_sd/stakelocal/consensus_apis` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/consensus_apis/teku.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:5051'] labels: network: 'Mainnet' host: 'Default Host' service: 'Teku' group: 'Default Group' explorer: 'beaconcha.in' ``` If this instance of Teku is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% endtabs %} An example consensus client API target file for the Mainnet network could look like the following. ``` - targets: ['127.0.0.1:5051'] labels: network: 'Mainnet' host: 'wopr' service: 'Backup Teku' group: 'Mainnet Teku/Besu' explorer: 'beaconcha.in' ``` # Execution Client Targets Customize the default execution client targets ## Execution Client Targets Execution client data is collected by Prometheus from three different sources: 1. Directly from the client metrics page 2. Indirectly from the client API via json\_exporter 3. Indirectly from the client API via Ethereum Metrics Exporter We will set up the first two here. Ethereum Metrics Exporter is covered in a later section. {% hint style="info" %} If you would like to monitor more than one instance of the same client, please see the Additional Modifications section for details. {% endhint %} ### Execution Client Metrics Configuration To configure the Prometheus target for client metrics, follow the instructions for your client below. {% tabs %} {% tab title="Besu" %} Edit the Besu configuration file in the `/etc/prometheus/files_sd/stakelocal/execution_metrics` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/execution_metrics/besu.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:9545'] labels: network: 'Mainnet' host: 'Default Host' service: 'Besu' group: 'Default Group' explorer: 'etherscan.io' ``` If this instance of Besu is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% tab title="Erigon" %} Edit the Erigon configuration file in the `/etc/prometheus/files_sd/stakelocal/execution_metrics` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/execution_metrics/erigon.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:6060'] labels: network: 'Mainnet' service: 'Erigon' host: 'Default Host' group: 'Default Group' explorer: 'etherscan.io' ``` If this instance of Erigon is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% tab title="Geth" %} Edit the Geth configuration file in the `/etc/prometheus/files_sd/stakelocal/execution_metrics` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/execution_metrics/geth.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:6060'] labels: network: 'Mainnet' host: 'Default Host' service: 'Geth' group: 'Default Group' explorer: 'etherscan.io' ``` If this instance of Geth is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% tab title="Nethermind" %} Edit the Nethermind configuration file in the `/etc/prometheus/files_sd/stakelocal/execution_metrics` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/execution_metrics/nethermind.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:8080'] labels: network: 'Mainnet' host: 'Default Host' service: 'Nethermind' group: 'Default Group' explorer: 'etherscan.io' ``` If this instance of Nethermind is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% endtabs %} An example execution client API target file for the Sepolia network could look like the following. ``` - targets: ['localhost:8080'] labels: network: 'Sepolia' host: 'nuc' service: 'Nethermind' group: 'Sepolia Nimbus/Nethermind' explorer: 'sepolia.etherscan.io' ``` ### Execution Client API Configuration To configure the Prometheus target for client APIs, follow the instructions for your client below. {% tabs %} {% tab title="Besu" %} Edit the Besu configuration file in the `/etc/prometheus/files_sd/stakelocal/execution_apis` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/execution_apis/besu.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:8545'] labels: network: 'Mainnet' host: 'Default Host' service: 'Besu' group: 'Default Group' explorer: 'etherscan.io' json_exporter: '127.0.0.1:7979' client: 'Besu' ``` If this instance of Besu is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. The `json_exporter` and `client` labels may be left as-is. Save the file and exit the editor. {% endtab %} {% tab title="Erigon" %} Edit the Erigon configuration file in the `/etc/prometheus/files_sd/stakelocal/execution_apis` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/execution_apis/erigon.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:8545'] labels: network: 'Mainnet' host: 'Default Host' service: 'Erigon' group: 'Default Group' explorer: 'etherscan.io' json_exporter: '127.0.0.1:7979' client: 'Erigon' ``` If this instance of Erigon is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. The `json_exporter` and `client` labels may be left as-is. Save the file and exit the editor. {% endtab %} {% tab title="Geth" %} Edit the Geth configuration file in the `/etc/prometheus/files_sd/stakelocal/execution_apis` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/execution_apis/geth.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:8545'] labels: network: 'Mainnet' host: 'Default Host' service: 'Geth' group: 'Default Group' explorer: 'etherscan.io' json_exporter: '127.0.0.1:7979' client: 'Geth' ``` If this instance of Geth is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. The `json_exporter` and `client` labels may be left as-is. Save the file and exit the editor. {% endtab %} {% tab title="Nethermind" %} Edit the Nethermind configuration file in the `/etc/prometheus/files_sd/stakelocal/execution_apis` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/execution_apis/nethermind.yml ``` It should look like the following. ``` - targets: ['127.0.0.1:8545'] labels: network: 'Mainnet' host: 'Default Host' service: 'Nethermind' group: 'Default Group' explorer: 'etherscan.io' json_exporter: '127.0.0.1:7979' client: 'Nethermind' ``` If this instance of Nethermind is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. The `json_exporter` and `client` labels may be left as-is. Save the file and exit the editor. {% endtab %} {% endtabs %} An example execution client metrics target file for the Goerli network could look like the following. ``` - targets: ['192.168.0.12:8545'] labels: network: 'Goerli' host: 'mrwhiskers' service: 'Erigon' group: 'Goerli Lighthouse/Erigon' explorer: 'goerli.etherscan.io' json_exporter: '127.0.0.1:7979' client: 'Erigon' ``` # Validator Client Targets Customize the default validator client targets ## Validator Client Targets Prometheus may also collect metrics data from validator clients that run as a separate process from the beacon chain client. Some validator clients produce more detailed validator status information than available from beacon chain clients. {% hint style="warning" %} Validator clients for Lighthouse, Lodestar, and Prysm have been tested with the dashboard. {% endhint %} {% hint style="info" %} If you would like to monitor more than one instance of the same client, please see the Additional Modifications section for details. {% endhint %} ### Validator Client Metrics Configuration To configure the Prometheus target for client metrics, follow the instructions for your client below. {% tabs %} {% tab title="Lighthouse" %} Edit the Lighthouse configuration file in the `/etc/prometheus/files_sd/stakelocal/validator_metrics` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/validator_metrics/lighthouse.yml ``` It should look like the following. ```yaml - targets: ['127.0.0.1:5064'] labels: network: 'Mainnet' host: 'Default Host' service: 'Lighthouse Validator' group: 'Default Group' ``` If this instance of Lighthouse is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% tab title="Lodestar" %} Edit the Lodestar configuration file in the `/etc/prometheus/files_sd/stakelocal/validator_metrics` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/validator_metrics/lodestar.yml ``` It should look like the following. ```yaml - targets: ['127.0.0.1:5064'] labels: network: 'Mainnet' host: 'Default Host' service: 'Lodestar Validator' group: 'Default Group' ``` If this instance of Lodestar is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% tab title="Prysm" %} Edit the Prysm configuration file in the `/etc/prometheus/files_sd/stakelocal/validator_metrics` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/validator_metrics/prysm.yml ``` It should look like the following. ```yaml - targets: ['127.0.0.1:8081'] labels: network: 'Mainnet' host: 'Default Host' service: 'Prysm Validator' group: 'Default Group' ``` If this instance of Prysm is accessible at a different IP address or port, update those in the `targets` line. Update the `network`, `host`, `service`, `group` and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. {% endtab %} {% endtabs %} An example validator client metrics target file for the Goerli network could look like the following. ``` - targets: ['127.0.0.1:5064'] labels: network: 'Goerli' host: 'stakemaster5000' service: 'Lodestar Validator - Goerli' group: 'Goerli Lodestar/Geth' ``` # node\_exporter Target Customize the default node\_exporter target ## node\_exporter Target Edit the node\_exporter configuration file in the `/etc/prometheus/files_sd/stakelocal/other` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/other/node_exporter.yml ``` It should look like the following. ```yaml - targets: ['127.0.0.1:9100'] labels: host: 'Default Host' service: 'Node Exporter' client: 'Node Exporter' ``` If this instance of node\_exporter is accessible at a different IP address or port, update those in the `targets` line. Update the `host` or `service` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. An example target file could look like the following. ``` - targets: ['127.0.0.1:9100'] labels: host: 'host043' service: 'Node Exporter' client: 'Node Exporter' ``` # json\_exporter Target Customize the default json\_exporter target ## json\_exporter Target Edit the json\_exporter configuration file in the `/etc/prometheus/files_sd/stakelocal/other` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/other/json_exporter.yml ``` It should look like the following. ```yaml - targets: ['127.0.0.1:7979'] labels: host: 'Default Host' service: 'JSON Exporter' client: 'JSON Exporter' ``` If this instance of json\_exporter is accessible at a different IP address or port, update those in the `targets` line. Update the `host` or `service` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. An example target file could look like the following. ``` - targets: ['127.0.0.1:7979'] labels: host: 'MacBook' service: 'JSON Exporter' client: 'JSON Exporter' ``` # Ethereum Metrics Exporter Targets Customize the default Ethereum Metrics Exporter target ## Ethereum Metrics Exporter Targets For accessing Ethereum client APIs, Ethereum Metrics Exporter has an advantage over json\_exporter. It can craft API queries that json\_exporter cannot, and it maintains state over time, allowing it to subscribe to events, for example. The dashboard was nearly complete before Ethereum Metrics Exporter was incorporated. There is overlap in the data provided by Ethereum Metrics Exporter and json\_exporter, but both have been retained in the solution for now. {% hint style="info" %} The redundant API calls may be removed in the future. For now, having both sources of data provides some flexibility to not install Ethereum Metrics Exporter and still have much of the same functionality. A small number of dashboard panels will stop working entirely if Ethereum Metrics Exporter is not installed. {% endhint %} Edit the Ethereum Metrics Exporter configuration file in the `/etc/prometheus/files_sd/stakelocal/other/` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/other/eth-metrics.yml ``` It should look like the following. ```yaml - targets: ['127.0.0.1:9095'] labels: network: 'Mainnet' host: 'Default Host' service: 'Ethereum Metrics Exporter' group: 'Default Group' client: 'Ethereum Metrics Exporter' ``` If the Ethereum Metrics Exporter instance being queried is not on the local host or uses a non-standard port, update those on the `targets` line. Update the `network`, `host`, `service`, and `group` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. {% hint style="info" %} Each instance of Ethereum Metrics Exporter can only serve a single pair of consensus and execution clients. Repeat this configuration for each instance of Ethereum Metrics Exporter to be monitored. {% endhint %} Save the file and exit the editor. An example target file could look like the following. ``` - targets: ['192.168.2.6:9095'] labels: network: 'Mainnet' host: 'nuc' service: 'Ethereum Metrics - Lodestar/Erigon' group: 'Mainnet Lodestar/Erigon' client: 'Ethereum Metrics Exporter' ``` {% hint style="info" %} Reminder: We are running Ethereum Metrics Exporter on a custom port (9095). The default port is 9090. {% endhint %} # Third-Party Targets ## Third-Party Targets The following two targets allow the Stake Local dashboard to access information from external APIs. ### GitHub GitHub provides details about the latest release of software packages in use. View the GitHub configuration file in the `/etc/prometheus/files_sd/stakelocal/other` directory. ```bash cat /etc/prometheus/files_sd/stakelocal/other/github.yml ``` It should look like the following. ```yaml - targets: [ 'ledgerwatch/erigon' ] labels: client: 'Erigon' - targets: [ 'nethermindeth/nethermind' ] labels: client: 'Nethermind' - targets: [ 'ethereum/go-ethereum' ] labels: client: 'Geth' - targets: [ 'hyperledger/besu' ] labels: client: 'Besu' - targets: [ 'prysmaticlabs/prysm' ] labels: client: 'Prysm' - targets: [ 'consensys/teku' ] labels: client: 'Teku' - targets: [ 'status-im/nimbus-eth2' ] labels: client: 'Nimbus' - targets: [ 'sigp/lighthouse' ] labels: client: 'Lighthouse' - targets: [ 'attestantio/dirk' ] labels: client: 'Dirk' - targets: [ 'attestantio/vouch' ] labels: client: 'Vouch' - targets: [ 'chainsafe/lodestar' ] labels: client: 'Lodestar' - targets: [ 'prometheus/prometheus' ] labels: client: 'Prometheus' - targets: [ 'prometheus/node_exporter' ] labels: client: 'Node Exporter' - targets: [ 'prometheus/blackbox_exporter' ] labels: client: 'Blackbox Exporter' - targets: [ 'prometheus-community/json_exporter' ] labels: client: 'JSON Exporter' - targets: [ 'ethpandaops/ethereum-metrics-exporter' ] labels: client: 'Ethereum Metrics Exporter' - targets: [ 'grafana/grafana' ] labels: client: 'Grafana' ``` {% hint style="info" %} No changes are required for this file. {% endhint %} ### CoinGecko CoinGecko provides the latest ETH price in multiple fiat values. View the CoinGecko configuration file in the `/etc/prometheus/files_sd/stakelocal/other` directory. ```bash cat /etc/prometheus/files_sd/stakelocal/other/coingecko.yml ``` It should look like the following. ```yaml - targets: ['usd,eur,jpy,gbp,aud,cad,chf,cny,hkd,nzd,ils,krw,mxn,rub,sgd,twd,inr'] labels: source: 'CoinGecko' ``` {% hint style="info" %} No changes are required for this file. {% endhint %} # Ethereum Address Targets Add targets for Ethereum addresses to be monitored ## Ethereum Address Targets The Stake Local Dashboard can monitor the balance of fee addresses, withdrawal addresses, or any other Ethereum address. Due to the limited data available, it cannot provide accurate long-term reporting of earnings/fees. However, it can provide address balances and their annualized rates of change a calculated over various periods. Edit the addresses configuration file in the `/etc/prometheus/files_sd/stakelocal/` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/addresses.yml ``` It should look like the following. ```yaml - targets: [] labels: network: 'Mainnet' host: 'Default Host' service: '' group: 'Default Group' instance: '127.0.0.1:8545' explorer: 'etherscan.io' ``` Update the `targets` line to include a comma-delimited list of Ethereum addresses to be monitored in between the square brackets. Example values: * `-targets: [ 0x0000000000000000000000000000000000000000, 0x0000000000000000000000000000000000000000, 0x0000000000000000000000000000000000000000 ]` * `-targets: [ "0x0000000000000000000000000000000000000000", "0x0000000000000000000000000000000000000000", "0x0000000000000000000000000000000000000000" ]` * `-targets: [ '0x0000000000000000000000000000000000000000', '0x0000000000000000000000000000000000000000', '0x0000000000000000000000000000000000000000' ]` If the execution client being queried is not on the local host or uses a non-standard port, update those on the `instance` line. Update the `network`, `host`, `service`, `group`, and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. {% hint style="info" %} The service label is optional. It only serves to identify the type of client being queried in Ethereum address requests, but the value itself is not presently used in the dashboard. {% endhint %} An example `addresses.yml` file for the Goerli network could look like the following. ``` - targets: [ 0x0000000000000000000000000000000000000000, 0x0000000000000000000000000000000000000000 ] labels: network: 'Goerli' host: 'nuc' service: 'Geth' group: 'Goerli Prysm/Geth' instance: '127.0.0.1:8545' explorer: 'goerli.etherscan.io' ``` Save the file and exit the editor. # Validator Public Key Targets Add targets for Ethereum addresses to be monitored ## Validator Public Key Targets Prometheus collects individual validator balances and status information from consensus clients via json\_exporter. Validators may be specified by public key or validator index. Edit the validator configuration file in the `/etc/prometheus/files_sd/stakelocal/` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/validators.yml ``` It should look like the following. ```yaml - targets: [ ] labels: network: 'Mainnet' host: 'Default Host' service: '' group: 'Default Group' instance: '127.0.0.1:3500' explorer: 'beaconcha.in' ``` Update the `targets` line to include a comma-delimited list of validator public keys or indices to be monitored in between the square brackets. Example values: * `-targets: [ 1, 2, 3, 4, 5 ]` * `-targets: [ 0x0000000000000000000000000000000000000000, 0x0000000000000000000000000000000000000000, 0x0000000000000000000000000000000000000000 ]` * `-targets: [ 1, "2", '3', 0x0000000000000000000000000000000000000000, "0x0000000000000000000000000000000000000000", '0x0000000000000000000000000000000000000000']` If the consensus client being queried is not on the local host or uses a non-standard port, update those on the `instance` line. Update the `network`, `host`, `service`, `group`, and `explorer` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. {% hint style="info" %} The service label is optional. It only serves to identify the type of client being queried in Ethereum address requests, but the value itself is not presently used in the dashboard. {% endhint %} An example `validators.yml` file for the Goerli network could look like the following. ``` - targets: [ 1, 2, 3 ] labels: network: 'Goerli' host: 'nuc' service: 'Prysm Beacon' group: 'Goerli Prysm/Geth' instance: '127.0.0.1:3500' explorer: 'goerli.beaconcha.in' ``` Save the file and exit the editor. # Optional: Grafana Target Monitor some basic detail about Grafana ## Grafana ### Create Prometheus Target The Stake Local dashboard provides only basic, high-level information about Grafana. It may only include memory use, version information, and latest GitHub release. Configuring this target is optional. Edit the Grafana configuration file in the `/etc/prometheus/files_sd/stakelocal/other` directory.

sudo nano /etc/prometheus/files_sd/stakelocal/other/grafana.yml

It should look like the following. ```yaml - targets: ['127.0.0.1:3000'] labels: host: 'Default Host' service: 'Grafana' client: 'Grafana' ``` If this instance of Grafana is accessible at a different IP address or port, update those in the `targets` line. Update the `host` and `service` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. ### Configure Grafana Monitoring Turn on Grafana metrics by editing the Grafana configuration file. ```bash sudo nano /etc/grafana/grafana.ini ``` Uncomment the `enabled = true` and `disable_total_stats = false` lines in the `[metrics]` section by removing the leading semicolon. The following is an example from [Grafana's documentation on enabling metrics](https://grafana.com/docs/grafana/latest/setup-grafana/set-up-grafana-monitoring/#pull-metrics-from-grafana-into-prometheus). ``` # Metrics available at HTTP URL /metrics and /metrics/plugins/:pluginId [metrics] # Disable / Enable internal metrics enabled = true # Disable total stats (stat_totals_*) metrics to be generated disable_total_stats = false ``` Save the file and exit the editor. # Optional: Prometheus Target Monitor some basic details about Prometheus ## Prometheus Target The Stake Local dashboard provides only basic, high-level information about Prometheus. It may include memory use, version information, and latest GitHub release. Configuring this target is optional. Edit the Prometheus configuration file in the `/etc/prometheus/files_sd/stakelocal/other` directory. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/other/prometheus.yml ``` It should look like the following. ```yaml - targets: ['127.0.0.1:9090'] labels: host: 'Default Host' service: 'Prometheus' client: 'Prometheus' ``` If this instance of Prometheus is accessible at a different IP address or port, update those in the `targets` line. Update the `host` or `service` labels, as needed. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) for label definitions. Save the file and exit the editor. An example `prometheus.yml` file could look like the following. ``` - targets: ['10.0.2.54:9090'] labels: host: 'gringotts' service: 'Prometheus' client: 'Prometheus' ``` # Clean Up Unused Targets Remove Prometheus target configurations for clients not in use We started by unpacking an archive of Prometheu target files, and have now made modifications to files for clients we intend to use. At this point we may have many more target files installed for clients we do not intend to use. Remove the target configuration files for clients you do not intend to use. Do not delete the target configuration files we have already customized and intend to use. {% hint style="danger" %} **Warning:** Do not delete target configuration files for clients you are using. {% endhint %} Here are commands to delete unwanted execution client target configuration files. {% tabs %} {% tab title="Besu" %} Only run these commands if you are **NOT** monitoring Besu with the Stake Local Dashboard. ```bash sudo rm -i /etc/prometheus/files_sd/stakelocal/execution_metrics/besu.yml sudo rm -i /etc/prometheus/files_sd/stakelocal/execution_apis/besu.yml ``` {% endtab %} {% tab title="Erigon" %} Only run these commands if you are **NOT** monitoring Erigon with the Stake Local Dashboard. ```bash sudo rm -i /etc/prometheus/files_sd/stakelocal/execution_metrics/erigon.yml sudo rm -i /etc/prometheus/files_sd/stakelocal/execution_apis/erigon.yml ``` {% endtab %} {% tab title="Geth" %} Only run these commands if you are **NOT** monitoring Geth with the Stake Local Dashboard. ```bash sudo rm -i /etc/prometheus/files_sd/stakelocal/execution_metrics/geth.yml sudo rm -i /etc/prometheus/files_sd/stakelocal/execution_apis/geth.yml ``` {% endtab %} {% tab title="Nethermind" %} Only run these commands if you are **NOT** monitoring Nethermind with the Stake Local Dashboard. ```bash sudo rm -i /etc/prometheus/files_sd/stakelocal/execution_metrics/nethermind.yml sudo rm -i /etc/prometheus/files_sd/stakelocal/execution_apis/nethermind.yml ``` {% endtab %} {% endtabs %} Here are commands to delete unwanted consensus client target configuration files. {% tabs %} {% tab title="Lighthouse" %} Only run these commands if you are **NOT** monitoring Lighthouse with the Stake Local Dashboard. ```bash sudo rm -i /etc/prometheus/files_sd/stakelocal/consensus_metrics/lighthouse.yml sudo rm -i /etc/prometheus/files_sd/stakelocal/consensus_apis/lighthouse.yml ``` {% endtab %} {% tab title="Lodestar" %} Only run these commands if you are **NOT** monitoring Lodestar with the Stake Local Dashboard. ```bash sudo rm -i /etc/prometheus/files_sd/stakelocal/consensus_metrics/lodestar.yml sudo rm -i /etc/prometheus/files_sd/stakelocal/consensus_apis/lodestar.yml ``` {% endtab %} {% tab title="Nimbus" %} Only run these commands if you are **NOT** monitoring Teku with the Stake Local Dashboard. ```bash sudo rm -i /etc/prometheus/files_sd/stakelocal/consensus_metrics/nimbus.yml sudo rm -i /etc/prometheus/files_sd/stakelocal/consensus_apis/nimbus.yml ``` {% endtab %} {% tab title="Prysm" %} Only run these commands if you are **NOT** monitoring Teku with the Stake Local Dashboard. ```bash sudo rm -i /etc/prometheus/files_sd/stakelocal/consensus_metrics/prysm.yml sudo rm -i /etc/prometheus/files_sd/stakelocal/consensus_apis/prysm.yml ``` {% endtab %} {% tab title="Teku" %} Only run these commands if you are **NOT** monitoring Teku with the Stake Local Dashboard. ```bash sudo rm -i /etc/prometheus/files_sd/stakelocal/consensus_metrics/teku.yml sudo rm -i /etc/prometheus/files_sd/stakelocal/consensus_apis/teku.yml ``` {% endtab %} {% endtabs %} # Optional: Alternative Dashboard Compatibility Keep your other dashboards working with the Stake Local Dashboard ## Alternative Dashboard Compatibility If you used other Grafana dashboards with your Ethereum clients prior to installing the Stake Local Dashboard, you may want to continue using those dashboards into the future. Some of those dashboards may work seamlessly with the new Stake Local configuration as-is. Other dashboards may require configuration changes only if you want to avoid a temporary visual anomaly. And yet others will require configuration changes if you would like to use the alternative dashboard at all. See each client page for information about alternative Grafana dashboards and how the Stake Local configuration can be updated to support each dashboard. {% content-ref url="/pages/JD1wP3i4amhes3BRkAcy" %} [Besu Dashboard Compatibility](/dashboard-installation/optional-alternative-dashboard-compatibility/besu-dashboard-compatibility) {% endcontent-ref %} {% content-ref url="/pages/GOVNVwft8CE1GpmmpD3B" %} [Erigon Dashboard Compatibility](/dashboard-installation/optional-alternative-dashboard-compatibility/erigon-dashboard-compatibility) {% endcontent-ref %} {% content-ref url="/pages/gS3QNhgIq9sAqceDCnGB" %} [Geth Dashboard Compatibility](/dashboard-installation/optional-alternative-dashboard-compatibility/geth-dashboard-compatibility) {% endcontent-ref %} {% content-ref url="/pages/eYQ8XDES5VVL2nfU28JM" %} [Lighthouse Dashboard Compatibility](/dashboard-installation/optional-alternative-dashboard-compatibility/lighthouse-dashboard-compatibility) {% endcontent-ref %} {% content-ref url="/pages/MnDVGmfuMX9OmWNJwHAS" %} [Lodestar Dashboard Compatibility](/dashboard-installation/optional-alternative-dashboard-compatibility/lodestar-dashboard-compatibility) {% endcontent-ref %} {% content-ref url="/pages/kbr2gPtR2a49YR8AQb37" %} [Nethermind Dashboard Compatibility](/dashboard-installation/optional-alternative-dashboard-compatibility/nethermind-dashboard-compatibility) {% endcontent-ref %} {% content-ref url="/pages/ym8APKbRnLHdZRDqgMTq" %} [Nimbus Dashboard Compatibility](/dashboard-installation/optional-alternative-dashboard-compatibility/nimbus-dashboard-compatibility) {% endcontent-ref %} {% content-ref url="/pages/1l27lEEqnTOdWwrUkMiE" %} [Prysm Dashboard Compatibility](/dashboard-installation/optional-alternative-dashboard-compatibility/prysm-dashboard-compatibility) {% endcontent-ref %} {% content-ref url="/pages/ym8APKbRnLHdZRDqgMTq" %} [Nimbus Dashboard Compatibility](/dashboard-installation/optional-alternative-dashboard-compatibility/nimbus-dashboard-compatibility) {% endcontent-ref %} {% content-ref url="/pages/EqzKrQEuZYPQRxFtrAkT" %} [Teku Compatibility](/dashboard-installation/optional-alternative-dashboard-compatibility/teku-compatibility) {% endcontent-ref %} # Besu Dashboard Compatibility How to keep other Besu dashboards running with the Stake Local Dashboard ## **Besu Dashboard Compatibility** ### Besu Overview Dashboard - Grafana 10273 Dashboard: This Besu dashboard should work seamlessly with the Stake Local configuration if your prior Besu `job_name` was `besu`. `besu` is the default `job_name` for the [someresat](https://someresat.medium.com/guide-to-staking-on-ethereum-ubuntu-goerli-prysm-4a640794e8b5) and [Coin Cashew](https://www.coincashew.com/coins/overview-eth/guide-or-how-to-setup-a-validator-on-eth2-mainnet/part-i-installation/monitoring-your-validator-with-grafana-and-prometheus) guides to installing this Besu dashboard. If you used these guides, you may not need to make any changes. If your prior Besu `job_name` was `besu-nodes`, this dashboard will continue to work, but historical data may show as a different data series under a different color. To avoid this problem, you may update the Nimbus `job_name` in the new `prometheus.yml` file to match your prior `job_name` value. `besu-nodes` is the default `job_name` for this [Consensys ](https://consensys.net/blog/developers/setting-up-monitoring-for-hyperledger-besu-in-aws/)guide. **Optional** Edit the `/etc/prometheus/prometheus.yml` file. ```bash sudo vi /etc/prometheus/prometheus.yml ``` Find the line containing `- job_name: besu`. Change that line to `- job_name: `. Replace `` with the `job_name` value for Besu in your previous `prometheus.yml` file. If you used the Consensys guide to setting up your original dashboard, this value may be `besu-nodes`. Save the file and exit the editor. # Erigon Dashboard Compatibility How to keep other Erigon dashboards running with the Stake Local Dashboard ## **Erigon Dashboard Compatibility** ### LedgerWatch Erigon Dashboard Dashboard: [https://raw.githubusercontent.com/ledgerwatch/erigon/.../dashboards/erigon.json](https://raw.githubusercontent.com/ledgerwatch/erigon/devel/cmd/prometheus/dashboards/erigon.json) This Erigon dashboard should work seamlessly with the Stake Local configuration without any further changes to the `prometheus.yml` configuration file. # Geth Dashboard Compatibility How to keep other Geth dashboards running with the Stake Local Dashboard ## **Geth Dashboard Compatibility** ### Geth Overview Dashboard - Grafana 14053 Dashboard: This Geth dashboard should work seamlessly with the Stake Local configuration without any further changes to the `prometheus.yml` configuration file. ### Geth Server Dashboard - Grafana 15750 Dashboard: This Geth dashboard should work seamlessly with the Stake Local configuration without any further changes to the `prometheus.yml` configuration file. ### karalabe Geth Dashboard Dashboard: [https://gist.githubusercontent.com/karalabe/.../dashboard.json](https://gist.githubusercontent.com/karalabe/e7ca79abdec54755ceae09c08bd090cd/raw/3a400ab90f9402f2233280afd086cb9d6aac2111/dashboard.json) This Geth dashboard should work seamlessly with the Stake Local configuration without any further changes to the `prometheus.yml` configuration file. {% hint style="info" %} Alternative Geth dashboards that use InfluxDB instead of Prometheus are not impacted by the Stake Local Dashboard configuration. {% endhint %} # Lighthouse Dashboard Compatibility How to keep other Lighthouse dashboards running with the Stake Local Dashboard ## **Lighthouse Dashboard Compatibility** ### Lighthouse Summary Dashboard Dashboard: This Lighthouse dashboard should work seamlessly with the Stake Local configuration if your prior Lighthouse beacon chain `job_name` was `lighthouse_beacon`. and your prior Lighthouse validator `job_name` was `lighthouse_validator`. If your prior Lighthouse beacon chain or validator `job_name` were any other value, this dashboard will continue to work, but historical data may show as a different data series under a different color if it is in the selected time range. To avoid this problem, you may update the Lighthouse beacon chain and validator `job_name` values in the new `prometheus.yml` file to match your prior `job_name` values. `nodes` and `validators` are the default `job_name` values from this Coin Cashew guide. `lighthouse` is the default `job_name` from this someresat guide. `local-lighthouse` is the default `job_name` from this SigmaPrime guide. **Optional** Edit the `/etc/prometheus/prometheus.yml` file. ```bash sudo vi /etc/prometheus/prometheus.yml ``` Find the line containing `- job_name: lighthouse_beacon`. Change that line to `- job_name: `. Replace `` with the value for Besu in your previous `prometheus.yml` file. If you used the somersat guide to setting, this value may be `lighthouse`. Save the file and exit the editor. ### Yoldark34 ETH Staking Dashboard Dashboard: This Lighthouse dashboard is not compatible with the Stake Local configuration without some additional configuration. To continue to use this Lighthouse dashboard, update the Nimbus `job_name` in the new `prometheus.yml` file to match your prior value. `lighthouse` is the default for this Coin Cashew guide to installing a Nimbus dashboard. Edit the `/etc/prometheus/prometheus.yml` file. ```bash sudo vi /etc/prometheus/prometheus.yml ``` Find the line containing `- job_name: lighthouse_beacon`. Change that line to `- job_name: `. Replace `` with the value for Lighthouse beacon chain in your previous `prometheus.yml` file. If you used the Coin Cashew guide to setting up your original dashboard, this value may be `nodes`. Find the line containing `- job_name: lighthouse_validator`. Change that line to `- job_name: `. Replace `` with the value for Lighthouse validator in your previous `prometheus.yml` file. If you used the Coin Cashew guide to setting up your original dashboard, this value may be `validators`. {% hint style="danger" %} The Coin Cashew guide to installing this dashboard recommends the Lighthouse beacon `job_name` of `nodes` and the Lighthouse validator `job_name` of `validators`. Some queries in this dashboard require the use of `lighthouse` and `lighthouse_validator` for those `job_name` values, respectively. Changing the `job_name` values to `lighthouse` and `lighthouse_validator` instead of your previous values will enable all queries in this dashboard to work, but historical data may appear under a separate color/legend from present data. {% endhint %} Save the file and exit the editor. {% hint style="warning" %} If the Lighthouse beacon chain or Lighthouse validator job names are changed from their Stake Local defaults, please complete the steps to update the job name variables during [dashboard installation](/dashboard-installation/install-dashboard). {% endhint %} # Lodestar Dashboard Compatibility How to keep other Lodestar dashboards running with the Stake Local Dashboard ## Lodestar Dashboard Compatibility ### ChainSafe Lodestar Summary Dashboard Dashboard: [https://raw.githubusercontent.com/ChainSafe/lodestar/.../lodestar\_summary.json](https://raw.githubusercontent.com/ChainSafe/lodestar/unstable/dashboards/lodestar_summary.json) The Lodestar dashboard listed above has a hard-coded `job_name` in many queries. To continue to use this dashboard, the Lodestar `job_name` in the new configuration file must be updated to match the hard-coded value. Edit the `/etc/prometheus/prometheus.yml` file. ```bash sudo vi /etc/prometheus/prometheus.yml ``` Find the line containing `- job_name: lodestar_beacon`. Change that line to `- job_name: `. Replace `` with the value for the Lodestar beacon in your previous `prometheus.yml` file. If you used the Coin Cashew guide to setting up your original dashboard, this value may be `Lodestar`. If you used the someresat guide to setting up your original dashboard, this value may be `lodestarbeacon`. Find the line containing `- job_name: lodestar_validator`. Change that line to `- job_name: `. Replace `` with the value for the Lodestar beacon in your previous `prometheus.yml` file. You may not have this job in your prior `prometheus.yml` configuration file. {% hint style="danger" %} The Coin Cashew and someresat guides may recommend default `job_name` values that are incompatible with some portions of the alternative Lodestar dashboard. You may continue to use the prior values recommended by these guides for a consistent user experience. However, using the values `beacon` and `validator` in place of your prior values may provide more complete functionality for the original dashboard. This will cause the color change effect for historical data on the alternative Lodestar dashboard, but full functionality should be returned to the dashboard. {% endhint %} Replace `` with the value for Besu in your previous `prometheus.yml` file. If you used the Consensys guide to setting up your original dashboard, this value may be `besu-nodes`. Save the file and exit the editor. # Nethermind Dashboard Compatibility How to keep other Nethermind dashboards running with the Stake Local Dashboard ## **Nethermind Dashboard Compatibility** ### NethermindEth Nethermind Dashboard Dashboard: This Nethermind dashboard should work seamlessly with the Stake Local configuration without any further changes to the `prometheus.yml` configuration file. # Nimbus Dashboard Compatibility How to keep other Nimbus dashboards running with the Stake Local Dashboard ## Nimbus Dashboard Compatibility ### Status IM Beacon Nodes Grafana Dashboard Dashboard: This Nimbus dashboard should work seamlessly with the Stake Local configuration if your prior Nimbus `job_name` was `nimbus`. `nimbus` is the default `job_name` for the someresat guide. If your prior Nimbus `job_name` was anything else, this dashboard will continue to work, but historical data may show as a different data series under a separate color/legend. To avoid this problem, you may update the Nimbus `job_name` in the new `prometheus.yml` file to match your prior `job_name` value. `nodes` is the default `job_name` for the Coin Cashew guide. **Optional** Edit the `/etc/prometheus/prometheus.yml` file. ```bash sudo vi /etc/prometheus/prometheus.yml ``` Find the line containing `- job_name: teku`. Change that line to `- job_name: `. Replace `` with the `job_name` value for Teku in your previous `prometheus.yml` file. If you used the Coin Cashew or someresat guides to setting up this dashboard, this value may be `nodes`. Save the file and exit the editor. # Prysm Dashboard Compatibility How to keep other Prysm dashboards running with the Stake Local Dashboard ## **Prysm Dashboard Compatibility** ### Prysmatic Labs and GuillaumeMiralles Dashboards Dashboard: Dashboard: Dashboard: To continue to use the alternative Prysm dashboards listed above, update the Prysm`job_name` in the new `prometheus.yml` file to match your prior value. `nodes` is the default for this Coin Cashew guide to installing a Nimbus dashboard. Edit the `/etc/prometheus/prometheus.yml` file. ```bash sudo vi /etc/prometheus/prometheus.yml ``` Find the line containing `- job_name: prysm_beacon`. Change that line to `- job_name: `. Replace `` with the value for Prysm beacon chain in your previous `prometheus.yml` file. If you used the Coin Cashew guide to setting up your original dashboard, this value may be `beacon node`. If you used the someresat guide to setting up your original dashboard, this value may be `prysm-beacon`. Find the line containing `- job_name: prysm_validator`. Change that line to `- job_name: `. Replace `` with the value for Prysm validator in your previous `prometheus.yml` file. If you used the Coin Cashew guide to setting up your original dashboard, this value may be `validator`. If you used the someresat guide to setting up your original dashboard, this value may be `prysm-validator`. {% hint style="danger" %} The someresat guide to installing this dashboard recommends the Prysm beacon `job_name` of `prysm-beacon` and the Prysm validator `job_name` of `prysm-validator`. Some queries in this dashboard require the use of `beacon_node` and `validator` for those `job_name` values, respectively. Changing the `job_name` values to `beacon_node` and `validator` instead of your previous values will enable all queries in this dashboard to work, but historical data may appear under a separate color/legend from present data. {% endhint %} Save the file and exit the editor. {% hint style="warning" %} If the Prysm beacon chain or Prysm validator job names are changed from their Stake Local defaults, please complete the steps to update the job name variables during [dashboard installation](/dashboard-installation/install-dashboard). {% endhint %} # Teku Compatibility How to keep other Teku dashboards running with the Stake Local Dashboard ## Teku ### Teku Dashboard - Grafana 13457 Dashboard: This Teku dashboard should work seamlessly with the Stake Local configuration if your prior Teku `job_name` was `teku`. If your prior Teku `job_name` was anything else, this dashboard will continue to work, but historical data may show as a different data series under a separate color/legend. To avoid this problem, you may update the Teku `job_name` in the new `prometheus.yml` file to match your prior `job_name` value. `nodes` is the default `job_name` for the Coin Cashew and someresat guides. **Optional** Edit the `/etc/prometheus/prometheus.yml` file. ```bash sudo vi /etc/prometheus/prometheus.yml ``` Find the line containing `- job_name: teku`. Change that line to `- job_name: `. Replace `` with the `job_name` value for Teku in your previous `prometheus.yml` file. If you used the Coin Cashew or someresat guides to setting up this dashboard, this value may be `nodes`. Save the file and exit the editor. # Enable & Start/Restart Services Getting everything up, running, and starting on its own Everything is now configured and ready to be started or restarted. {% hint style="info" %} For any software installed using another guide, follow your usual instructions for starting or restarting the consensus client, the execution client, Prometheus, Grafana, node\_exporter, json\_exporter, and Ethereum Metrics Exporter. {% endhint %} Refresh the service files for systemd so that systemd knows there are changes to how services are to be run. {% code overflow="wrap" %} ```bash sudo systemctl daemon-reload ``` {% endcode %} Enable all services. {% code overflow="wrap" %} ```bash sudo systemctl enable prometheus grafana-server node_exporter json_exporter eth-metrics ``` {% endcode %}

The quick and easy method to get everything running/reloaded is to simply tell all services to restart. Everything already running will restart and the rest will start. {% code overflow="wrap" %} ```bash sudo systemctl restart prometheus grafana-server node_exporter json_exporter eth-metrics ``` {% endcode %}

The `systemctl restart` command does not provide any feedback on how things are running. We can check the status on the next page. {% hint style="warning" %} If you changed any command-line flags for your consensus or execution clients, reload and restart those services using your standard procedures. {% endhint %} # Checking Software Status Monitoring the monitors ## Checking Software Status ### systemctl The quick way to check the status of all of the supporting software is to run `systemctl status` on all of the services. ```bash sudo systemctl status prometheus grafana-server node_exporter json_exporter eth-metrics ``` You will see output like the following.

If you would like to see the logs individually for the supporting software, use the following commands to see the last 100 lines of log files. ``` sudo journalctl -u prometheus -n 100 sudo journalctl -u grafana-server -n 100 sudo journalctl -u node_exporter -n 100 sudo journalctl -u json_exporter -n 100 sudo journalctl -u eth-metrics -n 100 ``` Replace "100" in the commands above with any number of lines you would like to see. To see a never-ending stream of logs, use these commands, and use CTRL-C to exit. ```bash sudo journalctl -fu prometheus sudo journalctl -fu grafana-server sudo journalctl -fu node_exporter sudo journalctl -fu json_exporter sudo journalctl -fu eth-metrics ``` Healthy Startups Here is what a healthy start looks like in the logs for the supporting software. ### Logs #### Prometheus

#### Grafana

#### node\_exporter

#### json\_exporter

#### Ethereum Metrics Exporter

Prometheus Targets One way to check if your supporting software is running is to see whether Prometheus is able to scrape the software. Go to `http://127.0.0.1:9090`, replacing `127.0.0.1` with the IP address at which Prometheus is running. At the top of the screen you should see a Status menu. Hover over that menu and click on the Targets link.

Find the Targets link under the Status menu

The Targets screen will appear listing all of the Prometheus targets we previously set up. You may need click Clear Selection in the targets filter in the upper-left corner of the screen.

Clear the selection to see all Prometheus targets

Each line of blue or red text represents a single Prometheus job serving one or more targets. * Blue means Prometheus has successfully scraped data recently * Red means Prometheus has not successfully scraped data recently from one The numbers in parentheses show how many of the targets within each group Prometheus has successfully scraped recently.

The Targets screen showing the status of various Prometheus jobs

{% hint style="warning" %} If Prometheus has not had any data from a target within the last 15 minutes, it will show that target in the color red on this screen. Two of the jobs will often show a red color on this screen: * consensus\_beacon\_genesis - This job runs every few hours, because the genesis information never changes. Therefore, Prometheus often shows this job in red. * github\_latest\_release - This job runs every few hours, because GitHub has a low rate limit. Therefore, Prometheus often shows this job in red. Additionally, it may take a couple hours after a new release for that release to be reflected on the Prometheus Targets screen. The last job that may be red is blackbox\_exporter. That job is not presently covered in these instructions. {% endhint %} # Install Dashboard Configure Grafana and install the Stake Local Dashboard ### Install Dashboard ### Login to Grafana Login to Grafana at [http://127.0.0.1:3000/](http://xxx.xxx.xxx.xxx:3000/), replacing `127.0.0.1` with the IP address of your Grafana instance. If this is your first time logging in to Grafana, use the default username `admin` and the default password `admin`. Grafana will ask you to set a new password. ### Configure Prometheus Data Source Connect Grafana to your Prometheus instance by configuring the Prometheus Data Source in Grafana. On Grafana's left-hand menu, hover over the gear menu and click on Data Sources.

Then click on the Add Data Source button.

Click on Prometheus under "Time series databases".

Enter `http://127.0.0.1:9090/` into the URL field. If your Prometheus instance is accessible at a different IP address or port, update those here. Otherwise, click on the Add Data Source button.

Enter the URL to your Prometheus instance

Click Save & Test.

Grafana should report that the data source is working. #### **Install Grafana Dashboard** Hover over the Dashboards menu in Grafana's left-hand menu, then click on Import.

Copy and paste the dashboard at into the "Import via panel json" text box on the screen. [Dashboard JSON via GitHub](https://raw.githubusercontent.com/stakelocal/stakelocal-dashboard/main/dashboard/stakelocal-dashboard.json) It is also available in the cloned Stake Local Dashboard GitHub repository at `~/git/stakelocal-dashboard/dashboard/stakelocal-dashboard.json`. ```bash cat ~/git/stakelocal-dashboard/dashboard/stakelocal-dashboard.json ``` Then click the Load button.

Paste the JSON and click on the Load button

Select your Prometheus data source in the Prometheus field.

Optionally change the Currency Code value to one of the supported currencies: USD, EUR, JPY, GPB, AUD, CAD, CHF, CNY, HKD, NZD, ILS, KRW, MXN, RUB, SGD, TWD, or INR.

If you used the [Optional: Alternative Dashboard Compatibility](/dashboard-installation/optional-alternative-dashboard-compatibility) instructions to change the default Stake Local Prometheus `job_name` to another value for Lighthouse or Prysm, you must update the job names on this screen. Update the `Prysm Beacon Job Name`, `Prysm Validator Job Name`, `Lighthouse Beacon Job Name`, and `Lighthouse Validator Job Name` variables to match your new `job_name` values, if applicable.

Change Lighthouse and Prysm job names for dashboard compatibility

Click the Import button. The dashboard should appear and data should begin to populate. Some data may take additional time to populate, from an epoch to two hours. {% hint style="success" %} Congratulations, you have set up the Stake Local Dashboard. {% endhint %} # Additional Modifications What else can you do? The following are some changes you can make to your Stake Local Dashboard after you have completed setting up your first staking group. * [Additional Staking Groups](/additional-modifications/additional-staking-groups) - Add more consensus/execution pairs to the dashboard. * [Change or Add Currency](/additional-modifications/change-or-add-currency) - Change to another pre-configured currency for fiat value calculations, or add a new currency. # Additional Staking Groups Set up Prometheus to monitor multiple groups of consensus/execution clients ## Additional Staking Groups The Stake Local Dashboard can monitor multiple consensus/execution pairs at a time. Each consensus and execution client pair is assigned to a group. A new client group can be created as simply as configuring Prometheus targets for the new clients and assigning them a unique `group` label. ### Configure Prometheus Targets To start adding a new client: Create new consensus client targets, execution client targets, and validator client targets. Return to the these three previous sections and add a new `targets` sections to the client-specific configuration files. * [Consensus Client Targets](/dashboard-installation/prometheus-jobs-and-targets/consensus-client-targets) * [Execution Client Targets](/dashboard-installation/prometheus-jobs-and-targets/execution-client-targets) * [Validator Client Target](/dashboard-installation/prometheus-jobs-and-targets/validator-client-targets) Copy the existing targets in the files, or copy the default targets from this documentation. {% hint style="info" %} Multiple targets sections can appear in the same file, but they should always be for the same client type. {% endhint %} For each new `targets` section created, modify the IP address, port, and label values. #### IP Address and Port The new client will have a unique IP address or port. Update the IP address and port of the new `targets` line to the correct IP address and port for the new client. #### Labels Labels help classify data from different client instances. Depending on your architecture, one or more of these labels may need to change for your new `targets` section. See [About Stake Local Labels](/dashboard-installation/prometheus-jobs-and-targets/about-stake-local-labels) to refresh to refresh your knowledge of how the Stake Local Dashboard uses Prometheus labels. Below the `labels` line in the `targets` section for the new client: * Update the `network` label for the new client. The network of the new client may/may not be the same as for the original client. Examples: Mainnet, Goerli * Update the `group` label for the new client. The group of the new client is most likely a different from that of the original client. Examples: Mainnet Prysm/Geth, Goerli Teku/Besu, Group A * Update the `host` label for the new client. The host of the new client may/may not be the same as for the original client. Examples: nuc, node.stakelocal.io, srv-lax-03932, trogdor * Update the `service` label for the new client. The service label is used to distinguish individual instances of a client type. With only one instance of each type, you can keep it simple with "Nimbus" and "Nethermind", but with multiple instances of each type the service values should be unique. Examples: Mainnet Lodestar, Goerli Erigon,, Backup Lighthouse, srv-lax-03932-besu After adding a second Besu client, one target file might look like the following. ``` - targets: ['127.0.0.1:8545'] labels: network: 'Mainnet' host: 'srv01' service: 'Mainnet Besu' group: 'Mainnet Teku/Besu' explorer: 'etherscan.io' json_exporter: '127.0.0.1:7979' client: 'Besu' - targets: ['192.168.0.50:8545'] labels: network: 'Goerli' host: 'srv02' service: 'Goerli Besu' group: 'Goerli Teku/Besu' explorer: 'goerli.etherscan.io' json_exporter: '127.0.0.1:7979' client: 'Besu' ``` {% hint style="info" %} Prometheus targets are read in from files that follow a naming convention. All files that begin with the name of a client in lower case are read in by Prometheus and associated with jobs appropriate for that client. The default file names provided in this guide use only the client name in the file name (e.g. teku.yml, nethermind.yml), but Prometheus will also read in files such as the following: * `prysm-mainnet.yml` * `erigon-goerli.yml` * `nimbus-srv-lax-039302.yml` The only requirement is that the name of the client be in lower case at the beginning of the filename. This provides greater flexibility in organizing Prometheus target configurations. {% endhint %} {% hint style="warning" %} Why so many target files in the first place? Not all clients support all API calls. Separating targets by client type allows us to configure API-specific Prometheus jobs to execute for only those clients that correctly support the end point. {% endhint %} ### Configure Ethereum Exporter Ethereum Metrics Exporter can only support a single consensus/execution client pair. For full dashboard support of a new pair, a new instance of Ethereum Metrics Exporter must be run. To add another Ethereum Metrics Exporter instance, you must either run Ethereum Metrics Exporter on a separate system from your first instance, or you must configure Ethereum Metrics Exporter to run more than once on the same system under a different port. #### New Instance on Separate System To install a new instance of Ethereum Metrics Exporter on a separate system, follow all of the Ethereum Metrics Exporter instructions on the new system. 1. [Install Ethereum Metrics Exporter](/dashboard-installation/install-supporting-software/install-ethereum-metrics-exporter) on the new system. 2. [Configure Ethereum Metrics Exporter ](/dashboard-installation/configure-supporting-software/configure-ethereum-metrics-exporter)on the new system. 3. [Configure the Prometheus target for the new Ethereum Metrics Exporter](/dashboard-installation/prometheus-jobs-and-targets/ethereum-metrics-exporter-targets). 4. Enable and start the Ethereum Metrics Exporter software on the new system. 5. Add a new Ethereum Metrics Exporter target to your `eth-metrics.yml` file in `/etc/prometheus/files_sd/stakelocal/other/` directory. Modify the IP address, port, and labels, as necessary. You do not need to restart Prometheus. Prometheus will pick up the changes shortly after you save the target file. #### Second Instance on Same System To run a second instance of Ethereum Metrics Exporter on the same system: 1. Skip the installation steps, but follow the steps to [Configure Startup](/dashboard-installation/install-supporting-software/install-ethereum-metrics-exporter#configure-startup) for Ethereum Metrics Exporter from the [Install Ethereum Metrics Exporter](/dashboard-installation/install-supporting-software/install-ethereum-metrics-exporter) page. 1. Modify the commands on that page to use a unique filename for the second instance of Ethereum Metrics Exporter. For example: `eth-metrics-goerli.service`, `eth-metrics-backup.service` 2. Change the port in the `ExecStart` line to the port on which the new instance of Ethereum Metrics Exporter is running 3. Change the name of the configuration file in the `ExecStart` line to a name unique to the new instance of Ethereum Metrics Exporter. For example: `eth-metrics-goerli.yml`, `eth-metrics-backup.yml`. 2. Repeat the steps to [Configure Ethereum Metrics Exporter](/dashboard-installation/configure-supporting-software/configure-ethereum-metrics-exporter), using the configuration filename configured in the previous step. For example: `eth-metrics-goerli.yml`, `eth-metrics-backup.yml` 3. Repeat the steps to configure [Ethereum Metrics Exporter Targets](/dashboard-installation/prometheus-jobs-and-targets/ethereum-metrics-exporter-targets), creating a new `targets` section in the `/etc/prometheus/files_sd/stakelocal/other/eth-metrics.yml` file 4. Enable and start the new Ethereum Metrics Exporter service with `sudo systemctl enable --now eth-metrics-goerli`, using the name of the `.service` file created above. # Change or Add Currency Change the currency used to calculate fiat values of Ether on the dashboard. ## Change or Add Currency The dashboard comes preconfigured to support 17 different currencies for fiat value conversions. | USD | EUR | JPY | | --- | --- | --- | | GBP | AUD | CAD | | CHF | CNY | HKD | | NZD | ILS | KRW | | MXN | RUB | SGD | | TWD | INR | | It is easy to change to one of the supported currencies through the Grafana UI using the instructions below. If you want to add an unsupported currency, instructions are included for that afterward. ## Change Currency Follow these steps to change the selected currency to another supported currency. Click on the gear icon in the upper-right corner of the Grafana dashboard.

Click on the Variables link on the left-hand side of the screen.

Click on the `currency` variable name.

The currency variable in the variables list

Change the variable value to one of the 17 supported currencies listed in the Description field.

Click the Apply button to save the change.

Click the Apply button to apply the new currency value

Click on the Save Dashboard button in the upper-right corner of the screen.

Click the Save Dashboard button to open the save panel

The Save Dashboard panel will open. Enter a note and then click on the Save button.

## Add New Currency Additional languages may be added manually by modifying two configuration files and then following the above instructions to change the currency variable. Currencies must be supported by the [Coingecko API](https://www.coingecko.com/en/api/documentation). #### /etc/json\_exporter/json\_exporter.yml First, edit the `json_exporter` configuration file to add support for parsing the new currency values ```bash sudo nano /etc/json_exporter/json_exporter.yml ``` Find the `coingecko_eth_price` section and look for the list of currencies under the `values` line. That section should look similar to the following. ``` coingecko_eth_price: metrics: - name: eth type: object path: '{.ethereum}' help: Ether (ETH) price in fiat values: USD: '{.usd}' EUR: '{.eur}' JPY: '{.jpy}' GBP: '{.gbp}' AUD: '{.aud}' CAD: '{.cad}' CHF: '{.chf}' CNY: '{.cny}' HKD: '{.hkd}' NZD: '{.nzd}' ILS: '{.ils}' KRW: '{.krw}' MXN: '{.mxn}' RUB: '{.rub}' SGD: '{.sgd}' TWD: '{.twd}' INR: '{.inr}' ``` Add a new currency line following the model of the other currency lines, with the three-character currency code first in all caps and then in lowercase. For example, if we want to add support for Swedish krona, we would add the following line to the `values` section of the `coingecko_eth_price` module. There should be eight spaces before the currency code. ```yaml SEK: '{.sek}' ``` #### /etc/prometheus/files\_sd/other/currencies.yml Second, edit the `currencies.yml` configuration file to add support for the new currency to the CoinGeck query. ```bash sudo nano /etc/prometheus/files_sd/stakelocal/other/coingecko.yml ``` In the `targets` line, add the three-character currency code to the comma-delimited list of currencies to be queried. All three-character currency codes must be contained within the single set of quotes. For example, to continue adding support for Swedish Krona, we would add `,sek` to the end of the `targets` line list of currencies. The final result may look like the following. ``` - targets: ['usd,eur,jpy,gbp,aud,cad,chf,cny,hkd,nzd,ils,krw,mxn,rub,sgd,twd,inr,sek'] ``` After modifying these two files, restart json\_exporter only. The changes to the Prometheus configuration will be picked up automatically. ```bash sudo systemctl restart json_exporter ``` After completing the setup, set your new currency code as the `currency` variable value using the [instructions at the top of this page](#change-currency).