Stake Local Dashboard

A multi-client Grafana dashboard for Ethereum staking

Introduction

The Stake Local Dashboard is a Grafana dashboard and software configuration that provides access to Ethereum staking data from nine different Ethereum clients. Clients may be configured in one or more staking groups that are running on one or more hosts. The dashboard relies on data collected from client metrics pages and client API calls to populate the customizable dashboard. Third-party APIs are also used to collect the current ETH price (CoinGecko) and the latest client version information (GitHub). The result is a powerful tool for viewing the status of staking operations.

This really should be called a dashboard library, because it is meant to be customized to meet your staking needs. The panels within may not all be relevant to your clients or your monitoring area of focus. They may not be organized according to your priorities. Therefore, remove or reorganize panels, hide columns or queries, and make the dashboard your own.

The following screenshot shows a sample dashboard using a mix of panels that I selected for an at-a-glance view of testnet staking operations. This is just one way that the library of panels can be personalized to your staking operations, and the dashboard depicted below is only using one quarter of the panels available.

Additional screenshots of the sample dashboard for other Ethereum clients are in the Sample Dashboard section below.

Key Features

Consensus clients: Lighthouse, Lodestar, Nimbus, Prysm, and Teku
Execution clients: Besu, Erigon, Geth, and Nethermind
84+ primary multi-client dashboard panels with significant support across clients
28 additional library panels with limited support across clients
Group consensus and execution clients into pairs and easily switch between groups
Displays multiple staking groups at a time
Track the status of multiple hosts and easily switch between hosts
Displays multiple hosts at a time
Displays and calculates validator and address balances and rates of change over time
Supports for multiple currencies for fiat values, via CoinGecko. Includes: USD, EUR, JPY, GBP, AUD, CAD, CHF, CNY, HKD, NZD, ILS, KRW, MXN, RUB, SGD, TWD, and INR
Displays client current version and latest release from GitHub, with links to release notes
Links to Beaconcha.in for slot, block, epoch and validator details
Links to Etherscan.io for Ethereum address details
Explorer links can be customized to the network served. Links for Goerli nodes will go to Goerli explorers, for example.
Provides some status of supporting software: Grafana, Prometheus, json_exporter, node_exporter, and Ethereum Metrics Exporter

If you want to get started without reading more about the dashboard, begin the installation and configuration process with Dashboard Installation.

pageInstallation Overview

What is Stake Local?

Stake Local is the new name under which I will be publishing my contributions to the Ethereum staking community. I wanted to distinguish these efforts independent of my username, and I wanted the new name to have an obvious connection to staking. Stake Local is not just a name, though. It is also an imperative for decentralized and independent staking. Stake Local!

Screenshots

The screenshots in the sections that follow show each primary dashboard section for each of the five different Goerli staking groups listed in the image below.

Switching between groups or hosts can be done using the dashboard variable drop-downs at the top of the screen.

Clients Section

The Clients section displays panels related to the operation of the clients, including:

Sync status
Peer count
Peer count by client type
RPC received per second
Execution layer transaction pools

Some data is presented on multiple panels in different forms. For example, peer types are shown in the Clients section as a pie chart of current peer counts and also as as bar chart showing peer counts over time. You can choose the panels you would like to keep or move around within the dashboard.

Network Section

The Network section covers the status of the Ethereum network (i.e. Mainnet, Goerli, Sepolia). This information is independent from the clients being monitored and data should be similar from any staking group on the same network.

Participation Rate
Network liveness
Validator counts
Current slot, epoch, and block numbers
Base fee per gas
Gas used
Epochs since finalization
Expected time between proposals
Head block information
Beacon reorgs

Software Section

The Software section shows software information for both the selected group and the selected host. Software is displayed with self-reported current version information, version number of the latest GitHub release, software uptime, and database size.

The Latest Release link will take the user to the release notes for the latest release on GitHub.

Not all clients report current version information or database size.

Local Validators Section

The Local Validators section is intended to show status information for locally-run validators, but it can be used just as well for monitoring any other validator on the network. Data includes:

Individual Ethereum address balance
Individual validator balance and status
Total validators balance
Total Ethereum addresses balance
Balance changes over time and annualized rates of change, for validators, addresses, and the sum of both
Ethereum price in one of over a dozen supported currencies, including: USD, EUR, JPY, GBP, AUD, CAD, CHF, CNY, HKD, NZD, ILS, KRW, MXN, RUB, SGD, TWD, and INR
Total number of monitored validators

Prysm users have an advantage with access to proposal, attestation, and aggregation slot numbers provided directly by the client. Other consensus clients can still monitor validators through client API calls, but less detail is available.

The only limit to the number of validators and Ethereum addresses you can monitor is the number of queries your Ethereum clients can handle.

See known issues for a discussion about validator and address balance changes and how accurate "earnings" calculations are difficult with the tools used.

Host Section

The Host section displays panels related to the system or host on which one or more of the consensus/execution groups are running.

Memory use
CPU use and temperature
Network traffic
Disk use
Open file descriptors
I/O operations per second.

Not every data point/series available is represented by a panel, and some panels show the same data in different ways. Additional panels may be added in the future to cover any important gaps.

Library Sections

See the Library Sections page for screenshots of additional dashboard panels with more limited client support.

Feature Screenshots

Multiple Group Selection

Display multiple staking groups at a time by selecting more than one group in the Group variable drop-down at the top of the dashboard. Most sections will repeat for each selected staking group. The Host section repeats for each selected host. The Sample Dashboard section does not presently work with multiple selections.

Currency Support

The dashboard converts validator and address balances to fiat values using data provided by the CoinGecko API. Supported currencies include: USD, EUR, JPY, GBP, AUD, CAD, CHF, CNY, HKD, NZD, ILS, KRW, MXN, RUB, SGD, TWD, and INR. A few of the currencies are shown below.

Sample Dashboard

Users are encouraged to rearrange, remove, and modify the dashboard panels to meet their specific needs. Not all stakers use all clients and not all clients provide the same data. Stakers using only a single client pair may want to remove all of the panels not supported by their clients. Stakers focused on client internal metrics may want to remove or hide financial performance metrics to meet their needs.

A Sample Dashboard is provided as an example of a reorganization and prioritization of the panels from other dashboard sections.

All panels in the Sample Dashboard section are copies of the panels from the primary sections and can be safely removed or modified without affecting the originals in the primary sections.

Some of the panels in the Sample Dashboard have had columns from the original panels hidden in order to reduce the space required and make room for additional panels.

Supporting Software

Because this dashboard was intended to be an upgrade to the dashboards I previously created, I started with the same Grafana/Prometheus/json_exporter/node_exporter software stack used for the original dashboards. Investigating alternative software solutions (e.g. Loki) was something for a later phase. However, I later added Ethereum Metrics Exporter to the software stack after I realized that it gave me access to some additional API calls inaccessible to json_exporter.

The diagram below shows the communication paths configured between the various software in use.

Metrics for stand-alone validator clients are included for Lighthouse, Prysm, and Nimbus. Support for other stand-alone validators and remote key managers is intended for the future.

Known Issues

Validator and Address Balances

After withdrawals are enabled, calculations of accurate earnings and rates of earning over longer time periods on the dashboard are practically impossible with the software in use.

Prometheus cannot distinguish validator/address deposits from withdrawals. It cannot tell the difference between a bad week of earnings or a good week of earnings where everything has been withdrawn. It only sees a net balance change.

Prometheus is not intended for transactional type data, such as deposits and withdrawals. Using a different type of database could solve this problem, but that was out of scope for this project.

Instead, the dashboard displays the raw balance change and calculates annualized rates of change based on that raw balance change over a period of time. Withdrawals will skew annualized rates of change downward for as long as the withdrawal is still within the lookback period for the calculation.

For example: A withdrawal made within the last hour will negatively skew the annualized rate of return for the one-hour annualized rate of change. After one hour has passed, the one-hour annualized rate of change will again reflect an accurate annualized rate of earnings.

Due to the expectation that earnings will be skimmed from validators more often than once a week, anything greater than a one-day annualized rate of change will likely be inaccurate after withdrawals are enabled.

Queries

There is limited consistency to the names or meanings of client-provided metrics. Dashboard panels must query for many different metric names to cover all of the possible clients. For example, the panel that displays client version information queries Prometheus for metrics with the following names:

lighthouse_info
prysm_version
beacon_teku_version
execution_version
prometheus_build_info
lodestar_version
blackbox_exporter_build_info
node_exporter_build_info
dirk_release
vouch_release
process_start_time_seconds
eth_exe_web3_client_version
eth_con_node_version

All of these metric names appear in a single query to Prometheus and Prometheus spends processing power looking for each, even if you are running a small subset of the clients.

Differing units, subtle semantic differences, and special display needs may require that some clients have completely separate queries for their data. Similar to above, if the client to which that query applies is not in use, the query is still executed and some processing power is wasted.

In practice, running all of the supporting software on the same system as a single monitored client has negligible impact to system resources. See the screenshots in the Hosts section for examples of resource use under the Stake Local dashboard software stack. Only nuc06 is not running all supporting software at this time.

Installation and Configuration

Installation and configuration of the supporting software and dashboard involves many steps. None of steps are particularly challenging, but 5 software products must be installed and configured, and a dozen Prometheus targets must be configured and labeled to get up and running with a single consensus/execution client pair.

An archive containing template Prometheus target configuration files is provided for quick installation. Reasonable default values have been provided in these files, but you may want or need to make changes to customize your installation.

Notes

The dashboard is not a substitute for third-party monitoring services and explorers.

Third-party monitoring services and explorers have more historical data and more advanced querying capabilities than Prometheus provides.
Third-party monitoring services and explorers are more easily accessible from anywhere. Special configurations and precautions are required to access Grafana from anywhere.
Third party monitoring services can alert you when your Internet connection is down, while Grafana would not likely have a way to communicate with you.
If your Internet connection is down, third-party monitoring services can still monitor/alert on your staking status, but will not have access to host data.
Some Prometheus metrics reset to zero after your client has restarted. This will not happen with third party explorers.

Grafana does have some benefits.

Grafana dashboards are highly customizable.
Large variety of client and system performance metrics available.
Nearly all data on the dashboard is from local sources.
Grafana is still locally accessible when your Internet is down.

Though one dashboard panel may support multiple clients, the type of data displayed is not always equivalent or at the same level of detail.

Examples:

The RPC Received per Second panel shows RPC calls broken out by specific statuses for Lighthouse but only simple successes and failures for Geth.
The Validator Count panel shows pending/exiting queue lengths for Prysm but only the active validator count for all other clients.