[Installation] System monitoring with Grafana + Prometheus + Node Export on Linux

This guide use RHEL/Centos/RockyLinux. Other OS are same.

You should have 2 SERVERs:

1. Monitoring server: Install Grafana
2. Monitored server: Install Prometheus

PROMETHEUS INSTALL

(On monitored server)

Create user for this service:

useradd --no-create-home --shell /bin/false prometheus

// Swap to prometheus user:

// su -l prometheus

Create directories:

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Download Prometheus:

sudo dnf install wget -y
cd /opt
sudo wget https://github.com/prometheus/prometheus/releases/download/v2.37.1/prometheus-2.37.1.linux-amd64.tar.gz -O prometheus.tar.gz
sudo tar xvf prometheus.tar.gz
sudo mv prometheus-2.37.1.linux-amd64 prometheus

sudo cp prometheus/prometheus /usr/local/bin/
sudo cp prometheus/promtool /usr/local/bin/

sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

sudo cp -r prometheus/consoles /etc/prometheus
sudo cp -r prometheus/console_libraries /etc/prometheus

sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

sudo rm -rf prometheus*
cd -

Create a config file:

sudo tee /etc/prometheus/prometheus.yml<<EOF
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
EOF

Create a service file, this is an examle use port 9090 for API:

sudo tee /etc/systemd/system/prometheus.service<<EOF
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP \$MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.external-url=

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target
EOF

Start service:

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl status prometheus
sudo systemctl enable prometheus

Open Firewall 9090 port:

sudo firewall-cmd --add-port=9090/tcp --permanent
sudo firewall-cmd --reload

View result on your browser: http://SERVERIP:9090/metrics

EXPORTER INSTALL

(On monitored server)

node_exporter:

Create user for this service:

sudo useradd --no-create-home --shell /bin/false node_exporter

Download and install: https://prometheus.io/download/

cd /opt
sudo wget https://github.com/prometheus/node_exporter/releases/download/v1.4.0-rc.0/node_exporter-1.4.0-rc.0.linux-amd64.tar.gz -O node_exporter.tar.gz

sudo tar xvf node_exporter.tar.gz --strip 1

sudo cp node_exporter /usr/local/bin
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

sudo rm -rf node_exporter*
cd -

Create a service file, use port 9101/tcp:

sudo tee  /etc/systemd/system/node_exporter.service<<EOF
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --web.listen-address=:9101 --collector.systemd --collector.processes

[Install]
WantedBy=multi-user.target
EOF

Start Node Exporter service:

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Update Prometheus config, add new job:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9101']

Restart prometheus:

sudo systemctl restart prometheus

Open firewall port 9101 for API:

sudo firewall-cmd --add-port=9101/tcp --permanent
sudo firewall-cmd --reload

View result on: http://SERVERIP:9101/

Important: If you want more sercurity, don’t add port as above. Please add a firewall zone:

In Monitoring Server SSH terminal:

ip a

Look for LAN IP, something as: 10.*.*.*/*. As my case: 10.5.10.138/26. If your Monitoring Server connect to Monitored Server by public internet, you use public IP instead of LAN IP. –add-source help us to defined who can access by their IP, separate by comma!

In Monitored Server SSH terminal:

sudo firewall-cmd --remove-port={9090/tcp,9101/tcp} --permanent
sudo firewall-cmd --new-zone=monitor-access --permanent
sudo firewall-cmd --reload
sudo firewall-cmd --get-zones
sudo firewall-cmd --zone=monitor-access --add-source={10.5.10.138/26,100.169.147.101} --permanent
sudo firewall-cmd --zone=monitor-access --add-port={9090/tcp,9091/tcp}  --permanent
sudo firewall-cmd --reload
sudo firewall-cmd --list-all --zone=monitor-access

Result:

monitor-access (active)
  target: default
  icmp-block-inversion: no
  interfaces:
  sources: 10.5.10.138/26
  services:
  ports: 9090/tcp 9101/tcp
  protocols:
  forward: no
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Now, you can “curl MonitoredSeverIP:9090” or “curl MonitoredSeverIP:9101” from Mornitoring Server.

If you want to delete zone:

# sudo firewall-cmd --delete-zone=monitor-access --permanent

For all IP: –add-source={0.0.0.0/0}

INSTALL GRAFANA

Install Grafana (add sudo before any command if you are not root):

sudo wget https://dl.grafana.com/enterprise/release/grafana-enterprise-9.1.4-1.x86_64.rpm
sudo rpm -i --nodeps grafana-enterprise-9.1.4-1.x86_64.rpm
sudo systemctl enable grafana-server

Edit file: /etc/grafana/grafana.ini

Bellow [server], config:

• Uncomment “http_port =” then change port as you want. This is the port which you access to Grafana UI.

Option configs:

• Uncomment “http_addr” then change to “http_addr = “You_LAN_IP” or VPN IP if you dont want public this (recommend).
• Uncomment “domain = localhost”, change to your domain if you want (example.com).
• Uncomment “protocol = http” then change to “protocol = https” if you use SSL for your UI page. Set cert_file = path to fullchain.pem, cert_key = path to privakey.pem,…
• enable_gzip = true
• static_root_path = public
• …

My config example:

app_mode = production
instance_name = ${HOSTNAME}
[server]
protocol = https
http_addr = 10.5.11.12
http_port = 8888
domain = monitor.example.com
serve_from_sub_path = true
static_root_path = public
enable_gzip = true
cert_file =/etc/ssl/global/fullchain.pem
cert_key =/etc/ssl/global/privkey.pem


[security]
admin_user = admin
admin_password = mypasswordissomething
cookie_secure = true
allow_embedding = false
strict_transport_security = false
strict_transport_security_max_age_seconds = 86400
strict_transport_security_preload = false
strict_transport_security_subdomains = false
x_content_type_options = true
x_xss_protection = true
content_security_policy = false
content_security_policy_template = """script-src 'self' 'unsafe-eval' 'unsafe-inline' 'strict-dynamic' $NONCE;object-src 'none';font-src 'self';style-src 'self' 'unsafe-inline' blob:;img-src * data:;base-uri 'self';connect-src 'self' grafana.com ws://$ROOT_PATH wss://$ROOT_PATH;manifest-src 'self';media-src 'none';form-action 'self';"""

After save you new config:

sudo service grafana-server start
sudo service grafana-server status

or:

sudo systemctl restart grafana-server
sudo systemctl status grafana-server

Open firewall to access Grafana UI page:

sudo firewall-cmd --new-zone=monitor-access --permanent
sudo firewall-cmd --reload
sudo firewall-cmd --get-zones
sudo firewall-cmd --zone=monitor-access --add-source=10.5.11.1 --permanent
sudo firewall-cmd --zone=monitor-access --add-port=8888/tcp  --permanent
sudo firewall-cmd --reload

Access your Grafana page from Browers: https://monitor.example.com:8888 then login

Go to Configuration-> Data sources-> Prometheus

HTTP URL: http://MonitoredServerIP:9090

then click Save & test button.

Go to Dashboards-> Import -> Upload JSON file (download lastest from here: https://grafana.com/grafana/dashboards/1860-node-exporter-full/?tab=revisions)

Your Dashboard now working