> For the complete documentation index, see [llms.txt](https://docs.bitplanet.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bitplanet.ai/node-operation/operations.md).

# Operations

## Validator Restart Procedures

Understanding when and how to restart your validator safely is crucial for maintaining uptime and avoiding jailing.

### Graceful Restart

* **When to use**: Routine maintenance, configuration changes, or planned updates when you have time.
* **What it does**: Sends a termination signal (SIGTERM) to the evmd process, allowing it to cleanly shut down by:
  * Completing any in-progress operations
  * Flushing data to disk
  * Closing database connections gracefully
* **Expected downtime**: 10-30 seconds

```bash
# Stop the node gracefully
sudo systemctl stop evmd

# Verify process stopped (should return no results)
ps aux | grep evmd

# Start the node
sudo systemctl start evmd

# Monitor logs to ensure clean startup
sudo journalctl -u evmd -f --output cat
```

### Emergency Restart

* **When to use**: When the node is unresponsive, hung, or graceful restart fails.
* **What it does**: Forces immediate termination (SIGKILL) without waiting for cleanup. Use only when graceful restart doesn't work, as it can potentially cause database inconsistencies.
* **Risk**: May require database repair if process was killed during a write operation.

```bash
# Force stop if graceful stop hangs
sudo systemctl kill evmd

# Clear any leftover processes
pkill -9 evmd

# Start node
sudo systemctl start evmd
```

### Post-Restart Validation

* **Purpose**: Verify your validator is running correctly and signing blocks after restart.
* **What to check**:
  * Service is active and running
  * Node is syncing/synced with the network
  * Block height is increasing
  * Validator is participating in consensus (signing blocks)
* **Monitor for**: 5-10 minutes after restart to ensure stable operation.

```bash
# Check service status (should show "active (running)")
sudo systemctl status evmd

# Check sync status (catching_up should be false)
curl -s http://localhost:26657/status | jq '.result.sync_info'

# Check latest block height (should be increasing)
evmd status --node http://localhost:26657 | jq '.sync_info.latest_block_height'

# Check if validator is signing blocks (missed_blocks_counter should not be rapidly increasing)
# Note: Ensure VALCONS is set (see "Checking Validator Status" section above)
evmd query slashing signing-info $VALCONS \
  --node http://localhost:26657
```

## Backup and Recovery

Protecting your validator keys and state is critical. Lost keys mean permanent loss of validator identity and staked funds.

⚠️ **Double Signing Warning:** While backups are essential, NEVER restore your `priv_validator_key.json` to a second node while another node is running with the same key. This will cause double signing and permanent slashing. Always stop the old node before migrating.

### Regular Backup Schedule

**Daily Backups:**

* **Purpose**: Protect critical keys and configuration that define your validator identity.
* **What's backed up**:
  * `priv_validator_key.json` - Your validator signing key (most critical)
  * `node_key.json` - Your P2P network identity
  * `keyring-file/` - Your account keys and addresses
  * `config.toml` and `app.toml` - Your node configuration
* **Why daily**: Keys rarely change, but having recent backups ensures quick recovery if hardware fails.
* **Storage**: Keep backups in multiple locations (external drive, encrypted cloud storage, offline USB).
* **Backup Script:**

```bash
#!/bin/bash
# backup-validator.sh

set -e  # Exit on error

BACKUP_DIR="$HOME/evmd-backups/$(date +%Y-%m-%d)"
mkdir -p "$BACKUP_DIR"

# Backup critical keys (MOST IMPORTANT - without these you lose your validator)
cp $HOME/.evmd/config/priv_validator_key.json "$BACKUP_DIR/"
cp $HOME/.evmd/config/node_key.json "$BACKUP_DIR/"

# Backup keyring (your wallet keys)
cp -r $HOME/.evmd/keyring-file/ "$BACKUP_DIR/"

# Backup configuration (easier than reconfiguring from scratch)
cp $HOME/.evmd/config/config.toml "$BACKUP_DIR/"
cp $HOME/.evmd/config/app.toml "$BACKUP_DIR/"

# Create archive for easy storage
tar -czf "$BACKUP_DIR.tar.gz" -C "$HOME/evmd-backups" "$(basename $BACKUP_DIR)"

# Verify backup was created successfully
if [ ! -f "$BACKUP_DIR.tar.gz" ]; then
    echo "ERROR: Backup failed!" >&2
    exit 1
fi

# Remove old backups (keep last 30 days)
find $HOME/evmd-backups -name "*.tar.gz" -mtime +30 -delete

echo "Backup completed: $BACKUP_DIR.tar.gz"
```

**State Snapshot (Weekly):**

* **Purpose**: Backup blockchain data for faster recovery without re-syncing from genesis.
* **What's backed up**: Complete database state (`~/.evmd/data/`) containing all blockchain data.
* **Why weekly**: State is large (100s of GB), changes constantly, but you can always re-sync if needed.
* **Downtime required**: Yes (5-30 minutes depending on disk speed).
* **State Snapshot Script:**

```bash
#!/bin/bash
# state-snapshot.sh

set -e  # Exit on error

SNAPSHOT_FILE="evmd-state-$(date +%Y-%m-%d).tar.gz"

# Stop node (required for consistent snapshot)
sudo systemctl stop evmd

# Create state snapshot (this can take 10-30 minutes)
tar -czf "$SNAPSHOT_FILE" -C $HOME/.evmd/data .

# Verify snapshot was created successfully
if [ ! -f "$SNAPSHOT_FILE" ]; then
    echo "ERROR: Snapshot failed! Restarting node..." >&2
    sudo systemctl start evmd
    exit 1
fi

# Restart node
sudo systemctl start evmd

echo "Snapshot completed: $SNAPSHOT_FILE"

# Upload to secure storage (e.g., S3, cloud storage)
# aws s3 cp "$SNAPSHOT_FILE" s3://your-bucket/backups/
```

**Note**: State snapshots are optional but dramatically reduce recovery time (hours vs days).

### Recovery Procedures

**Recover from Key Backup:**

* **When to use**:
  * Lost or corrupted keys
  * Moving validator to new hardware (ONLY after stopping old hardware)
  * Disaster recovery
* **What it recovers**: Your validator identity and configuration (does not restore blockchain state).
* **Recovery time**: Keys restore instantly, but node may need to sync (hours to days without state snapshot).

```bash
# Stop node
sudo systemctl stop evmd

# Restore keys (use most recent backup)
cp evmd-backups/<date>/priv_validator_key.json $HOME/.evmd/config/
cp evmd-backups/<date>/node_key.json $HOME/.evmd/config/

# Restore keyring (your wallet keys)
cp -r evmd-backups/<date>/keyring-file/ $HOME/.evmd/

# Set proper permissions (critical for security)
chmod 600 $HOME/.evmd/config/priv_validator_key.json
chmod 600 $HOME/.evmd/config/node_key.json

# Restart node (will sync from network if no state data)
sudo systemctl start evmd
```

**Recover from State Snapshot:**

* **When to use**:
  * Corrupted database
  * Faster recovery after hardware migration
  * Node won't start due to state errors
* **What it does**: Restores blockchain data to a previous point, avoiding full re-sync.
* **Recovery time**: Extraction (10-30 min) + sync from snapshot height to current (minutes to hours).
* **Important**: State snapshots don't include keys. Always restore keys first if moving to new hardware.

```bash
# Stop node
sudo systemctl stop evmd

# Backup current corrupted state (just in case)
mv $HOME/.evmd/data $HOME/.evmd/data.backup

# Restore from snapshot (choose most recent snapshot)
mkdir -p $HOME/.evmd/data
tar -xzf evmd-state-<date>.tar.gz -C $HOME/.evmd/data

# Restart node (will sync remaining blocks from snapshot point)
sudo systemctl start evmd

# Monitor sync (should catch up from snapshot height)
sudo journalctl -u evmd -f
```

## Monitoring Setup

### Prometheus Metrics

Enable Prometheus metrics in `config.toml`:

```toml
[instrumentation]
prometheus = true
prometheus_listen_addr = ":26660"
max_open_connections = 3
namespace = "tendermint"
```

**Prometheus Configuration (prometheus.yml):**

```yaml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'bitplanet-validator'
    static_configs:
      - targets: ['localhost:26660']
        labels:
          instance: 'validator-1'
```

### Key Metrics to Monitor

```bash
# Block height
curl -s http://localhost:26657/status | jq '.result.sync_info.latest_block_height'

# Validator voting power (simplified)
VALOPER=$(evmd keys show validator --bech val -a --keyring-backend file --home $HOME/.evmd)
evmd query staking validator $VALOPER --node http://localhost:26657 | jq '.tokens'

# Missed blocks
VALCONS=$(evmd tendermint show-address --home $HOME/.evmd)
evmd query slashing signing-info $VALCONS --node http://localhost:26657 | jq '.missed_blocks_counter'

# Peer count
curl -s http://localhost:26657/net_info | jq '.result.n_peers'

# Mempool size
curl -s http://localhost:26657/num_unconfirmed_txs | jq '.result.total'
```

## Alerting Configuration

### Alert Script Example

**Purpose**: Automated monitoring script that sends Slack notifications when critical validator issues are detected.

**What it monitors**:

* Validator jailed status (critical)
* Missed block count (warning threshold: 100 blocks)
* Node sync status (out of sync detection)

**Setup Instructions:**

1. Create Slack webhook:
   * Go to <https://api.slack.com/messaging/webhooks>
   * Create a webhook for your workspace
   * Copy the webhook URL
2. Set environment variable:

   ```bash
   # Add to ~/.bashrc or /etc/environment
   export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
   ```
3. Create and configure the script:

```bash
#!/bin/bash
# alert-validator.sh
# Automated validator monitoring and alerting script

set -e  # Exit on error

# Configuration
HOME_DIR="${HOME_DIR:-$HOME/.evmd}"
NODE_URL="${NODE_URL:-http://localhost:26657}"
KEYRING_BACKEND="${KEYRING_BACKEND:-file}"
MISSED_BLOCKS_THRESHOLD=100

# Get validator addresses
VALOPER=$(evmd keys show validator --bech val -a --keyring-backend "$KEYRING_BACKEND" --home "$HOME_DIR" 2>/dev/null)
VALCONS=$(evmd tendermint show-address --home "$HOME_DIR" 2>/dev/null)

# Validate addresses were retrieved
if [ -z "$VALOPER" ] || [ -z "$VALCONS" ]; then
    echo "Error: Failed to retrieve validator addresses" >&2
    exit 1
fi

# SECURITY: Store webhook URL in environment variable, not in script
SLACK_WEBHOOK="${SLACK_WEBHOOK_URL}"

if [ -z "$SLACK_WEBHOOK" ] || [ "$SLACK_WEBHOOK" == "https://hooks.slack.com/services/YOUR/WEBHOOK/URL" ]; then
    echo "Error: SLACK_WEBHOOK_URL environment variable not set" >&2
    exit 1
fi

# Function to send alert
send_alert() {
    local message="$1"
    curl -s -X POST "$SLACK_WEBHOOK" \
        -H 'Content-Type: application/json' \
        -d "{\"text\":\"$message\"}" \
        > /dev/null 2>&1
}

# Check if validator is jailed
JAILED=$(evmd query staking validator "$VALOPER" --node "$NODE_URL" --output json 2>/dev/null | jq -r '.jailed')

if [ "$JAILED" == "true" ]; then
    send_alert "🚨 CRITICAL: Validator is JAILED! Immediate action required."
    echo "$(date): ALERT - Validator jailed"
fi

# Check missed blocks
MISSED=$(evmd query slashing signing-info "$VALCONS" --node "$NODE_URL" --output json 2>/dev/null | jq -r '.missed_blocks_counter')

if [ -n "$MISSED" ] && [ "$MISSED" -gt "$MISSED_BLOCKS_THRESHOLD" ]; then
    send_alert "⚠️  WARNING: Validator missed $MISSED blocks (threshold: $MISSED_BLOCKS_THRESHOLD)"
    echo "$(date): WARNING - Missed $MISSED blocks"
fi

# Check sync status
CATCHING_UP=$(curl -s "$NODE_URL/status" 2>/dev/null | jq -r '.result.sync_info.catching_up')

if [ "$CATCHING_UP" == "true" ]; then
    send_alert "⚠️  WARNING: Node is catching up (not fully synced)"
    echo "$(date): WARNING - Node out of sync"
fi

echo "$(date): Monitoring check completed"
```

**Script Features:**

* **Error handling**: Validates all retrieved values before use
* **Configurable**: Uses environment variables for flexibility
* **Logging**: Prints timestamped events to stdout
* **Security**: Webhook URL stored in environment, not in script
* **Thresholds**: Configurable missed blocks threshold

**Cron Setup:**

```bash
# Make script executable
chmod +x /home/validator/alert-validator.sh

# Add to crontab (run every 5 minutes)
crontab -e

# Add this line:
*/5 * * * * /home/validator/alert-validator.sh >> /var/log/validator-alerts.log 2>&1

# Or for system-wide (add to /etc/cron.d/validator-alerts):
*/5 * * * * validator /home/validator/alert-validator.sh >> /var/log/validator-alerts.log 2>&1
```

**Testing the Script:**

```bash
# Test with environment variable
export SLACK_WEBHOOK_URL="your-webhook-url"
./alert-validator.sh

# Check logs
tail -f /var/log/validator-alerts.log
```

**Alert Severity Levels:**

* 🚨 **CRITICAL**: Validator jailed - requires immediate action
* ⚠️ **WARNING**: High missed blocks or out of sync - investigate soon
* ✅ **INFO**: Regular monitoring checks (logged, not alerted)

## Next Steps

* Review setup procedures in [Setup & Configuration](/node-operation/setup-configuration.md)
* Follow step-by-step procedures in [Runbooks](/node-operation/runbooks.md)
* Troubleshoot common issues in [Troubleshooting](/node-operation/troubleshooting.md)
* Access quick reference in [Additional Resources](/node-operation/additional-resources.md)