> For the complete documentation index, see [llms.txt](https://docs.bitplanet.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bitplanet.ai/node-operation/runbooks.md). # Runbooks ## Overview This section provides step-by-step operational procedures for common validator tasks and emergency scenarios. These runbooks are designed for production validator operations and assume you have basic familiarity with Linux system administration, the Cosmos SDK, and the `evmd` command-line interface. **Important:** Always test procedures in a non-production environment first. For critical operations affecting validator uptime, ensure you have recent backups and a rollback plan. *** ## Runbook 1: Validator Restart **Purpose:** Gracefully restart validator without getting jailed **Prerequisites:** * SSH access to validator * Backup of critical files **Steps:** 1. **Pre-restart checks:** ```bash # Check current status curl -s http://localhost:26657/status | jq '.result.sync_info' # Check validator signing # Note: Set $VALCONS if not already set (see "Checking Validator Status" section) evmd query slashing signing-info $VALCONS --node http://localhost:26657 ``` 2. **Stop validator:** ```bash sudo systemctl stop evmd # Verify stopped ps aux | grep evmd ``` 3. **Perform maintenance:** ```bash # Update binary if needed make install # Verify new version evmd version ``` **Note:** The `make install` command must be run from the Bitplanet repository directory. It builds the binary with the correct version information and installs it to your `$GOPATH/bin` (typically `~/go/bin/evmd`). 4. **Start validator:** ```bash sudo systemctl start evmd ``` 5. **Post-restart validation:** ```bash # Check service status sudo systemctl status evmd # Monitor logs for 2 minutes sudo journalctl -u evmd -f --output cat # Verify syncing curl -s http://localhost:26657/status | jq '.result.sync_info.catching_up' # Check signing (VALCONS should already be set from "Checking Validator Status" section) sleep 60 evmd query slashing signing-info $VALCONS --node http://localhost:26657 ``` **Expected Duration:** 2-5 minutes **Rollback Plan:** Restore from backup if new binary fails ## Runbook 2: Consensus Failure Recovery **Purpose:** Recover from consensus failure and rejoin network **Symptoms:** * Node not producing blocks * Persistent errors in logs * Validator jailed **Steps:** 1. **Immediate actions:** ```bash # Stop node sudo systemctl stop evmd # Backup current state cp -r $HOME/.evmd/data $HOME/.evmd/data.backup-$(date +%Y%m%d-%H%M%S) ``` 2. **Diagnose issue:** ```bash # Check last logs sudo journalctl -u evmd --no-pager | tail -100 # Check database integrity evmd tendermint unsafe-reset-all --home $HOME/.evmd --dry-run ``` 3. **Recovery options:** **Option A: Reset Tendermint state (if app state is intact)** ```bash evmd tendermint unsafe-reset-all --home $HOME/.evmd sudo systemctl start evmd ``` **What this does:** Removes all blockchain data (blocks, consensus state, transaction index) but keeps your configuration and keys. The node will resync from the network starting from genesis or a state sync snapshot. This is useful when consensus state is corrupted but your keys and configuration are fine. **Option B: Restore from snapshot (if corruption is severe)** ```bash rm -rf $HOME/.evmd/data tar -xzf latest-snapshot.tar.gz -C $HOME/.evmd/data sudo systemctl start evmd ``` **Option C: Resync from genesis (last resort)** ```bash evmd tendermint unsafe-reset-all --home $HOME/.evmd # Ensure genesis.json is correct sudo systemctl start evmd ``` 4. **Monitor recovery:** ```bash # Watch sync progress watch -n 5 'curl -s http://localhost:26657/status | jq ".result.sync_info"' # Monitor logs sudo journalctl -u evmd -f ``` 5. **Re-enable validator (if was jailed):** ```bash # Wait until fully synced # Then unjail evmd tx slashing unjail --from validator --keyring-backend file --home $HOME/.evmd --chain-id 9001 --yes ``` **Expected Duration:** 30 minutes to several hours (depending on sync method) ## Runbook 3: Reward Claim Failure Resolution **Purpose:** Troubleshoot and resolve issues with claiming validator rewards **Symptoms:** * Transaction fails when claiming rewards * "insufficient funds" error * Rewards not visible **Steps:** 1. **Check rewards availability:** ```bash # Query commission rewards evmd query distribution commission $(evmd keys show validator --bech val -a --keyring-backend file --home $HOME/.evmd) \ --node http://localhost:26657 # Query delegation rewards evmd query distribution rewards $(evmd keys show validator -a --keyring-backend file --home $HOME/.evmd) \ --node http://localhost:26657 ``` 2. **Check account balance for fees:** ```bash evmd query bank balances $(evmd keys show validator -a --keyring-backend file --home $HOME/.evmd) \ --node http://localhost:26657 ``` 3. **Attempt reward withdrawal:** ```bash # Withdraw commission evmd tx distribution withdraw-rewards $(evmd keys show validator --bech val -a --keyring-backend file --home $HOME/.evmd) \ --commission \ --from validator \ --keyring-backend file \ --home $HOME/.evmd \ --chain-id 9001 \ --gas auto \ --gas-adjustment 1.5 \ --gas-prices 10000000bplcoin \ --yes ``` **Note:** The `--commission` flag withdraws validator commission only. To withdraw delegation rewards (your own staked tokens' rewards), omit the `--commission` flag. You can withdraw both in separate transactions or combine them. 4. **If transaction fails:** ```bash # Increase gas limit evmd tx distribution withdraw-rewards $(evmd keys show validator --bech val -a --keyring-backend file --home $HOME/.evmd) \ --commission \ --from validator \ --keyring-backend file \ --home $HOME/.evmd \ --chain-id 9001 \ --gas 300000 \ --gas-prices 10000000bplcoin \ --yes ``` 5. **Verify withdrawal:** ```bash # Check updated balance evmd query bank balances $(evmd keys show validator -a --keyring-backend file --home $HOME/.evmd) \ --node http://localhost:26657 ``` **Expected Duration:** 5-10 minutes ## Runbook 4: Emergency Validator Shutdown **Purpose:** Emergency procedure for immediate validator shutdown **When to Use:** * Security breach detected * Double signing risk * Critical infrastructure failure **Steps:** 1. **Immediate shutdown:** ```bash # Stop service immediately sudo systemctl stop evmd # Kill any remaining processes sudo pkill -9 evmd # Verify all processes stopped ps aux | grep evmd ``` 2. **Secure validator keys:** ```bash # Backup keys to secure location tar -czf emergency-backup-$(date +%Y%m%d-%H%M%S).tar.gz \ $HOME/.evmd/config/priv_validator_key.json \ $HOME/.evmd/config/node_key.json \ $HOME/.evmd/keyring-file/ # Move to secure offline storage # DO NOT leave on the server ``` **Critical:** The `priv_validator_key.json` is your validator's consensus key. If this is compromised, an attacker could double-sign blocks using your validator identity. Always store backups encrypted and in multiple secure locations (hardware security module, encrypted USB drive, secure cloud storage with strong encryption). 3. **Prevent automatic restart:** ```bash sudo systemctl disable evmd ``` 4. **Document incident:** ```bash # Create incident report cat > incident-report-$(date +%Y%m%d-%H%M%S).txt <