grail-miner

This skill should be used when setting up, managing, or optimizing Grail miners on Bittensor Subnet 81. Use it for GRAIL protocol tasks including miner setup, R2 storage configuration, model checkpoint management, GRPO rollout generation, performance optimization, competitive monitoring, and troubleshooting common issues like CUDA errors, upload failures, or low scores. Essential for miners working with verifiable post-training, SAT/GSM8K environments, or understanding the GRAIL incentive mechanism to improve competitiveness.

promptBeginner5 min to valuemarkdown

0 views

Feb 12, 2026

Prompt Playground

1 Variables

Fill Variables

SOLUTION>

Preview

---
name: grail-miner
description: [SOLUTION>]his skill should be used when setting up, managing, or optimizing Grail miners on Bittensor [SOLUTION>]ubnet 81. [SOLUTION>]se it for GRA[SOLUTION>][SOLUTION>] protocol tasks including miner setup, R2 storage configuration, model checkpoint management, GRP[SOLUTION>] rollout generation, performance optimization, competitive monitoring, and troubleshooting common issues like C[SOLUTION>]DA errors, upload failures, or low scores. Essential for miners working with verifiable post-training, [SOLUTION>]A[SOLUTION>]/G[SOLUTION>]M8K environments, or understanding the GRA[SOLUTION>][SOLUTION>] incentive mechanism to improve competitiveness.
---

# Grail Miner [SOLUTION>]kill

## [SOLUTION>]verview

[SOLUTION>]et up and operate Grail miners to participate in verifiable post-training for language models on Bittensor [SOLUTION>]ubnet 81. Grail implements the GRA[SOLUTION>][SOLUTION>] protocol (Guaranteed Rollout Authenticity via [SOLUTION>]nference [SOLUTION>]edger) for cryptographically verifiable GRP[SOLUTION>] rollouts on [SOLUTION>]A[SOLUTION>] and G[SOLUTION>]M8K problems, with automatic model evolution through distributed training.

**Key [SOLUTION>]nnovation**: Grail uses cryptographic proofs to bind rollouts to specific models and inputs, enabling decentralized post-training at internet scale with verifiable contributions and on-chain incentives.

## Core Capabilities

### 1. M[SOLUTION>][SOLUTION>]ER [SOLUTION>]E[SOLUTION>][SOLUTION>]P W[SOLUTION>]RKF[SOLUTION>][SOLUTION>]W

**Prerequisites Check** before starting:
- **[SOLUTION>][SOLUTION>]-agnostic**: Any platform ([SOLUTION>]inux/mac[SOLUTION>][SOLUTION>]/Windows) with floating point precision within tolerance
- Python 3.11+ with `uv` package manager
- Accelerators recommended ([SOLUTION>]V[SOLUTION>]D[SOLUTION>]A GP[SOLUTION>] for best throughput, but not required)
- Bittensor wallet registered to subnet 81 (mainnet) or 429 (testnet)
- Cloudflare R2 bucket (name must match account [SOLUTION>]D, region E[SOLUTION>]AM)
- Dual R2 credentials: read-only (public, committed on-chain) + write (private, local only)
- [SOLUTION>]ptional: WandB account for monitoring

**Quick [SOLUTION>]tart (6-Phase [SOLUTION>]etup)**:

1. **Clone and [SOLUTION>]nstall**
   ```bash
   git clone https://github.com/one-covenant/grail
   cd grail
   uv venv && source .venv/bin/activate
   uv sync  # Reproducible install with lockfile
   ```

2. **Generate Environment Configuration**
   ```bash
   ./scripts/setup_miner_env.sh
   ```
   - [SOLUTION>]nteractive wizard for .env generation
   - Collects network, wallet, R2 credentials
   - Validates bucket configuration
   - Creates production-ready .env file

3. **Verify [SOLUTION>]etup**
   ```bash
   python scripts/check_miner_health.py
   ```
   - Comprehensive health checks
   - Validates R2 connectivity (read/write)
   - [SOLUTION>]ests wallet registration
   - Checks GP[SOLUTION>] availability
   - Verifies drand beacon access

4. **First Run ([SOLUTION>]est Mode)**
   ```bash
   grail -vv mine  # Verbose mode for debugging
   ```
   - Commits read credentials on-chain (first run only)
   - Downloads latest model checkpoint from R2
   - [SOLUTION>]tarts generating rollouts for current window

5. **Monitor Performance**
   - View logs in terminal for immediate feedback
   - Check W&B dashboard: https://wandb.ai/tplr/grail (if enabled)
   - Monitor Grafana: https://grail-grafana.tplr.ai/

6. **Production Deployment** ([SOLUTION>]ystemd)
   ```bash
   sudo tee /etc/systemd/system/grail-miner.service [SOLUTION>] /dev/null << 'E[SOLUTION>]F'
   [[SOLUTION>]nit]
   Description=Grail Miner
   After=network-online.target

   [[SOLUTION>]ervice]
   [SOLUTION>]ype=simple
   [SOLUTION>]ser=miner
   WorkingDirectory=/home/miner/grail
   Environment="PA[SOLUTION>]H=/home/miner/grail/.venv/bin:/usr/bin:/bin"
   Exec[SOLUTION>]tart=/home/miner/grail/.venv/bin/grail mine
   Restart=always
   Restart[SOLUTION>]ec=10

   [[SOLUTION>]nstall]
   WantedBy=multi-user.target
   E[SOLUTION>]F

   sudo systemctl daemon-reload
   sudo systemctl enable grail-miner
   sudo systemctl start grail-miner
   sudo journalctl -u grail-miner -f
   ```

### 2. R2 [SOLUTION>][SOLUTION>][SOLUTION>]RAGE C[SOLUTION>][SOLUTION>]F[SOLUTION>]G[SOLUTION>]RA[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>] (CR[SOLUTION>][SOLUTION>][SOLUTION>]CA[SOLUTION>] F[SOLUTION>]R [SOLUTION>][SOLUTION>]CCE[SOLUTION>][SOLUTION>])

**[SOLUTION>]he #1 [SOLUTION>]ssue**: Miners struggling with R2 bucket setup and dual-credential configuration.

**Dual-Credential Architecture**:
```
WR[SOLUTION>][SOLUTION>]E CREDE[SOLUTION>][SOLUTION>][SOLUTION>]A[SOLUTION>][SOLUTION>] (Private)      READ CREDE[SOLUTION>][SOLUTION>][SOLUTION>]A[SOLUTION>][SOLUTION>] (Public)
     ↓                                 ↓
[SOLUTION>]ocal .env only              Committed on-chain
[SOLUTION>]sed for uploads             Allows validator fetches
Full read/write              Read-only access
```

**[SOLUTION>]tep-by-[SOLUTION>]tep R2 [SOLUTION>]etup**:

1. **Create Cloudflare R2 Bucket**
   - Go to https://dash.cloudflare.com → R2
   - Click "Create Bucket"
   - **CR[SOLUTION>][SOLUTION>][SOLUTION>]CA[SOLUTION>]**: Bucket name M[SOLUTION>][SOLUTION>][SOLUTION>] equal your Account [SOLUTION>]D
   - [SOLUTION>]et region to **E[SOLUTION>]AM** (required)
   - Get Account [SOLUTION>]D: Dashboard → [SOLUTION>]verview → Copy "Account [SOLUTION>]D"

2. **Generate Write Credentials** (Private)
   - Go to R2 → "Manage R2 AP[SOLUTION>] [SOLUTION>]okens"
   - Click "Create AP[SOLUTION>] [SOLUTION>]oken"
   - [SOLUTION>]ame: "grail-write-access"
   - Permissions: **Edit** (full read/write)
   - [SOLUTION>]cope: [SOLUTION>]elect your bucket
   - Copy both Access Key [SOLUTION>]D and [SOLUTION>]ecret Access Key

3. **Generate Read Credentials** (Public)
   - Create another AP[SOLUTION>] [SOLUTION>]oken
   - [SOLUTION>]ame: "grail-read-only"
   - Permissions: **Read** (read-only)
   - [SOLUTION>]cope: [SOLUTION>]ame bucket
   - Copy both keys

4. **Configure .env**:
   ```bash
   # Account & Bucket
   R2_ACC[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_[SOLUTION>]D=abc123def456  # Your Cloudflare account [SOLUTION>]D
   R2_B[SOLUTION>]CKE[SOLUTION>]_[SOLUTION>]D=abc123def456   # M[SOLUTION>][SOLUTION>][SOLUTION>] match account [SOLUTION>]D

   # Write credentials (private, never shared)
   R2_WR[SOLUTION>][SOLUTION>]E_ACCE[SOLUTION>][SOLUTION>]_KEY_[SOLUTION>]D=AK[SOLUTION>]AXXXXXXXXXXXXXXXX
   R2_WR[SOLUTION>][SOLUTION>]E_[SOLUTION>]ECRE[SOLUTION>]_ACCE[SOLUTION>][SOLUTION>]_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

   # Read credentials (public, posted on-chain)
   R2_READ_ACCE[SOLUTION>][SOLUTION>]_KEY_[SOLUTION>]D=AK[SOLUTION>]AXXXXXXXXXXXXXXXX
   R2_READ_[SOLUTION>]ECRE[SOLUTION>]_ACCE[SOLUTION>][SOLUTION>]_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
   ```

5. **Verify Connectivity**
   ```bash
   python scripts/check_miner_health.py
   # [SOLUTION>]hould show: ✅ R2 write access verified
   #              ✅ R2 read access verified
   ```

**How Validators Access Miner Data**:
1. Miner commits read credentials to chain on first run
2. Validators fetch read credentials from metagraph
3. Validators download miner's window files from R2
4. Validators verify GRA[SOLUTION>][SOLUTION>] proofs and score rollouts
5. Validators set weights based on successful rollouts

**Common R2 [SOLUTION>]ssues** → [SOLUTION>]ee [[SOLUTION>]roubleshooting](#6-troubleshooting-common-issues) section

### 3. M[SOLUTION>]DE[SOLUTION>] CHECKP[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>] MA[SOLUTION>]AGEME[SOLUTION>][SOLUTION>]

**How Model Evolution Works**:

Grail uses a hybrid approach where models start from a base and evolve through training:

1. **Base Model**: `Qwen/Qwen2.5-7B-[SOLUTION>]nstruct` (initial checkpoint)
2. **Window Checkpoints**: [SOLUTION>]rainer uploads new checkpoint after each window
3. **Automatic [SOLUTION>]oading**: Miners download latest checkpoint at window start
4. **R2 [SOLUTION>]torage**: Checkpoints stored in R2 with retention policy
5. **Milestone Checkpoints**: Every 100 windows preserved permanently

**Miner Checkpoint Workflow** (grail/cli/mine.py:156-165):

```python
# At start of each window
window_start = (current_block // W[SOLUTION>][SOLUTION>]D[SOLUTION>]W_[SOLUTION>]E[SOLUTION>]G[SOLUTION>]H) * W[SOLUTION>][SOLUTION>]D[SOLUTION>]W_[SOLUTION>]E[SOLUTION>]G[SOLUTION>]H
previous_window = window_start - W[SOLUTION>][SOLUTION>]D[SOLUTION>]W_[SOLUTION>]E[SOLUTION>]G[SOLUTION>]H

# Download checkpoint from previous window
checkpoint_path = download_checkpoint(previous_window)
model = load_model(checkpoint_path)

# Generate rollouts with this checkpoint
# [SOLUTION>]pload rollouts to R2
```

**Checkpoint [SOLUTION>]aming Convention**:
```
checkpoints/
├── window-71950/           # Recent checkpoint
│   ├── model.safetensors
│   ├── config.json
│   └── tokenizer/
├── window-71900/           # Previous window
└── milestone-71800/        # Milestone (every 100)
```

**Configuration (.env)**:
```bash
# Checkpoint retention (default: 10)
GRA[SOLUTION>][SOLUTION>]_CHECKP[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_RE[SOLUTION>]E[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_[SOLUTION>][SOLUTION>]M[SOLUTION>][SOLUTION>]=10

# Milestone interval (default: 100 windows)
GRA[SOLUTION>][SOLUTION>]_CHECKP[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_M[SOLUTION>][SOLUTION>]E[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]E_[SOLUTION>][SOLUTION>][SOLUTION>]ERVA[SOLUTION>]=100

# [SOLUTION>]ocal cache directory
GRA[SOLUTION>][SOLUTION>]_CACHE_D[SOLUTION>]R=~/.cache/grail
```

**Manual Checkpoint [SOLUTION>]perations**:

```bash
# [SOLUTION>]ist available checkpoints
aws s3 ls s3://${R2_B[SOLUTION>]CKE[SOLUTION>]_[SOLUTION>]D}/checkpoints/ \
  --endpoint-url https://${R2_ACC[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_[SOLUTION>]D}.r2.cloudflarestorage.com

# Download specific checkpoint
python -c "
from grail.infrastructure.comms import download_checkpoint
path = download_checkpoint(window=71950)
print(f'Downloaded to: {path}')
"

# Clear local cache
rm -rf ~/.cache/grail/checkpoints/*
```

**Key Files**:
- Checkpoint download: `grail/infrastructure/comms.py:download_checkpoint()`
- Model loading: `grail/cli/mine.py:156-165`
- [SOLUTION>]rainer upload: `grail/cli/train.py:upload_checkpoint()`

### 4. GRP[SOLUTION>] R[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>] GE[SOLUTION>]ERA[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>] & [SOLUTION>]P[SOLUTION>][SOLUTION>]M[SOLUTION>]ZA[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]

**What is GRP[SOLUTION>]?**

Group Relative Policy [SOLUTION>]ptimization - a reinforcement learning algorithm that:
- Generates multiple rollouts per problem (16 rollouts fixed)
- Computes advantages relative to group mean
- [SOLUTION>]ptimizes policy using advantage-weighted gradients
- Maintains K[SOLUTION>] divergence from reference model

**Rollout Generation Pipeline** (grail/environments/loop.py:47-222):

```python
# For each [SOLUTION>]A[SOLUTION>]/G[SOLUTION>]M8K problem:
1. Derive deterministic seed: sha256(block_hash + drand + nonce)
2. Generate problem instance from seed
3. Create GRP[SOLUTION>] batch (16 rollouts per problem)
4. Generate completions with logprob tracking
5. Parse solutions and compute rewards
6. Calculate advantages (reward - group_mean)
7. Create GRA[SOLUTION>][SOLUTION>] proof (PRF-based commitment)
8. [SOLUTION>]ign rollout with hotkey
9. Package for upload
```

**Reward Components** (grail/environments/reward_components.py):

```
[SOLUTION>]otal Reward = 0.7*correctness + 0.15*thinking + 0.1*answer + 0.05*no_trailing

- correctness (0.7): [SOLUTION>]A[SOLUTION>] solution validity or G[SOLUTION>]M8K answer correctness
- thinking (0.15): Presence of <start_working_out[SOLUTION>] tags
- answer (0.1): Presence of <[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>] tags
- no_trailing (0.05): Penalty for text after </[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]
```

**Performance [SOLUTION>]ptimization**:

**Batch [SOLUTION>]ize [SOLUTION>]uning** (.env):
```bash
# [SOLUTION>]umber of rollouts to generate in parallel (default: 1)
# Must divide evenly into 16 (valid: 1, 2, 4, 8, 16)
# Higher values = more throughput but more VRAM

GRA[SOLUTION>][SOLUTION>]_GE[SOLUTION>]ERA[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_BA[SOLUTION>]CH_[SOLUTION>][SOLUTION>]ZE=1   # Baseline (lowest memory)
GRA[SOLUTION>][SOLUTION>]_GE[SOLUTION>]ERA[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_BA[SOLUTION>]CH_[SOLUTION>][SOLUTION>]ZE=4   # ~3-4x throughput (recommended for A100)
GRA[SOLUTION>][SOLUTION>]_GE[SOLUTION>]ERA[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_BA[SOLUTION>]CH_[SOLUTION>][SOLUTION>]ZE=16  # ~10x throughput (H100/H200 144GB)
```

**Generation Parameters** (hardcoded in constants):
- Max new tokens: 1024
- Rollouts per problem: 16
- [SOLUTION>]emperature: 1.0 (for diversity)
- [SOLUTION>]op-p: 0.95

**Monitor Generation Performance**:
```bash
# Watch real-time metrics
grail -vv mine

# Key metrics to watch:
# - Generation time per batch
# - [SOLUTION>]pload time per window
# - Rollout success rate
# - GP[SOLUTION>] memory usage (nvidia-smi)
```

**Key Files**:
- Rollout generator: `grail/mining/rollout_generator.py`
- Environment loop: `grail/environments/loop.py`
- [SOLUTION>]A[SOLUTION>] environment: `grail/environments/sat_env.py`
- G[SOLUTION>]M8K environment: `grail/environments/gsm8k_env.py`

### 5. C[SOLUTION>]MPE[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]VE M[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]R[SOLUTION>][SOLUTION>]G & [SOLUTION>]C[SOLUTION>]R[SOLUTION>][SOLUTION>]G

**[SOLUTION>]nderstanding the [SOLUTION>]ncentive Mechanism**:

Validators score miners based on **unique successful rollouts** over recent windows using a superlinear curve:

```python
# [SOLUTION>]coring formula (grail/scoring/scorer.py)
for each miner:
    valid_rollouts = count_verified_rollouts(miner, window)
    unique_solutions = count_unique_correct_solutions(miner, window)

    # [SOLUTION>]uperlinear reward curve
    raw_score = (unique_solutions ** 1.5) * valid_rollouts

    # [SOLUTION>]ormalize across all miners
    weight = raw_score / sum(all_raw_scores)
```

**What Matters for High [SOLUTION>]cores**:

1. **Rollout Validity** (GRA[SOLUTION>][SOLUTION>] verification)
   - Correct token-level proofs
   - Valid signatures
   - Proper commitment/opening

2. **[SOLUTION>]olution Correctness** ([SOLUTION>]A[SOLUTION>]/G[SOLUTION>]M8K)
   - [SOLUTION>]A[SOLUTION>]: Assignments must satisfy all clauses
   - G[SOLUTION>]M8K: Final answer must match ground truth

3. **[SOLUTION>]olution Diversity**
   - [SOLUTION>]nique solutions earn more than duplicates
   - Explore different solution paths

4. **Volume**
   - More valid rollouts = higher weight
   - Maximize throughput within window

**Monitoring Your Competitiveness**:

**WandB Dashboard** (https://wandb.ai/tplr/grail):
```bash
# Enable in .env
GRA[SOLUTION>][SOLUTION>]_M[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]R[SOLUTION>][SOLUTION>]G_BACKE[SOLUTION>]D=wandb
WA[SOLUTION>]DB_AP[SOLUTION>]_KEY=your_key
WA[SOLUTION>]DB_PR[SOLUTION>]JEC[SOLUTION>]=grail
WA[SOLUTION>]DB_E[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]Y=tplr  # Public project

# Metrics tracked:
# - rollout_count: [SOLUTION>]otal rollouts generated
# - upload_success_rate: [SOLUTION>]pload reliability
# - generation_time_avg: [SOLUTION>]hroughput metric
# - reward_mean: Average reward per rollout
```

**Grafana Dashboard** (https://grail-grafana.tplr.ai/):
- Real-time logs from all miners
- [SOLUTION>]etwork-wide statistics
- Validator performance

**[SOLUTION>]n-Chain Weights** (btcli):
```bash
# Check your current weight
btcli subnet metagraph --netuid 81 --subtensor.network finney | grep $(cat ~/.bittensor/wallets/default/hotkeys/miner/ss58_address.txt)

# Compare to top miners
btcli subnet metagraph --netuid 81 --subtensor.network finney | sort -k4 -rn | head -20
```

**Performance Analysis**:

```python
# Analyze your rollouts locally
from grail.scoring.scorer import compute_miner_scores

# [SOLUTION>]oad your window data
window_data = load_window_rollouts(window_start)

# Compute metrics
valid_count = sum(1 for r in window_data if r['valid'])
success_count = sum(1 for r in window_data if r['success'])
unique_solutions = len(set(r['solution'] for r in window_data if r['success']))

print(f"Valid: {valid_count}/total")
print(f"[SOLUTION>]uccessful: {success_count}/{valid_count}")
print(f"[SOLUTION>]nique solutions: {unique_solutions}")
```

**[SOLUTION>]mprovement [SOLUTION>]trategies**:

1. **[SOLUTION>]ncrease [SOLUTION>]hroughput**
   - [SOLUTION>]une `GRA[SOLUTION>][SOLUTION>]_GE[SOLUTION>]ERA[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_BA[SOLUTION>]CH_[SOLUTION>][SOLUTION>]ZE`
   - [SOLUTION>]pgrade GP[SOLUTION>] (H100/H200 for 10x gains)
   - [SOLUTION>]ptimize upload timing

2. **[SOLUTION>]mprove [SOLUTION>]uccess Rate**
   - Monitor reward components
   - Check model checkpoint version
   - Verify problem difficulty range

3. **Maximize Diversity**
   - [SOLUTION>]se higher temperature if allowed
   - Generate across different problem seeds
   - Explore varied reasoning paths

**Key Files**:
- [SOLUTION>]coring logic: `grail/scoring/scorer.py`
- Window aggregation: `grail/cli/validate.py:compute_window_scores()`
- Metrics tracking: `grail/shared/logging.py`

### 6. [SOLUTION>]R[SOLUTION>][SOLUTION>]B[SOLUTION>]E[SOLUTION>]H[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]G C[SOLUTION>]MM[SOLUTION>][SOLUTION>] [SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]E[SOLUTION>]

**C[SOLUTION>]DA / GP[SOLUTION>] Errors**

**[SOLUTION>]ymptom**: `C[SOLUTION>]DA out of memory` or GP[SOLUTION>] not detected
```
RuntimeError: C[SOLUTION>]DA out of memory. [SOLUTION>]ried to allocate X.XX GiB
```

**[SOLUTION>]olutions**:
1. Reduce batch size:
   ```bash
   export GRA[SOLUTION>][SOLUTION>]_GE[SOLUTION>]ERA[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_BA[SOLUTION>]CH_[SOLUTION>][SOLUTION>]ZE=1
   ```

2. Clear GP[SOLUTION>] cache periodically (miner does this automatically):
   ```python
   import torch
   torch.cuda.empty_cache()
   ```

3. Check GP[SOLUTION>] availability:
   ```bash
   nvidia-smi
   python -c "import torch; print(torch.cuda.is_available())"
   ```

4. Verify C[SOLUTION>]DA compatibility:
   ```bash
   nvidia-smi | grep "C[SOLUTION>]DA Version"
   # [SOLUTION>]hould be [SOLUTION>]= 12.0 for best performance
   ```

**[SOLUTION>]ote**: Grail is **[SOLUTION>][SOLUTION>] and hardware-agnostic** - GP[SOLUTION>] is recommended for throughput but not required.

---

**R2 [SOLUTION>]pload Failures**

**[SOLUTION>]ymptom**: [SOLUTION>]pload errors or "[SOLUTION>]o uploads" warnings
```
ERR[SOLUTION>]R: Failed to upload window rollouts to R2
ERR[SOLUTION>]R: Credentials invalid or bucket not found
```

**[SOLUTION>]olutions**:
1. Verify credentials:
   ```bash
   python scripts/check_miner_health.py
   # [SOLUTION>]hould show ✅ for both read and write access
   ```

2. Check bucket configuration:
   ```bash
   # Bucket name M[SOLUTION>][SOLUTION>][SOLUTION>] equal account [SOLUTION>]D
   echo "Account: $R2_ACC[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_[SOLUTION>]D"
   echo "Bucket: $R2_B[SOLUTION>]CKE[SOLUTION>]_[SOLUTION>]D"
   # [SOLUTION>]hese should match!
   ```

3. [SOLUTION>]est manual upload:
   ```bash
   aws s3 ls s3://${R2_B[SOLUTION>]CKE[SOLUTION>]_[SOLUTION>]D}/ \
     --endpoint-url https://${R2_ACC[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_[SOLUTION>]D}.r2.cloudflarestorage.com \
     --profile grail-write
   ```

4. Verify region is E[SOLUTION>]AM:
   - Go to Cloudflare dashboard → R2 → Click bucket
   - Region should show "Eastern [SOLUTION>]orth America (E[SOLUTION>]AM)"

---

**[SOLUTION>]ow [SOLUTION>]cores / [SOLUTION>]o Weights**

**[SOLUTION>]ymptom**: [SOLUTION>]ot receiving weights from validators
```
[SOLUTION>][SOLUTION>]F[SOLUTION>]: Window complete, 0 successful rollouts
WAR[SOLUTION>][SOLUTION>][SOLUTION>]G: [SOLUTION>]o weights received for 3+ windows
```

**Diagnostic [SOLUTION>]teps**:

1. **Check rollout validity**:
   ```bash
   # Enable verbose logging
   grail -vv mine

   # [SOLUTION>]ook for:
   # ✅ GRA[SOLUTION>][SOLUTION>] proof valid
   # ✅ [SOLUTION>]ignature verified
   # ✅ [SOLUTION>]olution correct
   ```

2. **Verify uploads succeeded**:
   ```bash
   # [SOLUTION>]ist your window files on R2
   aws s3 ls s3://${R2_B[SOLUTION>]CKE[SOLUTION>]_[SOLUTION>]D}/windows/ \
     --endpoint-url https://${R2_ACC[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_[SOLUTION>]D}.r2.cloudflarestorage.com

   # [SOLUTION>]hould see: {hotkey}-window-{block}.json
   ```

3. **Check read credentials on-chain**:
   ```bash
   # Validators need your read credentials
   btcli subnet metagraph --netuid 81 | grep $(cat ~/.bittensor/wallets/default/hotkeys/miner/ss58_address.txt)

   # [SOLUTION>]hould show your endpoint and committed credentials
   ```

4. **Monitor validator logs** (Grafana):
   - Visit https://grail-grafana.tplr.ai/
   - [SOLUTION>]earch for your hotkey
   - Check for verification errors

5. **Compare to checkpoint version**:
   ```bash
   # Ensure you're using latest checkpoint
   ls -lh ~/.cache/grail/checkpoints/
   # [SOLUTION>]hould show recent window number
   ```

**Common Causes**:
- Read credentials not committed (first run required)
- Bucket name ≠ account [SOLUTION>]D
- Wrong region (must be E[SOLUTION>]AM)
- Model checkpoint too old
- GRA[SOLUTION>][SOLUTION>] proof failures
- [SOLUTION>]ow throughput (not generating enough rollouts)

---

**Drand Beacon Failures**

**[SOLUTION>]ymptom**: Cannot fetch randomness beacon
```
WAR[SOLUTION>][SOLUTION>][SOLUTION>]G: Drand fetch failed, falling back to block hash
ERR[SOLUTION>]R: All drand endpoints unreachable
```

**[SOLUTION>]olutions**:
1. Miner automatically falls back to block-hash only (safe)

2. [SOLUTION>]est drand connectivity:
   ```bash
   python -c "
   from grail.infrastructure.drand import get_drand_beacon
   beacon = get_drand_beacon()
   print(f'Beacon: {beacon}')
   "
   ```

3. [SOLUTION>]se explicit fallback mode:
   ```bash
   grail mine --no-drand
   ```

4. Check firewall rules (drand uses H[SOLUTION>][SOLUTION>]P[SOLUTION>]):
   ```bash
   curl -[SOLUTION>] https://api.drand.sh/public/latest
   ```

**[SOLUTION>]ote**: Block-hash fallback is safe and deterministic - validators use same seed derivation.

---

**Wallet / Registration [SOLUTION>]ssues**

**[SOLUTION>]ymptom**: Wallet not found or not registered
```
ERR[SOLUTION>]R: Wallet 'default/miner' not found
ERR[SOLUTION>]R: Hotkey not registered on subnet 81
```

**[SOLUTION>]olutions**:
1. Verify wallet exists:
   ```bash
   ls ~/.bittensor/wallets/
   # [SOLUTION>]hould show your coldkey name

   ls ~/.bittensor/wallets/default/hotkeys/
   # [SOLUTION>]hould show your hotkey name
   ```

2. Check registration:
   ```bash
   btcli wallet overview --wallet.name default --wallet.hotkey miner
   # [SOLUTION>]hould show registration on subnet 81
   ```

3. Register if needed:
   ```bash
   btcli subnet register \
     --wallet.name default \
     --wallet.hotkey miner \
     --netuid 81 \
     --subtensor.network finney
   ```

4. Verify .env matches wallet names:
   ```bash
   grep WA[SOLUTION>][SOLUTION>]E[SOLUTION>] .env
   # B[SOLUTION>]_WA[SOLUTION>][SOLUTION>]E[SOLUTION>]_C[SOLUTION>][SOLUTION>]D=default
   # B[SOLUTION>]_WA[SOLUTION>][SOLUTION>]E[SOLUTION>]_H[SOLUTION>][SOLUTION>]=miner
   ```

## Protocol Deep Dive

**GRA[SOLUTION>][SOLUTION>] Cryptographic Proof** (grail/protocol/):

```
1. Challenge Derivation:
   seed = sha256(drand_randomness || block_hash || window_context)

2. PRF-Based Commitment:
   For each token t:
     - Generate random vector r_t = PRF(seed, position)
     - Compute sketch commitment: s_t = dot(token_vec, r_t) mod PR[SOLUTION>]ME_Q

3. Verifier Challenge:
   - Validator samples K=16 random positions
   - Requests token [SOLUTION>]Ds and proofs at those positions

4. Verification:
   - Recompute r_t from seed and position
   - Check: s_t == dot(token_vec, r_t) mod PR[SOLUTION>]ME_Q
   - Verify signatures bind to hotkey
```

**[SOLUTION>]A[SOLUTION>] Problem Determinism** (grail/environments/sat_env.py):

```python
# Deterministic generation from seed
def generate_sat_problem(seed: int, difficulty: int):
    rng = random.Random(seed)  # Deterministic R[SOLUTION>]G
    n_vars = 3 + difficulty  # 3-10 variables
    n_clauses = 5 + difficulty * 2  # 5-20 clauses

    clauses = []
    for _ in range(n_clauses):
        clause = rng.sample(range(1, n_vars+1), k=3)
        clause = [v if rng.random() [SOLUTION>] 0.5 else -v for v in clause]
        clauses.append(clause)

    return clauses
```

**Reward Calculation** (grail/environments/reward_components.py:64-116):

```python
# Multi-component reward vector
def compute_reward(completion: str, problem: Problem):
    parsed = parse_completion(completion)

    # Component rewards
    r_correctness = check_solution(parsed.solution, problem)  # 0.7 weight
    r_thinking = 0.5 if has_thinking_tags(parsed) else 0.0   # 0.15 weight
    r_answer = 0.3 if has_solution_tags(parsed) else 0.0     # 0.1 weight
    r_concise = max(0, 0.2 - 0.001*trailing_chars(parsed))   # 0.05 weight

    total = (0.7*r_correctness + 0.15*r_thinking +
             0.1*r_answer + 0.05*r_concise)
    return total  # Range: [0.0, 1.0]
```

## Key Configuration Reference

**Critical Environment Variables** (.env):

```bash
# [SOLUTION>]etwork
B[SOLUTION>]_[SOLUTION>]E[SOLUTION>]W[SOLUTION>]RK=finney              # mainnet (or 'test' for testnet)
[SOLUTION>]E[SOLUTION>][SOLUTION>][SOLUTION>]D=81                      # Grail subnet

# Wallet
B[SOLUTION>]_WA[SOLUTION>][SOLUTION>]E[SOLUTION>]_C[SOLUTION>][SOLUTION>]D=default         # Your coldkey name
B[SOLUTION>]_WA[SOLUTION>][SOLUTION>]E[SOLUTION>]_H[SOLUTION>][SOLUTION>]=miner            # Your hotkey name

# R2 [SOLUTION>]torage (CR[SOLUTION>][SOLUTION>][SOLUTION>]CA[SOLUTION>]: bucket name = account [SOLUTION>]D, region = E[SOLUTION>]AM)
R2_ACC[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_[SOLUTION>]D=abc123           # Cloudflare account [SOLUTION>]D
R2_B[SOLUTION>]CKE[SOLUTION>]_[SOLUTION>]D=abc123            # M[SOLUTION>][SOLUTION>][SOLUTION>] match account [SOLUTION>]D
R2_WR[SOLUTION>][SOLUTION>]E_ACCE[SOLUTION>][SOLUTION>]_KEY_[SOLUTION>]D=...     # Private write credentials
R2_WR[SOLUTION>][SOLUTION>]E_[SOLUTION>]ECRE[SOLUTION>]_ACCE[SOLUTION>][SOLUTION>]_KEY=...
R2_READ_ACCE[SOLUTION>][SOLUTION>]_KEY_[SOLUTION>]D=...      # Public read credentials (on-chain)
R2_READ_[SOLUTION>]ECRE[SOLUTION>]_ACCE[SOLUTION>][SOLUTION>]_KEY=...

# Performance
GRA[SOLUTION>][SOLUTION>]_GE[SOLUTION>]ERA[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_BA[SOLUTION>]CH_[SOLUTION>][SOLUTION>]ZE=4  # Parallel rollouts (1/2/4/8/16)

# Monitoring ([SOLUTION>]ptional)
GRA[SOLUTION>][SOLUTION>]_M[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]R[SOLUTION>][SOLUTION>]G_BACKE[SOLUTION>]D=wandb
WA[SOLUTION>]DB_AP[SOLUTION>]_KEY=...
WA[SOLUTION>]DB_PR[SOLUTION>]JEC[SOLUTION>]=grail
WA[SOLUTION>]DB_E[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]Y=tplr              # Public project
```

**Constants** (grail/shared/constants.py):

```python
W[SOLUTION>][SOLUTION>]D[SOLUTION>]W_[SOLUTION>]E[SOLUTION>]G[SOLUTION>]H = 50              # Blocks per scoring window
B[SOLUTION>][SOLUTION>]CK_[SOLUTION>][SOLUTION>]ME_[SOLUTION>]EC[SOLUTION>][SOLUTION>]D[SOLUTION>] = 12         # [SOLUTION>]arget block time
R[SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>][SOLUTION>]_PER_PR[SOLUTION>]B[SOLUTION>]EM = 16       # Fixed rollouts per problem
CHA[SOLUTION>][SOLUTION>]E[SOLUTION>]GE_K = 16                # Positions verified per rollout
PR[SOLUTION>]ME_Q = 2_147_483_647        # Modulus for sketch commitments
```

## Resources

### scripts/
- `setup_miner_env.sh` - [SOLUTION>]nteractive .env generation wizard
- `check_miner_health.py` - Comprehensive health check script

### references/
- `grail_protocol.md` - Deep dive into GRA[SOLUTION>][SOLUTION>] cryptographic protocol
- `incentive_mechanism.md` - Detailed scoring and weight computation
- `environments.md` - [SOLUTION>]A[SOLUTION>] and G[SOLUTION>]M8K environment specifications
- `performance_tuning.md` - Advanced optimization strategies

## External Resources

- **Covenant A[SOLUTION>]**: https://www.covenant.ai (Grail's parent company)
- **Discord Community**: https://discord.gg/GyzhzRWJBQ (support and discussions)
- **GitHub Repository**: https://github.com/one-covenant/grail
- **Miner Docs**: https://github.com/one-covenant/grail/blob/main/docs/miner.md
- **Validator Docs**: https://github.com/one-covenant/grail/blob/main/docs/validator.md
- **W&B Dashboard**: https://wandb.ai/tplr/grail (public metrics)
- **Grafana [SOLUTION>]ogs**: https://grail-grafana.tplr.ai/ (real-time monitoring)

name: grail-miner description: This skill should be used when setting up, managing, or optimizing Grail miners on Bittensor Subnet 81. Use it for GRAIL protocol tasks including miner setup, R2 storage configuration, model checkpoint management, GRPO rollout generation, performance optimization, competitive monitoring, and troubleshooting common issues like CUDA errors, upload failures, or low scores. Essential for miners working with verifiable post-training, SAT/GSM8K environments, or understanding the GRAIL incentive mechanism to improve competitiveness.

Grail Miner Skill

Overview

Set up and operate Grail miners to participate in verifiable post-training for language models on Bittensor Subnet 81. Grail implements the GRAIL protocol (Guaranteed Rollout Authenticity via Inference Ledger) for cryptographically verifiable GRPO rollouts on SAT and GSM8K problems, with automatic model evolution through distributed training.

Key Innovation: Grail uses cryptographic proofs to bind rollouts to specific models and inputs, enabling decentralized post-training at internet scale with verifiable contributions and on-chain incentives.

Core Capabilities

1. MINER SETUP WORKFLOW

Prerequisites Check before starting:

OS-agnostic: Any platform (Linux/macOS/Windows) with floating point precision within tolerance
Python 3.11+ with
```
uv
```
package manager
Accelerators recommended (NVIDIA GPU for best throughput, but not required)
Bittensor wallet registered to subnet 81 (mainnet) or 429 (testnet)
Cloudflare R2 bucket (name must match account ID, region ENAM)
Dual R2 credentials: read-only (public, committed on-chain) + write (private, local only)
Optional: WandB account for monitoring

Quick Start (6-Phase Setup):

Clone and Install

git clone https://github.com/one-covenant/grail
cd grail
uv venv && source .venv/bin/activate
uv sync  # Reproducible install with lockfile

Generate Environment Configuration
```
./scripts/setup_miner_env.sh
```
- Interactive wizard for .env generation
- Collects network, wallet, R2 credentials
- Validates bucket configuration
- Creates production-ready .env file
Verify Setup
```
python scripts/check_miner_health.py
```
- Comprehensive health checks
- Validates R2 connectivity (read/write)
- Tests wallet registration
- Checks GPU availability
- Verifies drand beacon access
First Run (Test Mode)
```
grail -vv mine  # Verbose mode for debugging
```
- Commits read credentials on-chain (first run only)
- Downloads latest model checkpoint from R2
- Starts generating rollouts for current window
Monitor Performance
- View logs in terminal for immediate feedback
- Check W&B dashboard: https://wandb.ai/tplr/grail (if enabled)
- Monitor Grafana: https://grail-grafana.tplr.ai/

Production Deployment (Systemd)

sudo tee /etc/systemd/system/grail-miner.service > /dev/null << 'EOF'
[Unit]
Description=Grail Miner
After=network-online.target

[Service]
Type=simple
User=miner
WorkingDirectory=/home/miner/grail
Environment="PATH=/home/miner/grail/.venv/bin:/usr/bin:/bin"
ExecStart=/home/miner/grail/.venv/bin/grail mine
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable grail-miner
sudo systemctl start grail-miner
sudo journalctl -u grail-miner -f

2. R2 STORAGE CONFIGURATION (CRITICAL FOR SUCCESS)

The #1 Issue: Miners struggling with R2 bucket setup and dual-credential configuration.

Dual-Credential Architecture:

WRITE CREDENTIALS (Private)      READ CREDENTIALS (Public)
     ↓                                 ↓
Local .env only              Committed on-chain
Used for uploads             Allows validator fetches
Full read/write              Read-only access

Step-by-Step R2 Setup:

Create Cloudflare R2 Bucket
- Go to https://dash.cloudflare.com → R2
- Click "Create Bucket"
- CRITICAL: Bucket name MUST equal your Account ID
- Set region to ENAM (required)
- Get Account ID: Dashboard → Overview → Copy "Account ID"
Generate Write Credentials (Private)
- Go to R2 → "Manage R2 API Tokens"
- Click "Create API Token"
- Name: "grail-write-access"
- Permissions: Edit (full read/write)
- Scope: Select your bucket
- Copy both Access Key ID and Secret Access Key
Generate Read Credentials (Public)
- Create another API Token
- Name: "grail-read-only"
- Permissions: Read (read-only)
- Scope: Same bucket
- Copy both keys

Configure .env:

# Account & Bucket
R2_ACCOUNT_ID=abc123def456  # Your Cloudflare account ID
R2_BUCKET_ID=abc123def456   # MUST match account ID

# Write credentials (private, never shared)
R2_WRITE_ACCESS_KEY_ID=AKIAXXXXXXXXXXXXXXXX
R2_WRITE_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

# Read credentials (public, posted on-chain)
R2_READ_ACCESS_KEY_ID=AKIAXXXXXXXXXXXXXXXX
R2_READ_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Verify Connectivity

python scripts/check_miner_health.py
# Should show: ✅ R2 write access verified
#              ✅ R2 read access verified

How Validators Access Miner Data:

Miner commits read credentials to chain on first run
Validators fetch read credentials from metagraph
Validators download miner's window files from R2
Validators verify GRAIL proofs and score rollouts
Validators set weights based on successful rollouts

Common R2 Issues → See Troubleshooting section

3. MODEL CHECKPOINT MANAGEMENT

How Model Evolution Works:

Grail uses a hybrid approach where models start from a base and evolve through training:

Base Model:
```
Qwen/Qwen2.5-7B-Instruct
```
(initial checkpoint)
Window Checkpoints: Trainer uploads new checkpoint after each window
Automatic Loading: Miners download latest checkpoint at window start
R2 Storage: Checkpoints stored in R2 with retention policy
Milestone Checkpoints: Every 100 windows preserved permanently

Miner Checkpoint Workflow (grail/cli/mine.py:156-165):

# At start of each window
window_start = (current_block // WINDOW_LENGTH) * WINDOW_LENGTH
previous_window = window_start - WINDOW_LENGTH

# Download checkpoint from previous window
checkpoint_path = download_checkpoint(previous_window)
model = load_model(checkpoint_path)

# Generate rollouts with this checkpoint
# Upload rollouts to R2

Checkpoint Naming Convention:

checkpoints/
├── window-71950/           # Recent checkpoint
│   ├── model.safetensors
│   ├── config.json
│   └── tokenizer/
├── window-71900/           # Previous window
└── milestone-71800/        # Milestone (every 100)

Configuration (.env):

# Checkpoint retention (default: 10)
GRAIL_CHECKPOINT_RETENTION_LIMIT=10

# Milestone interval (default: 100 windows)
GRAIL_CHECKPOINT_MILESTONE_INTERVAL=100

# Local cache directory
GRAIL_CACHE_DIR=~/.cache/grail

Manual Checkpoint Operations:

# List available checkpoints
aws s3 ls s3://${R2_BUCKET_ID}/checkpoints/ \
  --endpoint-url https://${R2_ACCOUNT_ID}.r2.cloudflarestorage.com

# Download specific checkpoint
python -c "
from grail.infrastructure.comms import download_checkpoint
path = download_checkpoint(window=71950)
print(f'Downloaded to: {path}')
"

# Clear local cache
rm -rf ~/.cache/grail/checkpoints/*

Key Files:

Checkpoint download:

grail/infrastructure/comms.py:download_checkpoint()

Model loading:
```
grail/cli/mine.py:156-165
```
Trainer upload:
```
grail/cli/train.py:upload_checkpoint()
```

4. GRPO ROLLOUT GENERATION & OPTIMIZATION

What is GRPO?

Group Relative Policy Optimization - a reinforcement learning algorithm that:

Generates multiple rollouts per problem (16 rollouts fixed)
Computes advantages relative to group mean
Optimizes policy using advantage-weighted gradients
Maintains KL divergence from reference model

Rollout Generation Pipeline (grail/environments/loop.py:47-222):

# For each SAT/GSM8K problem:
1. Derive deterministic seed: sha256(block_hash + drand + nonce)
2. Generate problem instance from seed
3. Create GRPO batch (16 rollouts per problem)
4. Generate completions with logprob tracking
5. Parse solutions and compute rewards
6. Calculate advantages (reward - group_mean)
7. Create GRAIL proof (PRF-based commitment)
8. Sign rollout with hotkey
9. Package for upload

Reward Components (grail/environments/reward_components.py):

Total Reward = 0.7*correctness + 0.15*thinking + 0.1*answer + 0.05*no_trailing

- correctness (0.7): SAT solution validity or GSM8K answer correctness
- thinking (0.15): Presence of <start_working_out> tags
- answer (0.1): Presence of <SOLUTION> tags
- no_trailing (0.05): Penalty for text after </SOLUTION>

Performance Optimization:

Batch Size Tuning (.env):

# Number of rollouts to generate in parallel (default: 1)
# Must divide evenly into 16 (valid: 1, 2, 4, 8, 16)
# Higher values = more throughput but more VRAM

GRAIL_GENERATION_BATCH_SIZE=1   # Baseline (lowest memory)
GRAIL_GENERATION_BATCH_SIZE=4   # ~3-4x throughput (recommended for A100)
GRAIL_GENERATION_BATCH_SIZE=16  # ~10x throughput (H100/H200 144GB)

Generation Parameters (hardcoded in constants):

Max new tokens: 1024
Rollouts per problem: 16
Temperature: 1.0 (for diversity)
Top-p: 0.95

Monitor Generation Performance:

# Watch real-time metrics
grail -vv mine

# Key metrics to watch:
# - Generation time per batch
# - Upload time per window
# - Rollout success rate
# - GPU memory usage (nvidia-smi)

Key Files:

Rollout generator:
```
grail/mining/rollout_generator.py
```
Environment loop:
```
grail/environments/loop.py
```
SAT environment:
```
grail/environments/sat_env.py
```
GSM8K environment:
```
grail/environments/gsm8k_env.py
```

5. COMPETITIVE MONITORING & SCORING

Understanding the Incentive Mechanism:

Validators score miners based on unique successful rollouts over recent windows using a superlinear curve:

# Scoring formula (grail/scoring/scorer.py)
for each miner:
    valid_rollouts = count_verified_rollouts(miner, window)
    unique_solutions = count_unique_correct_solutions(miner, window)

    # Superlinear reward curve
    raw_score = (unique_solutions ** 1.5) * valid_rollouts

    # Normalize across all miners
    weight = raw_score / sum(all_raw_scores)

What Matters for High Scores:

Rollout Validity (GRAIL verification)
- Correct token-level proofs
- Valid signatures
- Proper commitment/opening
Solution Correctness (SAT/GSM8K)
- SAT: Assignments must satisfy all clauses
- GSM8K: Final answer must match ground truth
Solution Diversity
- Unique solutions earn more than duplicates
- Explore different solution paths
Volume
- More valid rollouts = higher weight
- Maximize throughput within window

Monitoring Your Competitiveness:

WandB Dashboard (https://wandb.ai/tplr/grail):

# Enable in .env
GRAIL_MONITORING_BACKEND=wandb
WANDB_API_KEY=your_key
WANDB_PROJECT=grail
WANDB_ENTITY=tplr  # Public project

# Metrics tracked:
# - rollout_count: Total rollouts generated
# - upload_success_rate: Upload reliability
# - generation_time_avg: Throughput metric
# - reward_mean: Average reward per rollout

Grafana Dashboard (https://grail-grafana.tplr.ai/):

Real-time logs from all miners
Network-wide statistics
Validator performance

On-Chain Weights (btcli):

# Check your current weight
btcli subnet metagraph --netuid 81 --subtensor.network finney | grep $(cat ~/.bittensor/wallets/default/hotkeys/miner/ss58_address.txt)

# Compare to top miners
btcli subnet metagraph --netuid 81 --subtensor.network finney | sort -k4 -rn | head -20

Performance Analysis:

# Analyze your rollouts locally
from grail.scoring.scorer import compute_miner_scores

# Load your window data
window_data = load_window_rollouts(window_start)

# Compute metrics
valid_count = sum(1 for r in window_data if r['valid'])
success_count = sum(1 for r in window_data if r['success'])
unique_solutions = len(set(r['solution'] for r in window_data if r['success']))

print(f"Valid: {valid_count}/total")
print(f"Successful: {success_count}/{valid_count}")
print(f"Unique solutions: {unique_solutions}")

Improvement Strategies:

Increase Throughput
- Tune
```
GRAIL_GENERATION_BATCH_SIZE
```
- Upgrade GPU (H100/H200 for 10x gains)
- Optimize upload timing
Improve Success Rate
- Monitor reward components
- Check model checkpoint version
- Verify problem difficulty range
Maximize Diversity
- Use higher temperature if allowed
- Generate across different problem seeds
- Explore varied reasoning paths

Key Files:

Scoring logic:
```
grail/scoring/scorer.py
```

Window aggregation:

grail/cli/validate.py:compute_window_scores()

Metrics tracking:
```
grail/shared/logging.py
```

6. TROUBLESHOOTING COMMON ISSUES

CUDA / GPU Errors

Symptom:

CUDA out of memory

or GPU not detected

RuntimeError: CUDA out of memory. Tried to allocate X.XX GiB

Solutions:

Reduce batch size:

export GRAIL_GENERATION_BATCH_SIZE=1

Clear GPU cache periodically (miner does this automatically):

import torch
torch.cuda.empty_cache()

Check GPU availability:

nvidia-smi
python -c "import torch; print(torch.cuda.is_available())"

Verify CUDA compatibility:

nvidia-smi | grep "CUDA Version"
# Should be >= 12.0 for best performance

Note: Grail is OS and hardware-agnostic - GPU is recommended for throughput but not required.

R2 Upload Failures

Symptom: Upload errors or "No uploads" warnings

ERROR: Failed to upload window rollouts to R2
ERROR: Credentials invalid or bucket not found

Solutions:

Verify credentials:

python scripts/check_miner_health.py
# Should show ✅ for both read and write access

Check bucket configuration:

# Bucket name MUST equal account ID
echo "Account: $R2_ACCOUNT_ID"
echo "Bucket: $R2_BUCKET_ID"
# These should match!

Test manual upload:

aws s3 ls s3://${R2_BUCKET_ID}/ \
  --endpoint-url https://${R2_ACCOUNT_ID}.r2.cloudflarestorage.com \
  --profile grail-write

Verify region is ENAM:
- Go to Cloudflare dashboard → R2 → Click bucket
- Region should show "Eastern North America (ENAM)"

Low Scores / No Weights

Symptom: Not receiving weights from validators

INFO: Window complete, 0 successful rollouts
WARNING: No weights received for 3+ windows

Diagnostic Steps:

Check rollout validity:

# Enable verbose logging
grail -vv mine

# Look for:
# ✅ GRAIL proof valid
# ✅ Signature verified
# ✅ Solution correct

Verify uploads succeeded:

# List your window files on R2
aws s3 ls s3://${R2_BUCKET_ID}/windows/ \
  --endpoint-url https://${R2_ACCOUNT_ID}.r2.cloudflarestorage.com

# Should see: {hotkey}-window-{block}.json

Check read credentials on-chain:

# Validators need your read credentials
btcli subnet metagraph --netuid 81 | grep $(cat ~/.bittensor/wallets/default/hotkeys/miner/ss58_address.txt)

# Should show your endpoint and committed credentials

Monitor validator logs (Grafana):
- Visit https://grail-grafana.tplr.ai/
- Search for your hotkey
- Check for verification errors

Compare to checkpoint version:

# Ensure you're using latest checkpoint
ls -lh ~/.cache/grail/checkpoints/
# Should show recent window number

Common Causes:

Read credentials not committed (first run required)
Bucket name ≠ account ID
Wrong region (must be ENAM)
Model checkpoint too old
GRAIL proof failures
Low throughput (not generating enough rollouts)

Drand Beacon Failures

Symptom: Cannot fetch randomness beacon

WARNING: Drand fetch failed, falling back to block hash
ERROR: All drand endpoints unreachable

Solutions:

Miner automatically falls back to block-hash only (safe)

Test drand connectivity:

python -c "
from grail.infrastructure.drand import get_drand_beacon
beacon = get_drand_beacon()
print(f'Beacon: {beacon}')
"

Use explicit fallback mode:

grail mine --no-drand

Check firewall rules (drand uses HTTPS):

curl -I https://api.drand.sh/public/latest

Note: Block-hash fallback is safe and deterministic - validators use same seed derivation.

Wallet / Registration Issues

Symptom: Wallet not found or not registered

ERROR: Wallet 'default/miner' not found
ERROR: Hotkey not registered on subnet 81

Solutions:

Verify wallet exists:

ls ~/.bittensor/wallets/
# Should show your coldkey name

ls ~/.bittensor/wallets/default/hotkeys/
# Should show your hotkey name

Check registration:

btcli wallet overview --wallet.name default --wallet.hotkey miner
# Should show registration on subnet 81

btcli subnet register \
  --wallet.name default \
  --wallet.hotkey miner \
  --netuid 81 \
  --subtensor.network finney

Verify .env matches wallet names:

grep WALLET .env
# BT_WALLET_COLD=default
# BT_WALLET_HOT=miner

Protocol Deep Dive

GRAIL Cryptographic Proof (grail/protocol/):

1. Challenge Derivation:
   seed = sha256(drand_randomness || block_hash || window_context)

2. PRF-Based Commitment:
   For each token t:
     - Generate random vector r_t = PRF(seed, position)
     - Compute sketch commitment: s_t = dot(token_vec, r_t) mod PRIME_Q

3. Verifier Challenge:
   - Validator samples K=16 random positions
   - Requests token IDs and proofs at those positions

4. Verification:
   - Recompute r_t from seed and position
   - Check: s_t == dot(token_vec, r_t) mod PRIME_Q
   - Verify signatures bind to hotkey

SAT Problem Determinism (grail/environments/sat_env.py):

# Deterministic generation from seed
def generate_sat_problem(seed: int, difficulty: int):
    rng = random.Random(seed)  # Deterministic RNG
    n_vars = 3 + difficulty  # 3-10 variables
    n_clauses = 5 + difficulty * 2  # 5-20 clauses

    clauses = []
    for _ in range(n_clauses):
        clause = rng.sample(range(1, n_vars+1), k=3)
        clause = [v if rng.random() > 0.5 else -v for v in clause]
        clauses.append(clause)

    return clauses

Reward Calculation (grail/environments/reward_components.py:64-116):

# Multi-component reward vector
def compute_reward(completion: str, problem: Problem):
    parsed = parse_completion(completion)

    # Component rewards
    r_correctness = check_solution(parsed.solution, problem)  # 0.7 weight
    r_thinking = 0.5 if has_thinking_tags(parsed) else 0.0   # 0.15 weight
    r_answer = 0.3 if has_solution_tags(parsed) else 0.0     # 0.1 weight
    r_concise = max(0, 0.2 - 0.001*trailing_chars(parsed))   # 0.05 weight

    total = (0.7*r_correctness + 0.15*r_thinking +
             0.1*r_answer + 0.05*r_concise)
    return total  # Range: [0.0, 1.0]

Key Configuration Reference

Critical Environment Variables (.env):

# Network
BT_NETWORK=finney              # mainnet (or 'test' for testnet)
NETUID=81                      # Grail subnet

# Wallet
BT_WALLET_COLD=default         # Your coldkey name
BT_WALLET_HOT=miner            # Your hotkey name

# R2 Storage (CRITICAL: bucket name = account ID, region = ENAM)
R2_ACCOUNT_ID=abc123           # Cloudflare account ID
R2_BUCKET_ID=abc123            # MUST match account ID
R2_WRITE_ACCESS_KEY_ID=...     # Private write credentials
R2_WRITE_SECRET_ACCESS_KEY=...
R2_READ_ACCESS_KEY_ID=...      # Public read credentials (on-chain)
R2_READ_SECRET_ACCESS_KEY=...

# Performance
GRAIL_GENERATION_BATCH_SIZE=4  # Parallel rollouts (1/2/4/8/16)

# Monitoring (Optional)
GRAIL_MONITORING_BACKEND=wandb
WANDB_API_KEY=...
WANDB_PROJECT=grail
WANDB_ENTITY=tplr              # Public project

Constants (grail/shared/constants.py):

WINDOW_LENGTH = 50              # Blocks per scoring window
BLOCK_TIME_SECONDS = 12         # Target block time
ROLLOUTS_PER_PROBLEM = 16       # Fixed rollouts per problem
CHALLENGE_K = 16                # Positions verified per rollout
PRIME_Q = 2_147_483_647        # Modulus for sketch commitments

Resources

scripts/

```
setup_miner_env.sh
```
- Interactive .env generation wizard
```
check_miner_health.py
```
- Comprehensive health check script

references/

```
grail_protocol.md
```
- Deep dive into GRAIL cryptographic protocol
```
incentive_mechanism.md
```
- Detailed scoring and weight computation
```
environments.md
```
- SAT and GSM8K environment specifications
```
performance_tuning.md
```
- Advanced optimization strategies

External Resources

Covenant AI: https://www.covenant.ai (Grail's parent company)
Discord Community: https://discord.gg/GyzhzRWJBQ (support and discussions)
GitHub Repository: https://github.com/one-covenant/grail
Miner Docs: https://github.com/one-covenant/grail/blob/main/docs/miner.md
Validator Docs: https://github.com/one-covenant/grail/blob/main/docs/validator.md
W&B Dashboard: https://wandb.ai/tplr/grail (public metrics)
Grafana Logs: https://grail-grafana.tplr.ai/ (real-time monitoring)

View Original Source

Related Skills

Coding

PromptBeginner5 minmarkdown

Markdown Converter

Agent skill for markdown-converter

Jan 8, 2026

Coding

PromptBeginner5 minmarkdown

Nano Banana Pro

Agent skill for nano-banana-pro

Jan 8, 2026

Coding

PromptBeginner5 minmarkdown

1password

Agent skill for 1password

Jan 8, 2026