Setting up a full Erigon Ethereum node on AWS - Part 3/4 Erigon and RPC Daemon

This post is part of a multi-series write-up about setting up Erigon on AWS. We previously looked at setting up the AWS infrastructure in a reproducible way using Terraform and also automated the creation of our admin users and security for all the Linux instance. Now we are ready to install the actual software that will run our node.

Table of contents

1. Terraforming AWS
2. Linux Security hardening with Ansible
3. Erigon and RPC Daemon (this guide)
4. Metrics and monitoring with Prometheus and Grafana

Erigon Server

Let's start with the main playbook file for this one and work our way through the roles we created:

---
- hosts: eth_node
  become: true
  collections:
    - devsec.hardening
  vars:
    users:
      - name: raz
        # generated using openssl passwd -salt <salt> -1 <plaintext>
        password: '$1$salty$BnuYTcuR3sS3eurvygJ.H1'
        pub_keys:
          - templates/users/raz/key.pub
    sysctl_overwrite:
      # Enable IPv4 traffic forwarding.
      net.ipv4.ip_forward: 1
  roles:
    - users
    - devsec.hardening.os_hardening
    - golang
    - erigon

We reused the same users and os_hardening from the previous step. The other two roles we need is golang (for installing GO) and the erigon role itself to compile and configure Erigon.

Go Lang

This is the basic structure for our Go role:

.
├── golang
│   ├── defaults
│   │   └── main.yml
│   └── tasks
│       └── main.yml

This is not the most idiomatic Ansible way of defining variables, but it will work for our purposes:

---
go_binary: https://go.dev/dl/go1.18.4.linux-amd64.tar.gz
go_version: go1.18.4.linux-amd64
go_path: /opt/go
go_install_path: /usr/local/

And the main playbook:

---
- stat:
    path: "{{ go_install_path }}go/bin/go"
  register: go_installed
  tags:
    - golang

- name: download package
  get_url:
    url: "{{ go_binary }}"
    dest: "/tmp/{{ go_version }}.tar.gz"
  when: go_installed.stat.exists == false
  tags:
    - golang

- name: extract go package
  unarchive:
    src: "/tmp/{{ go_version }}.tar.gz"
    dest: "{{ go_install_path }}"
    remote_src: yes
  when: go_installed.stat.exists == false
  tags:
    - golang

- name: make go executable
  file:
    path: "{{ go_install_path }}go/bin/go"
    mode: a+x
  tags:
    - golang

- name: link to binary
  file:
    path: "{{ go_install_path }}bin/go"
    src: "{{ go_install_path }}go/bin/go"
    state: link
  tags:
    - golang

Erigon

Erigon is equally simple. The files contain templates for systemd configurations. We are running the node and the rpc daemon as separate services.

.
├── erigon
│   ├── files
│   │   ├── erigon.service
│   │   └── rpcdaemon.service
│   └── tasks
│       └── main.yml

Let's list out the main playbook. It's pretty straight forward, clone the repo, compile, create the necessary users and configure systemd.

---
  - stat:
      path: /usr/local/bin/erigon
    register: erigon_built
    tags:
      - erigon

  - stat:
      path: /usr/local/bin/rpcdaemon
    register: rpcdaemon_built
    tags:
      - erigon

  - stat:
      path: /home/erigon
    register: erigon_user
    tags:
      - erigon

  - name: Install build-essential
    apt:
      name: build-essential
      state: present
      update_cache: yes
    tags:
      - erigon

  - name: Clone erigon repo
    git:
      repo: https://github.com/ledgerwatch/erigon.git
      dest: /tmp/erigon
      depth: 8
      clone: yes
      update: yes
      recursive: yes
    when: erigon_built.stat.exists == false
    tags: 
      - erigon

  - name: Make erigon
    make:
      chdir: /tmp/erigon
      target: erigon
    when: erigon_built.stat.exists == false
    tags: 
      - erigon

  - name: Make rpcdaemon
    make:
      chdir: /tmp/erigon
      target: rpcdaemon
    when: rpcdaemon_built.stat.exists == false
    tags: 
      - erigon

  - name: Move erigon binary
    command: mv /tmp/erigon/build/bin/erigon /usr/local/bin
    when: erigon_built.stat.exists == false
    tags: 
      - erigon

  - name: Move rpcdaemon binary
    command: mv /tmp/erigon/build/bin/rpcdaemon /usr/local/bin
    when: rpcdaemon_built.stat.exists == false
    tags: 
      - erigon

  - name: Erigon binary permissions
    file:
      path: /usr/local/bin/erigon
      owner: erigon
      group: erigon
    tags: 
      - erigon

  - name: Rpcdaemon binary permissions
    file:
      path: /usr/local/bin/rpcdaemon
      owner: erigon
      group: erigon
    tags: 
      - erigon

  - name: Make erigon user
    make:
      chdir: /tmp/erigon
      target: user_linux
    when: erigon_user.stat.exists == false
    tags: 
      - erigon

  - name: Copy systemd service file to server
    copy:
      src: erigon.service
      dest: /etc/systemd/system
      owner: erigon
      group: erigon
    when: erigon_built.stat.exists == true
    tags: 
      - erigon

  - name: Copy rpcdaemon systemd service file to server
    copy:
      src: rpcdaemon.service
      dest: /etc/systemd/system
      owner: erigon
      group: erigon
    when: rpcdaemon_built.stat.exists == true
    tags: 
      - erigon

  - name: Start erigon service
    systemd:
      name: rpcdaemon.service
      daemon_reload: yes
      enabled: true
      state: started
    tags:
      - erigon

Running Erigon with Systemd

This is pretty straight forward as well, we're enabling metrics, as well as the API.

[Unit]
Description=Erigon Node
After=network.target network-online.target
Wants=network-online.target

[Service]
WorkingDirectory=/usr/local/bin
ExecStart=/usr/local/bin/erigon --datadir=/home/erigon/mainnet --private.api.addr=0.0.0.0:8090 --prune=htc --prune.r.before=11052984 --metrics --metrics.addr 0.0.0.0
User=erigon
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

We want to be efficient with the footprint of the data, so we are using pruning - this is an important consideration if you're planning on running this as an Execution Layer for your Beacon chain (post-merge).

The erigon repo lists out some block numbers in the Beacon section of their README:

Beacon Chain (Consensus Layer)

Erigon can be used as an Execution Layer (EL) for Consensus Layer clients (CL). Default configuration is OK. CL relies on availability of receipts – don't prune them: don't add character r to --prune flag. However, old receipts are not needed for CL and you can safely prune them with --prune htc.

This means we can prune everything but the receipt. But even for those, we don't need receipts older than the ETH2 Deposit Contract Block Number (11052984). The rest of the prune flag are:

   --prune value                             Choose which ancient data delete from DB:
                                             h - prune history (ChangeSets, HistoryIndices - used by historical state access, like eth_getStorageAt, eth_getBalanceAt, debug_traceTransaction, trace_block, trace_transaction, etc.)
                                             r - prune receipts (Receipts, Logs, LogTopicIndex, LogAddressIndex - used by eth_getLogs and similar RPC methods)
                                             t - prune transaction by it's hash index
                                             c - prune call traces (used by trace_filter method)
                                             Does delete data older than 90K blocks, --prune=h is shortcut for: --prune.h.older=90_000
                                             Example: --prune=hrtc (default: "disabled")

RPC Daemon

Unlike other nodes such as Geth, Erigon separates the RPC daemon for the node daemon. They communicate with each other using the GRPC protocol, so when you run your typical JSON-RPC queries you are used to, you are in fact hitting a separate process. Here's the systemd configuration for that

[Unit]
Description=Erigon RPC Daemon
After=erigon.service

[Service]
WorkingDirectory=/usr/local/bin
ExecStart=/usr/local/bin/rpcdaemon --datadir=/var/data/mainnet --txpool.api.addr=0.0.0.0:9090 --private.api.addr=0.0.0.0:9090 --http.addr=0.0.0.0 --http.api=eth,erigon,web3,net,debug,txpool --ws
User=erigon
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

This is pretty straight forward. We're enabling websockets and a bunch of APIs. We're also allowing external traffic (via the --http.addr=0.0.0.0). In a future post I will detail how to make this more performant and secure by putting a NGINX server in front of the RPC Daemon.

Continue

Part 4 of this series is out.