On-chain or Lightning

This article explains how to set up Bitcoin Core, Electrum Personal Server (EPS), and Electrum Desktop Wallet, and then privately connect the Wallet to Bitcoin Core via EPS.

It would be nice if your Electrum Wallet could just get blockchain data from your own Bitcoin Core, but it can’t. It needs an Electrum Server to be the middle man. The server does not copy the Blockchain data, it is not a node — Bitcoin Core is.

Note the Electrum Personal Server (EPS) is one of several server options. Others include Electrum Server in Rust (Electrs), the no-longer-supported ElectrumX, and ElectrumX/Spesmilo which is an ElectrumX variant that is supported. There are others too, each with different trade-offs. I will make guides for these later.

One thing to note about EPS is that it really is “personal” — it only connects to one Electrum Wallet at a time. So if your desktop wallet is connected, don’t spend hours tinkering with your mobile Electrum Wallet trying to figure out why it doesn’t connect (true story). You also can’t be the “Uncle Jim” of your social circle with EPS, by providing a trusted node for them to connect to. I suggest getting EPS working first, then I’ll show you how to install the more powerful servers next time, after I figure it out. I’ll go through more hours of pain and suffering, so you all won’t have to.

This is slightly more technical than running a bundled node like MyNode (link) or RaspiBlitz (link); these have many other applications you can run with a relatively straightforward install — but because there are more moving parts, when something goes wrong it’s harder to troubleshoot. By all means, use those easy-to-install nodes, but I would do that IN ADDITION to this. When something goes wrong and you end up needing to re-sync the blockchain on your bundled node, at least you have this simple setup up that will not let you down. The MyNode setup (and Ronin Dojo) are inside docker containers — you need to be a computer scientist to have any hope of accessing the files manually.

This article will guide you with some command line. The most important tip from here on is PAY ATTENTION TO DETAIL, a small typo can send you down a frustrating search for “why does this not work?” I promise it will be easy if you follow the instructions, ask me if you get stuck.

This does require you to know how to navigate around the file system in Linux/Mac. If you don’t, then watch this 11-minute video to learn some SSS (super simple stuff). It’s about Linux, but it works the same on a Mac. If it’s mildly interesting, watch this one as well, it’s a bit longer.

The Hard Drive

You can connect and format an external hard drive, or use your internal hard drive. Be prepared to use up about 350 Gb (in 2020), and have space for it to grow. Connecting a 1 or 2-terabyte solid-state drive is preferable but is an extra cost.

Download Bitcoin Core

Download the Mac version from http://bitcoin.org/en/download

Also, click Verify release signatures to download a file containing the SHA 256 hashes of each Bitcoin Core OS version, for reference.

Next, using the terminal, navigate to where Bitcoin Core was downloaded (usually ~/Downloads/ ) and type:

shasum -a 256 NameOfDownloadedFile

This will give you a hash output. Compare the output with the listed hash you downloaded, to make sure the file has not been tampered with.

Extract the downloaded file using the graphical interface. I suggest moving the extracted directory to the desktop, and renaming it “Bitcoin”

Run Bitcoin Core graphical user interface by double-clicking the icon.

It will ask you where to store the bitcoin directory — custom or default. Select default if you are using the internal drive, and if you are using an external drive, select “custom” and provide the location of your drive.

Before clicking OK, make sure “discard blocks after verification…” is not selected.

It should now start downloading the blockchain and will take several days. HOWEVER, before you get too far, go to the menu “settings” and click “options…” and make sure that the “prune block storage” option is not checked — assuming you want your node to be a full un-pruned node. If you changed this now, close the program and run it again. It will delete the blockchain data it has downloaded and will start again.

Meanwhile, we need to edit the bitcoin configuration file.

To access the bitcoin.conf file is through the graphical user interface. Go to menu “settings” and select “options…” and click “open configuration file”.

You’ll see it is blank. Add a line txindex=1— this allows any transaction to be looked up by Bitcoin Core, not just your own wallet. On the next line, add server=1. This allows Electrum Personal Server to talk to Bitcoin Core.

Save and exit.

You’ll need to shut down Bitcoin Core and restart it for the config file changes to take effect, but don’t worry, this time you won’t lose any blockchain data that you’ve already downloaded.

At some point, you will have the option to create a new wallet. Do that, but remember, do not encrypt the wallet. Also, don’t use this wallet for any funds – it’s a hot wallet and that’s not safe. Your bitcoin should be stored offline.

Install GPG

My mac did not have GPG installed. Newer ones may. To install gpg, in the terminal, type:

brew install gpg

Download Electrum Personal Server

You can do this while the Bitcoin blockchain is downloading.

Go to the Github release page. Link

There are two files you need to download. The source code (don’t worry, we aren’t compiling) and the signature.

Download one of the source codes (either the “.zip” compression or the “.tar.gz” compression). Also, download the corresponding “.asc” file. Eg, download “SourceCode.zip)” and “eps-v0.2.1.1.zip.asc”.

Next, download Chris Belcher’s public key; there is a link on the page, and it’s also here: Link

Click “raw” and then save the data to your computer. Note its location. Open the terminal, and navigate to the location of the public key. (If it’s in Downloads, type cd ~/Downloads and <enter>.

Type

gpg --import NameOfPublicKey

In Linux/Mac, the double dash precedes options for commands that are longer than one letter. A single dash means that if more than one letter follows, each letter is an individual option. Eg -abcd is four separate options, and--abcd is one option that’s called “abcd”.

With the above command, Chris’s public key should be now stored on your computer.

Next, in the terminal, navigate to where the source code and signature file is. Type

gpg --verify NameOfSignatureFile NameOfZipFile

Note the space after NameOfSignatureFile.

If the signature file and the zip file only differ by a “.asc” at the end of the signature file (eg abcdefg.zip & abcdefg.zip.asc), then in the gpg command you don’t need to type the “NameOfZipFile”. Gpg will know what you want.

The output should say something like this:

gpg: assuming signed data in ‘eps-v0.2.1.1.zip’gpg: Signature made Tue 9 Jun 23:30:42 2020 AESTgpg: using RSA key EF734EA677F31129gpg: Good signature from “Chris Belcher <false@email.com>” [unknown]gpg: WARNING: This key is not certified with a trusted signature!gpg: There is no indication that the signature belongs to the owner.Primary key fingerprint: 0A8B 038F 5E10 CC27 89BF CFFF EF73 4EA6 77F3 1129

The Good signature from is the crucial part. The warning is not relevant; safe to ignore.

Next, unzip the source code and put the directory where you want it. I put it on the Desktop. Then rename it to something easy to type, eg “eps”.

Install Electrum Personal Server

Next is the bit that I seriously tripped up on, and I want it to go smoothly for you.

Make sure you have the latest version of pip. Type this in the terminal:

sudo pip3 install --upgrade pip

In the terminal, check that you have Python 3 installed. Type Python3 --version. If you don’t have at least version 3, download it from here.

Run this command to make sure you have the latest version of “pip”:

sudo pip3 install --upgrade pip

Then, navigate in terminal to the unzipped directory which you wisely placed on the Desktop and renamed ~/Desktop/eps/. Inside, there should be several files (and directories) including a file called setup.cfg.

Within this directory, type

pip3 install --user .

Pay attention… there is a “.” after “user”. I have tripped up, trying on various operating systems and computers, and always failed at this point, until I realised I wasn’t typing that pesky “.”

Also, note user is typed as is, it’s not your username. Don’t replace bit with “MacLoginUserName”.

Also note, if you accidentally type the command with sudo at the start it will install EPS as the root which is not what we want. We want to install it to “MacLoginUserName”

I got the following error…

ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: ‘/Users/MyMacLoginUserName/Library/Python/3.8/lib’Check the permissions.

Wait, “Errno”? is that a typo in the error message?

It’s fiddly but it can be fixed. If we typed sudo before the install command, the software gets installed in an administrator directory and is hidden from the user account; we don’t want that. So instead, we can give the user access to the destination directory.

Open a Finder window and navigate to /Users/YourUserName. Then press the Apple Key, Shift, and. simultaneously to reveal hidden files and directories. Library will appear. Go into Library, then Python. You’ll see a 3.8 with a “no entry” sign. We are not allowed in there. Minimise the window and leave it open for reference.
In the terminal, navigate to /Users/YourUserName/Library, and type sudo chown YourUserName: Python — don’t forget the : with a space after it. This changes the owner of the Python folder from the root administrator to the user. (We are going to reverse this after installation for safety)
In the terminal, navigate to /Users/YourUserName/Library/Python, and type sudo chmod 750 3.8/ (Note the space after 750). This allows anyone with a username and password for the machine to look in this directory. (We are going to reverse this after installation for safety)
In the terminal, navigate to /User/YourUserName/Desktop/eps/ assuming that’s where you put the unzipped directory for eps, and you renamed it eps.
Type pip3 install --user . .There’s a space after install . Type user as is. Don’t forget the .

You should get an output similar to this…

WARNING: The script electrum-personal-server is installed in ‘/Users/YourUserName/Library/Python/3.8/bin’ which is not on PATH.Consider adding this directory to PATH or, if you prefer to suppress this warning, use — no-warn-script-location.NOTE: The current PATH contains path(s) starting with `~`, which may not be expanded by all applications.Successfully installed electrum-personal-server-0.2.0.dev0

IGNORE THE WARNING ABOUT “PATH”

6. Next, in the terminal, navigate to

/Users/YourUserName/Library/Python/3.8/bin

and type

mv electrum-personal-server /Users/YourUserName/Desktop/eps

This will move the executable file out of the folder we are going to make hidden again, and into somewhere convenient.

7. Next, again in the terminal, navigate to

/Users/YourUserName/Library/Python

and type

sudo chmod 700 3.8/

(this reverses the less secure access)

8. Lastly, navigate to /Users/YourUserName/Library, and type sudo chown root: Python — don’t forget the : with a space after it. This changes the owner of the Python folder back from the YourUserName to the root. Reversed back as promised.

Don’t run EPS yet

Before running EPS, you should:

Wait for Bitcoin Core to fully synchronise
Have Electrum Wallet setup up
Have ready a public key (xpub, ypub, or zpub), or multiple public keys (Xpub’s Ypub’s or Zpub’s) for a multisignature wallet.

Set up an Electrum Wallet

To set up an Electrum Wallet, go to https://electrum.org and download the Mac executable. Make sure you verify the download, it’s extremely important. Instructions on how to do this are the same as described above for verifying Electrum Personal Server, except you need a different key, this time ThomasV’s. It is listed on Electrum Website. Double-check that you are downloading from the right website address.

Gather your public keys

Copy your public keys from your Electrum wallet, or your hardware wallet(s) and keep them handy in a text file on your desktop.

In your Electrum Wallet, go to the menu “wallet” and “information” and you’ll see this…

This is a multisignature wallet with five Zpubs. You can select each one and copy the text that appears below as needed. If you have a single signature wallet, you’ll see just one “keystore”.

Edit EPS config file

Go to the unzipped eps directory again and rename the file config.ini_sample to config.ini

The config file needs to be edited properly for the program to work.

Open this text file and begin editing as I will explain. Note, anything that starts with a# is just a comment and will be ignored by EPS. Also note that for some lines, it may appear not to start with a #, but in fact, it might just be wrapped around from the line above that didn’t fit in the narrow window.

This is a screenshot of part of the config.ini file:

Notice (it’s hard to see at first glance) the config file is divided up by headings in square brackets. EPS looks for specific instructions under these headings.

Also, notice the line that says #multisig wallet = 2 followed by three lines of xpubs. This is an example of text wrapping around that is all actually on one line.

[master public keys]

Under this first heading, add your public keys that you’ll find in Electrum Wallet. I couldn’t get EPS running without first doing this.

For a multisig wallet, for example, a 3 of 5 wallet, in the EPS config.ini file, add (all in one line, separating each Zpub with a space)

AnyNameForYourWallet = 3 Zpub1 Zpub2 Zpub3 Zpub4 Zpub5

The number after the equals sign indicates the minimum number of private keys to spend from the wallet, and the number of Zpubs you list defines the number of public keys in the wallet.

This is just like the example given in the config file, except, don’t type a # at the start.

For a single signature wallet, it’s much easier, under the same heading in the config.ini file, add the line

AnyNameForWallet2 = zPub

— or use an xPub or yPub

[bitcoin-rpc]

This is the next heading in the config.ini file. Your Bitcoin Core Node will be running on the same computer as your EPS (I do recommend this, because I have not yet succeeded in running EPS and Bitcoin Core on different computers while talking to each other), so leave this address as-is:

host = 127.0.0.1
port = 8332

The IP address 127.0.0.1 is code for “this computer” and is universal.

If your EPS really needs to connect to a Bitcoin Core on a different computer on the same home network, then you add the bitcoin core computer’s IP address. I believe you also need to add a line in the Bitcoin Core config file, rpcallow=ip_address_of_EPS_computer. I haven’t tried hard enough to make this work, so I have no tested instructions for you. It might not even be developed for that, and so it may not be possible yet. If you want to try yourself, don’t let me stop you.

The next part is a bit tricky and non-intuitive:

Note the following lines in the config.ini file:

#add the bitcoin datadir to search for the .cookie file created by the
# node, which avoids the need to configure rpc_user/pass
#leave this option empty to have it look in the default location
datadir =
#if you dont want to use the .cookie method with datadir, uncomment to config u/p here
#rpc_user =
#rpc_password =

Basically, you have a choice

uncomment the datadir= line

2. uncomment the rpc_user= and rpc_password= lines.

The default is using the datadir= line and commenting out the 2nd option. Do this.

If during your bitcoin installation, you didn’t move the data directory (where the blockchain is stored), you can leave the line datadir= as is, BUT if you moved it to the external hard drive, you need to enter the path to the drive after datadir=

You can find the path name by going to the terminal, typing cd /Volumes and then ls — You’ll see the drive name there. Enter that after datadir=
in the EPS config file. If there are spaces in the name of the drive, use quotes around the name. eg.

datadir= /Volumes/“Untitled Drive

Notice this is a directory where .cookie will be found. You shouldn’t actually enter the file as well, just the directory.

Bitcoin Core’s config file starts empty, however, if you have been playing around with Bitcoin Core prior to this, you may have these lines (below) already in there:

rpcuser=UserNameOfYourChoice

and

rpcpassword=PasswordOfYourChoice

If these lines exist in bitcoin.conf, Bitcoin Core does not create a .cookie file. Electrum Personal Server will look for that file, and because it doesn’t exist, it will create an error. This pesky trap blocked me from getting this project done for months. I eventually found this solution from an obscure online forum post in the depths of internet history, among many many non-solutions. Comment out these lines with a # or delete them, and it will create the elusive .cookie file.

An alternative is to comment out the datadir= line in EPS config.ini, add include rpc_user=UserNameOfYourChoic and rpc_password=PasswordOfYourChoice to that file, and only THEN, you can leave rpcuser and rpcpassword in the bitcoin.conf file. During my many weeks of failure, this option still created an error though. Theoretically, it should work. Many people posted this issue in forums, and the solution was to go with the datadir= option. Which didn’t work for me because of the missing .cookie file.

Just in case you want to know:

How to find where the DEFAULT data directory for Bitcoin Core is?

On a Mac, the default location is: ~/Library/Application\ Support/Bitcoin/

(For Linux, it’s a hidden file. Located here ~/.bitcoin/. To navigate there, you type cd ~ and then cd .bitcoin — you must type that . which indicates the directory is hidden)

[watch-only-addresses]

You have to enter at least an address or a public key in the appropriate location of the config file for EPS to run.

If you are not ready with your public keys, you can enter one of your (or anyone else’s) random Bitcoin addresses here, just to get EPS working.

A_Name = bc1qGivEmE6poINt15BitcOINplez

Other config.ini headings

There are other headings, but this is basically all you need to do. The rest of the config.ini file you can read, and probably leave as is, unless you find a good reason to change something.

Run Electrum Personal Server

Once the EPS config file is edited, and once Bitcoin Core is fully synchronised, and also running, we can run Electrum Personal Server. Do remember Bitcoin Core needs to be running.

Go to the eps directory while Bitcoin Core is running, and type

./electrum-personal-server config.ini

This is just two filenames separated by a space.

If you are not in the directory for a file you type into a command, you have to type its full path in the command. Here we put both files in one directory, otherwise, we would have had to type the path for each file in the command.

Once you run this for the first time, you’ll get a message about re-scanning, and it will exit. This is normal. It says:

If recovering a wallet which already has existing transactions, then run the rescan script. If you’re confident that the wallets are new and empty then there’s no need to rescan, just restart this script

EPS does some magic I don’t understand, but I think of it as having its own address list to look for. It needs to be told how back in the blockchain to look. For this, it needs to scan the blockchain. You only need to scan as far back as your first bitcoin UTXO was received in the wallet you are going to watch, but I would scan the whole blockchain to remain maximally flexible. It takes a bit longer of course. This is how to do it:

Navigate to your EPS directory. Make sure Bitcoin Core is running, and from there, type

./electrum-personal-server --rescan config.ini

It will ask for the earliest wallet creation date. Enter 1 to start from block height 1, and y to confirm.

Now, wait a while, possibly a day or two.

Once it is done, run EPS again

./electrum-personal-server config.ini

Connect your Electrum Desktop Wallet

When you run Electrum Desktop Wallet for the first time, and load your real addresses or public keys, be disconnected from the internet so you don’t accidentally leak private information before you have the settings sorted out correctly. It’s difficult to get the settings right without first running Electrum. So here are the steps and my video demo link:

Stop Bitcoin Core. Stope EPS. Disconnect the internet (double check by failing to load a webpage)
Run Electrum
Choose Select server manually

4. Make sure Select server automatically is not ticked, and type in your local host with port 50002 (it’s always 127.0.0.1:50002), then click Next

5. Proceed to making or loading up a wallet, or a watching address. Whichever you choose, make sure it is the same as what you entered in Electrum Personal Server’s config.ini file.

6. Go to the menu “Tools” then “Electrum Preferences”, then change the Base unit from mBTC to BTC, then shut down Electrum.

7. Navigate with the terminal to ~/.electrum , by typing cd ~ then cd .electrum (this directory is only created once Electrum is run for the first time)

8. Type nano config and make some changes as follows: auto_connect should be false. check_updates should be false. onserver is false, but change it to true — this is really important to prevent your server from leaking private information. server should be 127.0.0.1:50002:s

9. Connect to the internet. Make sure Bitcoin Core is running. Make sure EPS is running.

10. Load Electrum Wallet. It should be working and showing your balance, verified by your own Node!

Troubleshooting:

Make sure you have only one wallet open at a time.

Make sure any wallet you open is also specified in the config.ini file of EPS.

Make sure Bitcoin Core is fully synced and running.

Make sure EPS is running.

If you created a wallet with Bitcoin Core and encrypted it, I have no idea why, but EPS fails. The solution is to back up your wallet, and remove it or delete it from the data directory. Then restart Bitcoin Core and create a new wallet. This time, don’t encrypt it.

Tips:

Static Lightning Address: dandysack84@walletofsatoshi.com

On-chain or Lightning