Hydra node health check
This article is all about the health of your HYDRA staking node, and the important things you need to know to win block rewards on the HYDRA blockchain.
1. The probability game and when to expect to win a block reward
You intuitively know how probability works for dice. Probability lets us make calculations about random events like rolling dice (or HYDRA block rewards). We will start considering a die (a single dice) and define success as rolling a “1”. You expect to get a “1” every 6 rolls (“Expected Time” = 6), but also you know you are not guaranteed to get a “1” every 6 rolls because this is a random process.
The probability of an event occurring is given on a scale of 0 (will never happen) to 1.0 (definitely going to happen). The probability of getting a “1” in a single roll of the die is 1/6 or 0.167.
We can use this same approach to calculate the probability of winning a HYDRA block reward for the next block. To make the math easier, let’s consider a wallet with 1000 HYDRA, and a Network Weight of 2,000,000. The wallet software makes the probability of winning a block reward 1,000 / 2,000,000 = 0.0005 = 1⁄2000. What does this mean, compared to the die?
Die | HYDRA | |
Probability of success | 0.167 | 0.0005 |
Expected time | 6 rolls | 2000 blocks |
Here we see the probability of success for a single trial, getting a “1” in a single roll of the die, or a block reward on the next block, and the Expected Time to the successful result.
This “Expected Time” is the same answer given by the HYDRA wallet for a given wallet weight and network weight, which the wallet calculates based on the difficulty of minting a new block. For our example, the expected time is 2,000 blocks or about 71 hours (2.9 days). In other words the "Expected Time" is another way to express the probability of the event of winning a block reward.
Rolling the die or trying for a HYDRA block reward are called “Independent Trails” in probability, which means each roll of the die or HYDRA block reward is a new probability event with an identical chance of success. The previous results have no influence on the results for independent trials. Contrast this, for example, with double-deck Blackjack, and if you can count cards this is not a game with independent trials.
Let’s look at “Expected Time” more, with a brain teaser. The Expected Time is 2000 blocks (about 2.9 days). After half the Expected Time has elapsed from the last block reward, what is the Expected Time to the next block reward?
Answer: the Expected Time is still 2.9 days!
This has to be true because of independent trials. The "kernel" hash random number generator lives in the moment, has no history, and can only tell who wins the next block reward. “Independent Trials” means each new block reward, like each roll of the die, is independent of any previous results.
To get a more useful answer for “when block reward?” let’s look at this chart representing Expected time vs Probability to win a block reward ⇓
How to read this chart? The horizontal axis gives the multiple of Expected Time. The vertical axis gives the percent chance of a block reward. The red lines show that there is a 63% chance of receiving a block reward at the Expected Time. This means after your wallet receives 100 block rewards, about 63 of them will have arrived before the Expected Time. Some block rewards however would arrive much later than the Expected Time, and blocks arriving 5~9x the Expected Time are still possible.
2. Optimizing UTXOs
One possible optimization of your wallet is to optimize the number and size of UTXOs in your wallet.
UTXOs are like bills in the wallet. Imagine you have one 10$ bill and one 5$ bill (two UTXOs), and you want to spend 3$. Then one of the bills (UTXOs) in the wallet will be spent and you will receive a change. The wallet will determine automatically which UTXO to use, but you can also manually point which UTXO should be used to spend. For this to be possible you need to enable Coin Control features option here: SETTINGS > OPTIONS > WALLET ⇓
When the wallet is staking, for each block a "kernel" hash is calculated. The "difficulty" of the blockchain is weighted for each UXTO and compared to the "kernel" hash.
If
"kernel" hash < "difficulty" x UTXOvalue
then the wallet announces a "mined" block and the winning UXTO (together with the block reward) has to mature again for 2,000 blocks (approx. 17 hours). The rest of the balance remains available for staking. Obviously if you have all the HYDRAs in one UTXO, if it wins a block it will not be staking until mature again. So It makes sense to split larger UXTOs into smaller ones.
The wallet will split automatically (free of fees) UTXOs larger than 200 HYDRAs and will merge UXTOs smaller than 100 when a new block is won. But you can speed up the process, by manually selecting bigger UTXOs (like 1000 HYDTA or more) and sending part of each UTXO back to yourself. You will receive what you sent plus the rest as "change" (minus the fee for the transaction) and next time you win a block only part of your balance (the winning UTXO + reward) will have to mature again.
3. The importance of time synchronizing the node
The wallet (when staking) calculates the "kernel" hash every block. Then the hash is compared to the weighted "difficulty". If the "kernel" hash is less than the weighted "difficulty", then the wallet announces to its peers a new block with the winning reward. The peers check the validity of the "kernel" hash and can accept it (and "gossip" to their peers) or reject it.
The formula for calculation of the "kernel" hash uses several inputs, that are essential for the randomness of the calculated hash. One of these inputs is the time stamp (in UTC time) of the newly created and announced block. Obviously if the peers do not find the new block to be valid (for many reasons, including the time stamp being "in the past" or "in the future") they will reject it and the wallet will automatically remove the reward from its local copy of the blockchain.
4. Checking if node time is synchronized properly
There are several ways to check if the system clock of the node is in sync with UTC time. Simplest way is to compare to the clock of your smartphone or laptop. The clock on the node and any other device you have should tick in perfect sync to the second. If not, then one of the devices or may be more of them are not synced with UTC time.
What is UTC time (i.e. Coordinated Universal Time)? UTC is provided by selected reference atomic clocks for all devices on the internet to synchronize time. Here is a good brief article explaining UTC > https://www.timeanddate.com/time/aboututc.html .
You can compare your node system time with any of the websites displaying UTC, like time.is or timeanddate.com .
The staking wallet can show the time offsets with other connected peers. Here (on the screen below) is how you find the time offsets: HELP > INFORMATION WINDOW > PEERS
These offsets can be used as indication whether your system clock is running in sync, for example if you see most of your peers offsets like + 3~5 seconds, or like - 2~4 seconds, then this can indicate your clock is not properly time synced. Bear in mind this is just an indication, you node should be synced with UTC time. So in case you observe majority of your peers at 0 and some at +⁄- 1~2 sec offset, you should do nothing. However in case you observe most of your peers with larger offset, then stop/exit the wallet, re-sync your system clock with UTC, and run the wallet again.
How to set up system clock synchronize automatically?
First set properly your time zone. If your node is at home, the time zone should be the same as your local time zone. In case your node is an instance on AWS, then the time zone of the instance should be the same as the location of the instance.
Second set your system clock to sync automatically with internet atomic time (UTC).
For Windows go to the screen below and check when last time the system clock was auto-synced.
Then press a few times the "Update Now" button. If necessary change the time server and press the "Update Now" button a few times again (until you see no error displayed).
In case the articles above are not helping, then search google for how to sync your system clock on your specific Operating System.
5. Mini-forks and how to avoid and resolve them
Mini-forks can happen for different reasons. Most often they occur when the system clock of the node is not properly synced with UTC. When your wallet is not displaying correctly the balance (the balance is different from the balance you can see on explorer.hydrachain.org), first thing that should be checked is a problem with system time synchronization.
To quickly solve a mini-fork do the following: 1.Stop the wallet; 2.Re-sync time; 3.Reboot the OS; 4.Start the wallet; 5.Re-index the wallet.
Here is how to re-index the wallet. Go to SETTINGS > OPTIONS > MAIN ⇓
6. Orphan blocks and how to reduce the chance of getting one
Orphan blocks can happen, even if time sync is on track. They happen when two different wallets purely by coincidence find the same winning block. Orphan blocks are more frequent when the total number of nodes on the network is growing larger. Here is a good article where you can learn in detail about orphan blocks > https://jb395official.medium.com/orphan-blocks-june-16-2018-a8f4799dcc2c
To reduce the chance to get orphan blocks is recommended the following:
To see if your wallet has mined an orphan block, open the debug.log file and search upwards from the bottom for “conflicts with wallet transaction”. If your wallet has mined an orphan you will find this log line and a log line a few lines later showing “CT_DELETED” which removes the orphan transaction from the table of transactions in the wallet.
7. Hardware problems and resources shortages
We saw already how slower and unreliable internet or slower node in general could influence a bit the staking performance (orphan blocks). But there could be other more serious problems on your running node, like insufficient RAM or insufficient storage space, or even corrupt hardware.
When you experience unexplained and unexpected freezes of the wallet, or the wallet closing by itself, first thing that may cause this is RAM shortage or storage space shortage. This becomes a problem especially when you run the node on a Raspberry Pi, the SD card can ware fast, pile up errors and the useful free space can shrink. Sometimes the Pi needs some extra RAM and if the swap file is not configured to handle this, the Pi will crush. By default the swap file is set to a minimal size on the Pi (100MB), which is certainly not enough when running the wallet and swap is needed.
Here in this guide you can learn more about swap and swappiness > https://peppe8o.com/set-raspberry-pi-swap-memory/
Yet when you reconfigure the swap file to 2GB and swappiness to 10%, still there could be very little space for the swap file to exist on the SD card. You need a new SD card or a SSD for the Pi. I would recommend this article for adding a SSD to the USB3 port of the Pi4 > https://www.tomshardware.com/how-to/boot-raspberry-pi-4-usb
For Windows users here is some useful info > https://www.windowscentral.com/what-swapfilesys-and-do-i-need-it-my-windows-10-pc . Generally it should work by default and Windows will take care of it. For AWS Windows instance users in case you experience unexpected crashes of the wallet, a possible solution is to configure an alternative Windows instance in a different location.
8. Backups and recovery
In some cases the node can get into a fork and simple reindex would not help to recover. The data stored by the node could also get corrupted for different reasons. Usually the reliable solution in these situations is to resync the node from scratch.
A BACKUP of the "datadir" can be very handy, as it will allow in such cases to resync the node not from the very first block of the blockchain, but from a later point (from the date when the specific backup was made). A BACKUP can save a lot of time when a resync is required.
Here are the steps to resync the node from scratch:
Before you begin be sure to have a copy of your private key at hand!
1.Stop the wallet; 2.Delete (or rename) the datadir; 3.Start the wallet; 4.A new datadir will be created by the node, confim default location of the datadir when asked; 5.Wait for the node to fully sync; 6.Stop the wallet; 7.Make a BACKUP of the datadir; 8.Start the wallet; 9.Import the private key(s).
To make a BACKUP of the "datadir" first you need to know where to find it.
√ On the Raspberry Pi you will find the wallet data directory here: /home/pi/.hydra . It can be set "invisible", right-click with the mouse when inside the upper directory and make hidden directories "visible" to be able to see it.
√ On Linux: ~/.hydra
√ On MacOS: ~/Library/Application Support/hydra
√ On Windows: C:\Users\Administrator\AppData\Roaming\HYDRA or C:\Users\username\AppData\Roaming\HYDRA
Users of the GUI staking wallet can easily find the full path for the location of the datadir in the INFO WINDOW of the wallet:
After finding the directory you will need to make a simple copy/paste of the whole folder. Stop the wallet and copy/paste the "datadir" folder in a location on your node that you will know later how to find. Also you can rename it to something like "/.hydra-25-07-2023" (for Linux) or "\HYDRA-25-07-2023" (for Windows) to know when you created this particular BACKUP. A good habit is to make a BACKUP of your datadir once per month, older backups can be deleted of course, just keep let say the latest two backups.
Here are the steps to make a BACKUP of the datadir:
Before you begin be sure to have a copy of your private key at hand!
1.Get sure your wallet is propely (fully) synced and your balance is correct; 2.Stop the wallet; 3.Make a BACKUP of the datadir; 4.Start the wallet.
You can also download a copy of the the blockchain which is updated until 10th December 2023 from here.
Here are the steps to resync the node from a BACKUP:
Before you begin be sure to have a copy of your private key at hand!
1.Stop the wallet; 2.Delete (or rename) the datadir; 3.Copy/paste the BACKUP datadir into the original location (folder) where the datadir was; 4.Rename the copied BACKUP datadir, so it has the original name of the datadir, i.e. "/.hydra" (for Linux) or "\HYDRA" (for Windows); 5.Start the wallet; 6.Wait for the node to fully sync; 7.Import the private key(s).
Users of cloud services (i.e. AWS, Digital Ocean, etc.) can explore the backups their cloud service providers offer.
9. Things to avoid when running a HYDRA staking wallet
10. Short Guide to confirm your wallet is set up properly for staking
Last updated
