Hydra node health check

This article is all about the health of your HYDRA staking node, and the important things you need to know to win block rewards on the HYDRA blockchain.
1. The probability game and when to expect to win a block reward
You intuitively know how probability works for dice. Probability lets us make calculations about random events like rolling dice (or HYDRA block rewards). We will start considering a die (a single dice) and define success as rolling a “1”. You expect to get a “1” every 6 rolls (“Expected Time” = 6), but also you know you are not guaranteed to get a “1” every 6 rolls because this is a random process.
The probability of an event occurring is given on a scale of 0 (will never happen) to 1.0 (definitely going to happen). The probability of getting a “1” in a single roll of the die is 1/6 or 0.167.
We can use this same approach to calculate the probability of winning a HYDRA block reward for the next block. To make the math easier, let’s consider a wallet with 1000 HYDRA, and a Network Weight of 2,000,000. The wallet software makes the probability of winning a block reward 1,000 / 2,000,000 = 0.0005 = 1⁄2000. What does this mean, compared to the die?
​
Die
HYDRA
Probability of success
0.167
0.0005
Expected time
6 rolls
2000 blocks
Here we see the probability of success for a single trial, getting a “1” in a single roll of the die, or a block reward on the next block, and the Expected Time to the successful result.
This “Expected Time” is the same answer given by the HYDRA wallet for a given wallet weight and network weight, which the wallet calculates based on the difficulty of minting a new block. For our example, the expected time is 2,000 blocks or about 71 hours (2.9 days). In other words the "Expected Time" is another way to express the probability of the event of winning a block reward.
Rolling the die or trying for a HYDRA block reward are called “Independent Trails” in probability, which means each roll of the die or HYDRA block reward is a new probability event with an identical chance of success. The previous results have no influence on the results for independent trials. Contrast this, for example, with double-deck Blackjack, and if you can count cards this is not a game with independent trials.
Let’s look at “Expected Time” more, with a brain teaser. The Expected Time is 2000 blocks (about 2.9 days). After half the Expected Time has elapsed from the last block reward, what is the Expected Time to the next block reward?
Answer: the Expected Time is still 2.9 days!
This has to be true because of independent trials. The "kernel" hash random number generator lives in the moment, has no history, and can only tell who wins the next block reward. “Independent Trials” means each new block reward, like each roll of the die, is independent of any previous results.
To get a more useful answer for “when block reward?” let’s look at this chart representing Expected time vs Probability to win a block reward ⇓
How to read this chart? The horizontal axis gives the multiple of Expected Time. The vertical axis gives the percent chance of a block reward. The red lines show that there is a 63% chance of receiving a block reward at the Expected Time. This means after your wallet receives 100 block rewards, about 63 of them will have arrived before the Expected Time. Some block rewards however would arrive much later than the Expected Time, and blocks arriving 5~9x the Expected Time are still possible.
2. Optimizing UTXOs
One possible optimization of your wallet is to optimize the number and size of UTXOs in your wallet.
UTXOs are like bills in the wallet. Imagine you have one 10$ bill and one 5$ bill (two UTXOs), and you want to spend 3$. Then one of the bills (UTXOs) in the wallet will be spent and you will receive a change. The wallet will determine automatically which UTXO to use, but you can also manually point which UTXO should be used to spend. For this to be possible you need to enable Coin Control features option here: SETTINGS > OPTIONS > WALLET ⇓
When the wallet is staking, for each block a "kernel" hash is calculated. The "difficulty" of the blockchain is weighted for each UXTO and compared to the "kernel" hash.
If
"kernel" hash < "difficulty" x UTXOvalue
then the wallet announces a "mined" block and the winning UXTO (together with the block reward) has to mature again for 500 blocks (approx. 17 hours). The rest of the balance remains available for staking. Obviously if you have all the HYDRAs in one UTXO, if it wins a block it will not be staking until mature again. So It makes sense to split larger UXTOs into smaller ones.
The wallet will split automatically (free of fees) UTXOs larger than 200 HYDRAs and will merge UXTOs smaller than 100 when a new block is won. But you can speed up the process, by manually selecting bigger UTXOs (like 1000 HYDTA or more) and sending part of each UTXO back to yourself. You will receive what you sent plus the rest as "change" (minus the fee for the transaction) and next time you win a block only part of your balance (the winning UTXO + reward) will have to mature again.
3. The importance of time synchronizing the node
The wallet (when staking) calculates the "kernel" hash every block. Then the hash is compared to the weighted "difficulty". If the "kernel" hash is less than the weighted "difficulty", then the wallet announces to its peers a new block with the winning reward. The peers check the validity of the "kernel" hash and can accept it (and "gossip" to their peers) or reject it.
The formula for calculation of the "kernel" hash uses several inputs, that are essential for the randomness of the calculated hash. One of these inputs is the time stamp (in UTC time) of the newly created and announced block. Obviously if the peers do not find the new block to be valid (for many reasons, including the time stamp being "in the past" or "in the future") they will reject it and the wallet will automatically remove the reward from its local copy of the blockchain.
4. Checking if node time is synchronized properly
There are several ways to check if the system clock of the node is in sync with UTC time. Simplest way is to compare to the clock of your smartphone or laptop. The clock on the node and any other device you have should tick in perfect sync to the second. If not, then one of the devices or may be more of them are not synced with UTC time.
What is UTC time (i.e. Coordinated Universal Time)? UTC is provided by selected reference atomic clocks for all devices on the internet to synchronize time. Here is a good brief article explaining UTC > https://www.timeanddate.com/time/aboututc.html .
You can compare your node system time with any of the websites displaying UTC, like time.is or timeanddate.com .
The staking wallet can show the time offsets with other connected peers. Here (on the screen below) is how you find the time offsets: HELP > INFORMATION WINDOW > PEERS
These offsets can be used as indication whether your system clock is running in sync, for example if you see most of your peers offsets like + 3~5 seconds, or like - 2~4 seconds, then this can indicate your clock is not properly time synced. Bear in mind this is just an indication, you node should be synced with UTC time. So in case you observe majority of your peers at 0 and some at +⁄- 1~2 sec offset, you should do nothing. However in case you observe most of your peers with larger offset, then stop/exit the wallet, re-sync your system clock with UTC, and run the wallet again.
How to set up system clock synchronize automatically?
First set properly your time zone. If your node is at home, the time zone should be the same as your local time zone. In case your node is an instance on AWS, then the time zone of the instance should be the same as the location of the instance.
Second set your system clock to sync automatically with internet atomic time (UTC).
For Windows go to the screen below and check when last time the system clock was auto-synced.
Then press a few times the "Update Now" button. If necessary change the time server and press the "Update Now" button a few times again (until you see no error displayed).
Here is an article with detailed step by step instructions for Windows > https://mashtips.com/fix-time-not-updating-syncing-windows10/​
For Linux you can check out this article > https://vitux.com/debian-ntp-time-sync/​
For MacOS > https://www.macinstruct.com/tutorials/synchronize-your-macs-clock-with-a-time-server/​
The latest RaspberryPi OS (Buster) for Raspberry Pi auto-syncs time by default once every 34 minutes. In earlier versions of the OS (like Wheezy or Stertch) the system time was synced just once at booting the system. Then it was useful to use a cron job to resync time every 24 hours. I still keep this cron job on the Pi I run as a stakng node, just to be sure time is always synced, even if the devs of the RaspberryPi OS change again something in the future with an update. Check it here > https://www.thedomainkiosk.com/trends/how-to-protect-the-ssh-session-with-2fa-on-raspberry-pi-3b-plus.html​
In case the articles above are not helping, then search google for how to sync your system clock on your specific Operating System.
5. Mini-forks and how to avoid and resolve them
Mini-forks can happen for different reasons. Most often they occur when the system clock of the node is not properly synced with UTC. When your wallet is not displaying correctly the balance (the balance is different from the balance you can see on explorer.hydrachain.org), first thing that should be checked is a problem with system time synchronization.
To quickly solve a mini-fork do the following: 1.Stop the wallet; 2.Re-sync time; 3.Reboot the OS; 4.Start the wallet; 5.Re-index the wallet.
Here is how to re-index the wallet. Go to SETTINGS > OPTIONS > MAIN ⇓
6. Orphan blocks and how to reduce the chance of getting one
Orphan blocks can happen, even if time sync is on track. They happen when two different wallets purely by coincidence find the same winning block. Orphan blocks are more frequent when the total number of nodes on the network is growing larger. Here is a good article where you can learn in detail about orphan blocks > https://jb395official.medium.com/orphan-blocks-june-16-2018-a8f4799dcc2c​
To reduce the chance to get orphan blocks is recommended the following:
Improve the reliability and speed of your internet connection;
Open/forward properly port 3338 to get more than 8 connections to your node by peers;
Run a faster node;
To see if your wallet has mined an orphan block, open the debug.log file and search upwards from the bottom for “conflicts with wallet transaction”. If your wallet has mined an orphan you will find this log line and a log line a few lines later showing “CT_DELETED” which removes the orphan transaction from the table of transactions in the wallet.
7. Hardware problems and resources shortage
We saw already how slower and unreliable internet or slower node in general could influence a bit the staking performance (orphan blocks). But there could be other more serious problems on your running node, like insufficient RAM or insufficient storage space, or even corrupt hardware.
When you experience unexplained and unexpected freezes of the wallet, or the wallet closing by itself, first thing that may cause this is RAM shortage or storage space shortage. This becomes a problem especially when you run the node on a Raspberry Pi, the SD card can ware fast, pile up errors and the useful free space can shrink. Sometimes the Pi needs some extra RAM and if the swap file is not configured to handle this, the Pi will crush. By default the swap file is set to a minimal size on the Pi (100MB), which is certainly not enough when running the wallet and swap is needed.
Here in this guide you can learn more about swap and swappiness > https://peppe8o.com/set-raspberry-pi-swap-memory/​
Yet when you reconfigure the swap file to 2GB and swappiness to 10%, still there could be very little space for the swap file to exist on the SD card. You need a new SD card or a SSD for the Pi. I would recommend this article for adding a SSD to the USB3 port of the Pi4 > https://www.tomshardware.com/how-to/boot-raspberry-pi-4-usb
For Windows users here is some useful info > https://www.windowscentral.com/what-swapfilesys-and-do-i-need-it-my-windows-10-pc . Generally it should work by default and Windows will take care of it. For AWS Windows instance users in case you experience unexpected crashes of the wallet, a possible solution is to configure an alternative Windows instance in a different location.
8. Things to avoid when running a HYDRA staking wallet
Do not stake on two identical wallets (with same public HYDRA address), this can cause issues with winning blocks. Of course you can have one wallet staking and another as backup. And when you perform sort of maintenance on the main wallet you could stake on the backup wallet (to reduce offline time).
Do not reboot without good reason. Stopping the wallet will cause all established connections to be lost and they will have to be built again, when the wallet is started. Of course if you think you tried everything to set up properly the node, and still are in very bad luck zone for winning a block, then a reboot of the system could help. Rebooting is recommended once per month, together with update/upgrade of the OS (for Linux), or in case Windows needs a reboot to complete an update.
Do not disconnect peer nodes with some time offset. They may re-sync their time with UTC and be on track again. In case a peer node stays with a considerable offset for a longer time, your node will automatically disconnect from it and temporarily ban it.
Do not leave for a very long time your staking node unattended. This is a live server and things can happen, like for example Windows updated and changed somethings on your node, or Computer clock stopped regularly syncing time which lead to a big time drift of your system clock.
9. Short Guide to confirm your wallet is set up properly for staking
Wallet updated to the latest version
Synced to the latest block
Unlocked for staking only
Staking is enabled
Computer clock set correctly