Full node sync failed - Peer stats: all 0, active 0, passive 0

[kkosowsk]

System information

tronprotocol/java-tron:GreatVoyage-v4.5.1
OS & Version: Linux

1. What did you do?

I am trying to sync the full node, it was fine and synced until last approx 3weeks ago. Now i cannot synchronize the node cause after around 10 min I start the node I lost all peers and it never get new peers again. Once i restart node i can connect to peers and sync is starting for maybe 10mins , after that time I can see Peer stats : all 0.

============ Peer stats: all 0, active 0, passive 0


08:51:06.719 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 47.251.48.82 18888 ab01a50667c00a4989c6491470443637cf6ccf537e27d9dd76e68eed0ada93a66a1b0e7474577de041e790ca6c404c3851179667017826cbdcad847ef86eb770
08:51:06.719 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 154.22.123.138 18888 e5c3a8abfba6904a5ce5c2698880b68c1a66c7fc0a62a9d4054d0ba88d0409a4a272d41c7de23875fb30204b961dd8815ad49cd3641056cc6ff85b32e511ddea
08:51:06.720 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 47.254.71.253 18888 e4e4b4c633b0b48d8f773a8890dcdb05acdbc716a0f9fab318242e4faa6c6da421c5912c6803f49c0248e457c84aa16e55647dffd5d368959b237143216e2f8d
08:51:06.720 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 66.85.129.90 18888 1b4befdb54fa9c2f85bceddc0c530f4c48d50dcfeb48e7c56e32b7a196f3da3f4eba138436faa205ce160ed488708d460b126415c03dd9d70667992f27341420
08:51:06.720 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 66.29.144.70 18888 9fb701c02d63c780c1727aa5359c10a59ff7abd1c3414f7aad50f19b70232130b32d487e148e5b43eef7625e4b2d0677a325b8317da9bc8351504bb96fca70d0
08:51:06.720 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 51.81.184.9 30001 2154a7bebe719c951554ead9bb0717e7d90c121ed7d044bc740ab9a18069050a92f091ddfe705028b607cfdb55bda65cd3163b8cdb6d73cdfaad668d0b6862a6
08:51:06.720 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 18.217.244.80 18888 80273d47296ee21a2863e39db3817cce52b4d5517dc9636f66413489dd3915ce3d8bdac2cf4a167469fea38afd86785386990483785841154811d079576038b7
08:51:06.720 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 18.133.82.227 18888 1d3bf76efac4d2d63cc48b67d375ac6556a7c95ebfb2b2984536b0532e71c637982b871246b5594189d6e792941da20a5eab4307261fe5b47f46a5ebb5b69186
08:51:06.720 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 34.96.146.43 18888 37277d8dee14471a9d10e4cce6a970168069950ed7841ada7032ae17f5ecefa237466c390b4ae9f1ea9b1ed625cf5a49b46361182e158f9de46c61150bb306b8
^[[A08:51:10.322 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 47.75.77.31 18888 19a946b2cb31df36efc50f56d3e6dc688a38eb8a96f292b519bf810758dd8e8c9d22099afc63f3c0fe52a6a21f8e4fd156af9f6e79ce5f229eab0b94880b145e
08:51:10.323 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 62.182.80.224 18888 c20ee921692168a33f1cc9c9e3d19735f90de9524965a654d5d24f7ac167371f3e46b54bf4fea03e951771d8c3d7ac7cbbd49da079e2ee087bfa5d20f64e7e4c
08:51:10.323 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 3.218.137.187 18888 100f732f6f2caeb6b42a4d6e9330e11128a82703e1bb056029271ed2aaa3c7e07ffe369e90522aa9228d9117fbefd13caadb0c411e805bc6fe58ca7b2fd16d35
08:51:10.323 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 3.16.25.51 18888 f8761afc4c74737c4224e7b600b392e159592ca8353ac16019bf424cf95323bc2cc8509492f35a04fc5d6eeec1fa81eac70614e03a85a1e634811aa9e0540e8b
08:51:10.323 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 51.81.184.18 30001 d649678a63c88107759b8c47d8580ff467bf47c6f5a319a57a90eb6a7a4231464143bb6640010b08631996ab9a90150144eaa0af0334447e82fac1a25e9379ac
08:51:10.323 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 51.81.184.85 30001 beb998370d62163e36147790af937dee46dd63850366b153121e54229da6cf99dfae72b78c62ba34b731ebcb46f90d25511f81336228680b96577163bbfbfca3
08:51:10.323 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 104.233.194.138 18888 5885a0c2eb3dbac07f17750f89a757fe3a54f864286ff98087bf90eebf9ecf646a0bc3bd50e56c81a0abe8f3f59b9d3b7520e3660f37e7a410257eeb4baeb56c
08:51:10.323 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 18.224.157.94 18888 48696e15d94cf93311fa5a62ec4b46d37608e39276e57b1f24d76a118e499f7853a7ccf77d7d86498e162d22237db0577671abc9aac6b4d1f810bb8f72ab5446
08:51:10.324 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 172.96.160.217 18888 56a98d696ea886a41045b2df5acf08d7967a93bd28e3dee1d74adf9f030988b4f1c9b7f2c5ce9fbdfb60b8b54c200f1cbf8aab696a1b02734667d08a36542f10
08:51:13.925 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 59.188.16.152 18888 0e200e14438aba9ee705478b5b5b380ad8ac3d54fe15f2cd03d564bc3d38907c1c39627500189d542aab6d64b970dcfec44b3fdd5086997b8c80986472b64cfe
08:51:13.926 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 47.89.243.195 18888 cc74a317c716adc8f9b2ffe066dc3f9ddf91c31f05590f51d7a5f77cc46eda5accab6ac7ebdb3646962f85cded9423280e7b94e5d317e008b43f3964caa8ab7f
08:51:13.926 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 15.222.19.181 18888 f36f422dd77f4a721336d646229d17b16cfdbd2485bae85e8d3ef5f10ee8b2cd159218c9bf3e9f58130c13b3ce71e9d467245619781c72fd7ffb470e00bec6be
08:51:13.926 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 34.237.210.82 18888 d00262a6625b98c94bcd28329c27ec69d256aae27b1df3e014326dda5b34ca102591cd4963fe730e9e6d0bf331ce102a495e56a1393c9ba355ca8cc81b2f0c5e
08:51:13.926 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 3.225.25.252 18888 20f9f9abe017bbcb64c292603378e8e43d3870fb5f93e1e8f44ee720f4577ac31d8f81a325f1ee60dd99217700408a4a717f24be89d9f5bbc435cba57025220c
08:51:13.926 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 52.4.20.209 18888 abe1ee2576983464f6598fabf736fd8ec37985087ae779aa03c9f460620efe33f08671f98635a86d4eba37ba8a2d1d586deace95f46f927b9269b789d1f34087
08:51:13.926 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 147.92.38.3 18888 7e78d079501f6c912229de907c977383100043f17a2aaf43ea25ed4a8ad855f997d2b26cddabea462fa30c8a24a6f99952300a7822d79807e7ff8ec1aa4e58a3
08:51:13.926 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 47.91.42.70 18888 1c85d76ba18694ede00241cb024faf91d4529ad808329543b73ecca34782d110ca6cad1e139284c4188a5ee255a77815aa63ba3ef7318c8b9d6aa881bc768e4a
08:51:13.926 INFO  [pool-1-thread-1] [net](PeerClient.java:70) connect peer 18.182.49.245 18888 fb257b5bdaf2be7fd53720d9be43d8d682ee22e9e8d161d7e08e388877cc7101c75e393dab6c7e3b4c6fa0714cd6f630af02d89b774807dee5b7e64a84e624da
08:51:14.545 INFO  [pool-2-thread-1] [net](SyncPool.java:158)

2. What did you expect to see?

that the node is fully synchronized
Currently i am on block {"jsonrpc":"2.0","id":1,"result":"0x270bc9b"}
Node is running on dedicated machine AWS EC2 instance - r6i.xlarge

3. What did you see instead?

Node is not able to synchronize and even if sync is running for short time it's very slow

[Omo-Coc]

Looks like there are other issues, the complete log file helps a lot~

[xxo1shine]

Can you provide a complete log？

[kkosowsk]

Please find the full log file from today, from moment when i started node till node lost all peers

tron.txt

[xxo1shine]

Are you using docker to deploy the node? What command did you use to close the process?

[kkosowsk]

Yes, I am using docker. In this case, I just collected logs without killing the process.
For testing purposes, i can run Tron without docker, but with the same storage?

[xxo1shine]

you can run Tron without docker, with the same storage.

[kkosowsk]

below you can find full from running full node directly on the node

tron_node.log
Running directly on the node i have similar problem with no peers but there is an additional information from java

11:50:29.076 ERROR [TronJClientWorker-2] net Close peer /211.72.186.238:18888, exception caught
java.lang.NullPointerException: null
at org.tron.core.ChainBaseManager.getGenesisBlockId(ChainBaseManager.java:388)

[kkosowsk]

i have managed to run full node directly on ec2 with java-1.8.0-openjdk, but after 10 min i see the same problem

`============ Peer stats: all 0, active 0, passive 0

12:36:07.985 INFO [pool-39-thread-1] net connect peer 46.4.73.115 18888 e67496deadf1d14b63990de2775d8d96090253d12d332be13700fbf550a49616535d32e4fa0216860de72e67c162eff1be8b1ff8fca39332acb4ef6d8197727a
12:36:07.985 INFO [pool-39-thread-1] net connect peer 65.108.125.224 18888 cccf0b044503f8afd6d6ee9a894224357e9a35f089cdaccf396b7b76952c2ef2609f1e21fa633bb19439e5c0f0a79134884f51782386b020edf86e4bfe449e7b
12:36:07.986 INFO [pool-39-thread-1] net connect peer 47.254.145.135 18888 b44e8536d31b8cf81a0d754a3738913a877f5436560754cbdf534a13b4d52769caef4b76b29e314c0c45815a683cc1918050ae0dbc7d8001c2087a2b2a37bc12
12:36:07.986 INFO [pool-39-thread-1] net connect peer 94.130.124.92 28888 a487590584a32a18f2e81190663713bc0970f03bd6ad25bfe189845353701f69fb1bace90661c7dd7a3df7a7dc16fd704b2d467dd55f1dd04ffe0c6561f4fe3a
12:36:07.986 INFO [pool-39-thread-1] net connect peer 47.254.150.38 18888 db52da64a404efd87702a19b7e13fa084ef432b3231468733aa88d43df6256723ecdf48bfec1c4ee906deca0904c7434a88d5981e5acb0230bd7c57dec533488
12:36:07.986 INFO [pool-39-thread-1] net connect peer 3.218.137.187 18888 100f732f6f2caeb6b42a4d6e9330e11128a82703e1bb056029271ed2aaa3c7e07ffe369e90522aa9228d9117fbefd13caadb0c411e805bc6fe58ca7b2fd16d35
12:36:07.986 INFO [pool-39-thread-1] net connect peer 47.251.48.82 18888 ab01a50667c00a4989c6491470443637cf6ccf537e27d9dd76e68eed0ada93a66a1b0e7474577de041e790ca6c404c3851179667017826cbdcad847ef86eb770
12:36:07.986 INFO [pool-39-thread-1] net connect peer 15.222.19.181 18888 f36f422dd77f4a721336d646229d17b16cfdbd2485bae85e8d3ef5f10ee8b2cd159218c9bf3e9f58130c13b3ce71e9d467245619781c72fd7ffb470e00bec6be
12:36:07.986 INFO [pool-39-thread-1] net connect peer 52.4.20.209 18888 abe1ee2576983464f6598fabf736fd8ec37985087ae779aa03c9f460620efe33f08671f98635a86d4eba37ba8a2d1d586deace95f46f927b9269b789d1f34087
12:36:11.587 INFO [pool-39-thread-1] net connect peer 164.132.168.137 18888 ddf13f7179175f5310d356fb3bba41a98c09003595fbcf24decb4d9772247b0cea37a37a56b35c0d2924374e01abd9a9cb69145390186c68cab629868567a4d4
12:36:11.588 INFO [pool-39-thread-1] net connect peer 142.132.206.26 18888 b31a7653e1c9c271182585953d1c6f9e99cd4e300df71b14e1979e78bc803f3cbf924dfaaf59beb274dab7afa66e6827630842156803ef3484c90c5703d7a5e3
12:36:11.588 INFO [pool-39-thread-1] net connect peer 78.46.73.9 60063 0837b3fff94162de013f897efd4231921b7118eaadee0eea424f3f27fb1c9c87878e8c379d3c4dee7dd786f3c3cbb08600268a0b98d3899207221ee480b55b32
12:36:11.588 INFO [pool-39-thread-1] net connect peer 195.123.188.191 18888 571f70b8446085ad51a69776205d5e44daf00f7a3fa4b8cb282588a371605ab2d1ed9923a166314f91909cd0bef598919dfdb817f35ee0171c29b9acbf4a64a3
12:36:11.588 INFO [pool-39-thread-1] net connect peer 65.108.138.106 18888 e42f35fffa6f794e933140ccb1380469b76fa3c50c55622a0f2588b8d1e0969a38137b83e883aaaec575d3ffc606d3acd0b25796b4aff70f81527bc9cf277c8a
12:36:11.588 INFO [pool-39-thread-1] net connect peer 51.81.184.9 30001 b9f78c578b2e7291387fc9082aeb7a07138c5b2994aa15075d7d3fedf5c14e8b444c5becb444f9f81f9d3d81812be854b36a128df0c161d93c037aee89af9b4b
12:36:11.588 INFO [pool-39-thread-1] net connect peer 52.60.187.83 18888 7ff1fee1fe06fae24cbff025624d7f7afde5d7230e806a40093093d188c15b0c82bea0a8f1194eecbbcce8c1055f5ca6344808dd97486051d1a7b48f51a6aa65
12:36:11.588 INFO [pool-39-thread-1] net connect peer 69.61.32.242 18888 73412b08dc7b0b2ad0206dfbc01569fef91da9153ea7d4c97d008c5e077aff146296ee3b1f6654502c8e9e59904d601701118095350c0ec75b74f76bd08205f8
12:36:11.588 INFO [pool-39-thread-1] net connect peer 3.238.111.90 9888 89e511be968c75f86773f60856d693cc31c680bed7ef2478de0f157d197fa9a24f5f4f002bd64aa7580b309828dfb4a21fe5dd451cdfdb8600a800f2d44ec26c`

[xxo1shine]

Building java-tron requires git and Oracle JDK 1.8 to be installed, other JDK versions are not supported yet. Make sure you operate on Linux and MacOS operating systems.

It is possible that you are using the wrong JDK version

[kkosowsk]

tronlog.log
Hello, please see the full log file. Tron running with Oracle JDK 1.8, on amazon ec2 instance with 16vcpu and 32gb memory.
Running Tron tronprotocol/java-tron:GreatVoyage-v4.5.1
cmd used to run Tron:
java -Xmx24g -XX:+UseConcMarkSweepGC -jar FullNode.jar -c main_net_config.conf -d /opt/tron/ --log-config logback.xml

Same problem ============ Peer stats: all 0, active 0, passive 0

[xxo1shine]

After the problem occurs, can you execute this command (jstack -l pid > stack.log) and send me the stack information, thank you!

[ethan1844]

Thanks for your contribution to java-tron, this issue will be closed as no update for a long time.

Please feel free to re-open it if you still see the issue, thanks.