Please enable javascript, or click here to visit my ecommerce web site powered by Shopify.

Community Forum > upgrade from early v3.x release issue, system won't come up [SOLVED]

Just a few days back I did the latest update on my production and lab servers and all was well. I had to take the lab server down yesterday for hardware service and when coming back up I have a ton of seg fault errors in dmesg and osnexus won't come up. I see "error 4 in libstdc++.so.6.0.16" in the error output. Any idea what has happened? Is it trying to use a wrong version of the libraries for some reason?

Now I am very concerned about my production box should it go down.

Thanks

November 19, 2013 | Registered CommenterJim Medlin

Here is more detailed output from dmesg:

00000000027e1d48 sp 00007fac1b7fde18 error 15 [111848.248182] qs_service[31249]: segfault at 34ba908 ip
00000000034ba908 sp 00007f16577fde18 error 15 [111908.500981] qs_service[31283]: segfault at 4592d48 ip
0000000004592d48 sp 00007fb4ac842e18 error 15 [111967.760470] qs_service[31317]: segfault at 30c3bb8 ip
00000000030c3bb8 sp 00007f17d7ffee18 error 15 [112028.040252] qs_service[31351]: segfault at 42d4118 ip
00000000042d4118 sp 00007eff9afe8e18 error 15 [112088.326787] qs_service[31388]: segfault at 3578d48 ip
0000000003578d48 sp 00007f500102fe18 error 15 [112147.619875] qs_service[31422]: segfault at 458ad48 ip
000000000458ad48 sp 00007f1ee77fde18 error 15 [112207.869144] qs_service[31457]: segfault at 36a77e8 ip
00000000036a77e8 sp 00007ff256d16e18 error 15 [112268.120526] qs_service[31491]: segfault at 400cd48 ip
000000000400cd48 sp 00007f6c7160ee18 error 15 [112328.370611] qs_service[31525]: segfault at 3f82d48 ip
0000000003f82d48 sp 00007ff4a7146e18 error 15 [112387.652058] qs_service[31559]: segfault at 2869388 ip
0000000002869388 sp 00007f761f7fde18 error 15 [112447.929438] qs_service[31593]: segfault at 3ef2d48 ip
0000000003ef2d48 sp 00007fc092cfee18 error 15 [112508.194011] qs_service[31627]: segfault at 2cd8d48 ip
0000000002cd8d48 sp 00007f511cf35e18 error 15 [112567.457775] qs_service[31661]: segfault at a ip
000000000000000a sp 00007fb9f9a86e18 error 14 in qs_service[400000+1fd8000] [112688.003042] qs_service[31732]: segfault at 41 ip
0000000000000041 sp 00007ff927168e18 error 14 in qs_service[400000+1fd8000] [112747.244082] qs_service[31766]: segfault at 268ed48 ip
000000000268ed48 sp 00007f3980d18e18 error 15 [112807.521832] qs_service[31800]: segfault at 32d2918 ip
00000000032d2918 sp 00007f3f1fffee18 error 15 [112867.785869] qs_service[31834]: segfault at 3e85918 ip
0000000003e85918 sp 00007f11837fde18 error 15 [112928.040876] qs_service[31868]: segfault at 31 ip
0000000000000031 sp 00007fcee6a8ee18 error 14 in qs_service[400000+1fd8000] [112987.296354] qs_service[31902]: segfault at 31fac58 ip
00000000031fac58 sp 00007f53f6f8ee18 error 15 [113047.565029] qs_service[31936]: segfault at 287c808 ip
000000000287c808 sp 00007f82d0c98e18 error 15 [113107.820255] qs_service[31970]: segfault at 30ebd48 ip
00000000030ebd48 sp 00007f0c75348e18 error 15 [113167.089405] qs_service[32004]: segfault at 2a3b918 ip
0000000002a3b918 sp 00007fb732bdce18 error 15 [113227.353040] qs_service[32038]: segfault at 31ffd48 ip
00000000031ffd48 sp 00007f3508b50e18 error 15 [113346.925965] qs_service[32109] general protection ip:7fc2caa5dc71sp:7fc2c355ae20 error:0 in libACE.so.6.1.0[7fc2ca93f000+16f000]
[113407.179573] qs_service[32146]: segfault at 44c2d48 ip
00000000044c2d48 sp 00007f76977fde18 error 15 [113467.453079] qs_service[32180]: segfault at 3819918 ip
0000000003819918 sp 00007fda4d22ae18 error 15 [113527.709598] qs_service[32214]: segfault at 18 ip
00007f74006f2bc3 sp 00007f73fd0528a8 error 6 in libstdc++.so.6.0.16[7f7400654000+e2000]
[113586.980707] qs_service[32248]: segfault at 2ef2918 ip
0000000002ef2918 sp 00007f55dadade18 error 15 [113647.231468] qs_service[32282]: segfault at 2691d48 ip
0000000002691d48 sp 00007f66aa640e18 error 15 [113707.520009] qs_service[32316]: segfault at 4333d48 ip
0000000004333d48 sp 00007f194ddc0e18 error 15 [113766.803473] qs_service[32350]: segfault at 3405bb8 ip
0000000003405bb8 sp 00007f6e0cbffe18 error 15 [113827.057639] qs_service[32384]: segfault at 31 ip
0000000000000031 sp 00007f999f365e18 error 14 in qs_service[400000+1fd8000] [113887.312048] qs_service[32421]: segfault at 361bd48 ip
000000000361bd48 sp 00007fc7fb55ae18 error 15 [113946.579928] qs_service[32455]: segfault at 335bd48 ip
000000000335bd48 sp 00007f2636b6ae18 error 15 [114006.838132] qs_service[32489]: segfault at 4149d48 ip
0000000004149d48 sp 00007f130d994e18 error 15
[114067.099622] qs_service[32523]: segfault at 0 ip (null)
sp 00007ff0bbffee18 error 14 in qs_service[400000+1fd8000] [114127.364960] qs_service[32557]: segfault at 2ed2d48 ip
0000000002ed2d48 sp 00007fd8f355ae18 error 15 [114186.614280] qs_service[32591]: segfault at 3cc6808 ip
0000000003cc6808 sp 00007f002e662e18 error 15 [114246.867824] qs_service[32625]: segfault at 2ed1118 ip
0000000002ed1118 sp 00007f9facf30e18 error 15 [114307.121439] qs_service[32659]: segfault at 3f5fd48 ip
0000000003f5fd48 sp 00007eff497c0e18 error 15 [114366.387183] qs_service[32693]: segfault at 3001d48 ip
0000000003001d48 sp 00007f8f7981ee18 error 15 [114426.646270] qs_service[32727]: segfault at 31ecd48 ip
00000000031ecd48 sp 00007f581f20ae18 error 15 [114486.901387] qs_service[32764]: segfault at 2a38d48 ip
0000000002a38d48 sp 00007fc473ffee18 error 15 [114546.151932] qs_service[333]: segfault at 3c2fd48 ip
0000000003c2fd48 sp 00007f2cc2545e18 error 15 [114606.430031] qs_service[373]: segfault at 36d6d48 ip
00000000036d6d48 sp 00007f0f2ca89e18 error 15 [114666.687634] qs_service[411]: segfault at 3342d48 ip
0000000003342d48 sp 00007ffdaa377e18 error 15 [114726.961729] qs_service[445]: segfault at 2e59808 ip
0000000002e59808 sp 00007f60cbffee18 error 15 [114846.504005] qs_service[517]: segfault at 3f89918 ip
0000000003f89918 sp 00007f93a3ffee18 error 15 [114906.744040] qs_service[555]: segfault at 3b9d918 ip
0000000003b9d918 sp 00007f2c330c9e18 error 15 [118563.964018] qs_service[590]: segfault at ffffffffffffffe7 ip
00007fd67e499e03 sp 00007fd67adfa948 error 4 in libstdc++.so.6.0.16[7fd67e3fc000+e2000]
[118564.330212] qs_service[1447]: segfault at 30be918 ip
00000000030be918 sp 00007f322f423e18 error 15 [118624.567786] qs_service[1494]: segfault at 3bf5bb8 ip
0000000003bf5bb8 sp 00007f2945f7ae18 error 15 [118683.814050] qs_service[1533]: segfault at 31 ip
0000000000000031 sp 00007f986e6dde18 error 14 in qs_service[400000+1fd8000] [118744.095243] qs_service[1569]: segfault at 3885918 ip
0000000003885918 sp 00007f16afb3de18 error 15 [118804.348304] qs_service[1612]: segfault at 2f8e7e8 ip
0000000002f8e7e8 sp 00007f8b7fd22e18 error 15 [118864.624463] qs_service[1656]: segfault at 39cc908 ip
00000000039cc908 sp 00007fa51fffee18 error 15 [118923.881683] qs_service[1690]: segfault at 30ed918 ip
00000000030ed918 sp 00007ffccfb04e18 error 15 [118984.113020] qs_service[1724]: segfault at 3d76d48 ip
0000000003d76d48 sp 00007f4096844e18 error 15 [119044.391359] qs_service[1758]: segfault at 385e7e8 ip
000000000385e7e8 sp 00007fc25227be18 error 15 [119103.669674] qs_service[1792]: segfault at 27d1d48 ip
00000000027d1d48 sp 00007f0a01d9ee18 error 15 [119163.958670] qs_service[1826]: segfault at 3b7bd48 ip
0000000003b7bd48 sp 00007f4bc5362e18 error 15 [119224.236679] qs_service[1861]: segfault at 3813bb8 ip
0000000003813bb8 sp 00007f2c0515ee18 error 15 [122881.438558] qs_service[1898]: segfault at ffffffffffffffe7 ip
00007f1d9ecade03 sp 00007f1d97559948 error 4 in libstdc++.so.6.0.16[7f1d9ec10000+e2000]
[122881.714290] qs_service[2724]: segfault at 18 ip 00007f7064de5bc3 sp 00007f70617458a8 error 6 in libstdc++.so.6.0.16[7f7064d47000+e2000]
[122942.005211] qs_service[2758]: segfault at 36947e8 ip
00000000036947e8 sp 00007fea1b37be18 error 15 [123002.257565] qs_service[2792]: segfault at 31 ip
0000000000000031 sp 00007f90b77fde18 error 14 in qs_service[400000+1fd8000] [123061.518113] qs_service[2826]: segfault at 4f8 ip
00000000000004f8 sp 00007fedce06fdd8 error 14 in qs_service[400000+1fd8000] [126719.748276] qs_service[2863]: segfault at ffffffffffffffe7 ip
00007f0e33b74e03 sp 00007f0e2bffd948 error 4 in libstdc++.so.6.0.16[7f0e33ad7000+e2000]
[126720.107729] qs_service[3685]: segfault at 39c0d48 ip
00000000039c0d48 sp 00007fc1df342e18 error 15 [130377.298278] qs_service[3719]: segfault at ffffffffffffffe7 ip
00007fb1e3d1ce03 sp 00007fb1dbffd948 error 4 in libstdc++.so.6.0.16[7fb1e3c7f000+e2000]
[130377.710554] qs_service[4541]: segfault at 3d91428 ip
0000000003d91428 sp 00007f3530d6ae18 error 15 [133298.524979] qs_service[5327]: segfault at 3920d48 ip
0000000003920d48 sp 00007f8969033e18 error 15 [133315.992763] qs_service[5363]: segfault at f0 ip 00000000000000f0 sp 00007f2101cf0dd8 error 14 in qs_service[400000+1fd8000] ro#

./quantastor status
OS NEXUS QuantaStor Service is stopped.

November 19, 2013 | Registered CommenterJim Medlin

Hi Jim,
We'd like to have a closer look at your system, do you have some time today for a GoToMeeting? Please email us at support@osnexus.com and we'll get it scheduled.
Thanks,
-Steve

November 19, 2013 | Registered CommenterSteve

I just wanted to update the community with the triage results on this ticket. Jim and I looked at the system to day and we found that this was an older v3.5 kernel system and the newer versions of QuantaStor require a v3.8 based kernel. If you are running QuantaStor v3.4 or older be sure to have a look at the Upgrade Guide here which outlines the process for upgrading the QuantaStor kernel which you'll need to do via ssh login or the console before upgrading to the latest QuantaStor version (which will be QuantaStor v3.8 as of this Friday 11/22/13).

http://wiki.osnexus.com/mediawiki/index.php/QuantaStor_Upgrade_Guide#Upgrading_QuantaStor_v3.x_to_v3.5_or_newer

In Jim's case there were some network complications so we're pulling down the kernel packages then installing them manually using dpkg like so but the above Upgrade Guide shows how to do this with apt-get in a few less steps.


wget http://packages.osnexus.com/packages/precise/main/linux-headers-3.8.0-8-quantastor_3.8.0-8.17_amd64.deb
wget http://packages.osnexus.com/packages/precise/main/linux-image-3.8.0-8-quantastor_3.8.0-8.17_amd64.deb
wget http://packages.osnexus.com/packages/precise/main/linux-headers-3.8.0-8_3.8.0-8.17_all.deb
sudo dpkg -i linux-headers-3.8.0-8-quantastor_3.8.0-8.17_amd64.deb linux-image-3.8.0-8-quantastor_3.8.0-8.17_amd64.deb linux-headers-3.8.0-8_3.8.0-8.17_all.deb
sudo update-grub

From there you would reboot and after the system comes up check the kernel version again, it should say v3.8.0-8-quantastor as the kernel version:


uname -a
Linux hat102 3.8.0-8-quantastor #17 SMP Thu Feb 28 01:49:21 PST 2013 x86_64 x86_64 x86_64 GNU/Linux

Worst case scenario in situations like this is to just reinstall QuantaStor onto your boot/system disk from ISO then use the Recovery Manager in the web UI to recover the rest of the meta data configuration. All of your data in your Storage Pools, Storage Volumes, Network Shares and such are auto-detected and imported when you reinstall the QuantaStor OS so the Recovery Manager step isn't necessary to bring all that back. What the Recovery Manager is doing is essentially just recovering a backup of the QuantaStor internal database from one of your storage pools. In this way you can easily recover all the internal configuration data for QuantaStor like user accounts, hosts, host groups, storage assignments, custom roles, etc. The internal database is backed up to the storage pools every hour so each pool has 6 backups which is perhaps overkill but our SQLite based database is typically just 2-10MB in size so it doesn't take up much room. If you're doing a deep dive on the system you'll see a few osn.db.XXX files in the storage pool root directory of each pool, these are the internal database backups.

Best,
-Steve

November 20, 2013 | Registered CommenterSteve