Please enable javascript, or click here to visit my ecommerce web site powered by Shopify.

Community Forum > Questions regarding HA and Gluster

Hi all,

I'm trying to setup a Gluster high-availability scale-out NAS with QuantaStor. The grid management IP address should be high-available, too. But I have some issues with this setup:

1. The virtual management IP address is pingable but I can't reach the web UI. I don't see the virtual interface on one of the nodes.
2. Neither the virtual management IP address nor the virtual IP address of the Gluster volume can be moved manually to another node. The errors are: "Failed to move corosync resource 'gv4470b500' to system 'quantastor3' [err=1]" and "Failed to move corosync resource 'svf6073c00' to system 'quantastor3' [err=1]"

And then I had an issue with the Gluster bricks: 2 bricks on one storage node were shown as 0 TB after this node had to be restarted. So I had to recreate the Gluster volume. I have 3 virtual nodes on 3 separate ESXi 6.5 hosts. Each node has 2 storage pools located on 2 separate virtual disks. I set up a Gluster volume with the 6 storage pools using erasure coding 2+1. Is this setup OK? Or is this a possible cause for the faulty bricks on one node?

Please help me. Thank you.

Kind Regards

Stefan

December 15, 2017 | Registered CommenterStefan Mössner

Hi all,

any ideas why the manual failover of the the cluster resources isn't working? And what about my question regarding the storage pool design? I can't go further with testing this very interesting storage solution with a really intuitive and easy to understand web UI.

Do I need to set up IGMP snooping and querier on the network switches for using QuantaStor as a grid with HA and data distribution?

Thank you

Stefan

December 19, 2017 | Registered CommenterStefan Mössner

Hello Stefan,

We need to get your HA VIF working first, Please review the document below for details on configuring the VIF:

High-Availability Virtual Network Interface Configuration
https://wiki.osnexus.com/index.php?title=Clustered_HA_SAN/NAS_with_iSCSI/FC_SAN_back-end#High-Availability_Virtual_Network_Interface_Configuration

Also can you verify that you can get to the WebUI on "eth0"?
You need to verify that you have a static IP and your subnet is set correctly for the VIF, this should work the same as your "eth0" interface to connect to the WebUI and should not need special switch configurations..

Your VIF and Heartbeat links must be working correctly before your failovers will function correctly so network connectivity must be verified first.

Here are some additional documents on setting up Gluster:

https://wiki.osnexus.com/index.php?title=Scale-out_NAS_(Gluster_based)_Storage_Setup

Failovers will

December 19, 2017 | Registered CommenterRobert Staffeld

Stefan,
Could you share more about your use case for the scale-out NAS configuration? If it's for archive purposes great, but if it is for VMs or databases it's not a good option. We're planning to upgrade QuantaStor to a newer version of Gluster in Q1 which may enable us to adjust our guidance but in our testing over the years it has not been a good performer and has various feature gaps and architectural issues that make it ill suited for VMs and databases.
Best,
-Steve

December 19, 2017 | Registered CommenterSteve

Hi,

@Steve: I want to use the system as file server for SMB and NFS. I don't want to run virtual machines or databases on it.
@Robert: I will have a look once again at my configuration.

Thank You

Stefan

December 20, 2017 | Registered CommenterStefan Mössner

Hi all,

something is wrong with my first Grid storage node: Again the Gluster bricks of this node are at 0 B free space. And this node can't be updated. There's an error message regarding a not running qs service. Or in the upgrade manager there's only the previous release and no update available.

The virtual management grid interface and the web UI are now online and available. But I can't find the option to manually failover the virtual Interfaces. This is strange.

What can I do? Should I reinstall the node with the 0 B Gluster bricks? Maybe there's a system issue causing the other problems with failover etc.

Kind Regards

Stefan

December 20, 2017 | Registered CommenterStefan Mössner

Hi all,

the issue with the node was a wrong default gateway: Although all nodes were configured identically the one node used the default gateway of the internal cluster interface. Now I deleted the default gateway setting on the internal cluster interfaces of all nodes. All nodes did update to the latest release.

But now I'm unable to setup the Gluster for my scale-out NAS. I can't delete the settings for a new setup. How can I reset the Gluster settings to set up a new Gluster volume?

Kind Regards

Stefan

December 23, 2017 | Registered CommenterStefan Mössner

Hi all again,

with removing /var/lib/glusterd (rm -r /var/lib/glusterd/) and starting GlusterFS Service (service glusterfs-server start) on all nodes all old Gluster settings were deleted and I was able to setup a new GlusterFS. But again on one node the bricks show 0B free space. This isn't corrcect because it's a new virtual disk without any data. The storage pool for these disks has the correct size and on the console the size is at correct size, too.

How can I resolve this issue?

I wish you a Merry Christmas and a happy New Year

Stefan

December 23, 2017 | Registered CommenterStefan Mössner

Hi all,

and as attachment to my former post, I'm unable again to failover the virtual interfaces of the grid management and for the GlusterFS.

I hope you can help me resolving these issues.

Thank You

Stefan

December 23, 2017 | Registered CommenterStefan Mössner

Hi all,

after restarting the QuantaStor VMs for upgrading the ESXi hosts more gluster bricks showed free space of 0B. Then I saw that the storage pools are at 0B, too. After rescanning the storage pools all gluster bricks showed the right storage size again.

Now I'm able to manually failover the grid management IP Interface to other nodes. It worked second times. But the manual failover of the Gluster IP Interface doesn't work.

I'm hoping that this issue could be solved because QuantaStor seems to be the best storage system for my purposes. I tested a lot of other systems (Compuverde, EMC Isilon etc.) but either the network requirements are very high or the support of the system is poor.

Kind Regards

Stefan

December 24, 2017 | Registered CommenterStefan Mössner

Hi Stefan,
Thanks for the feedback, glad you were able sort most of it out. The Gluster IP not movable is interesting, could you email us at support@osnexus.com so we can have a support engineer take a closer look on Wed or Thursday? Also, please send us logs from each system by right-clicking on each storage system in the tree view then selecting 'Send Log Report..'.
Happy holidays,
-Steve

December 24, 2017 | Registered CommenterSteve

With regard to removing everything under '/var/lib/glusterd' you did the right thing.. it's sort of the large hammer and should be avoided if you have data in the volumes but in this case, where you need to reset everything, it's good.
With regard to the bricks reporting 0 size, there's a delay between when the volume is created and when the grid has that attribute updated across all the bricks/volumes. If it doesn't update within some number of minutes then it could be some other configuration issue. Glad you were able to sort the default gateway issue, that's one of the things we check for in our pre-go-live process. In general, the gateway IP should only be set on one network interface on each node.
Best,
-Steve

December 24, 2017 | Registered CommenterSteve

Hi Steve,

I sent you the log files and an email as requested.

Thank You

Stefan

December 26, 2017 | Registered CommenterStefan Mössner

Thanks Stefan,
Also note that we'll be upgrading QuantaStor to Gluster 3.10 pretty soon. It might make it into a 4.4.x update release in a couple weeks else it'll be in 4.5 next month.
Best,
-Steve

January 23, 2018 | Registered CommenterSteve

Hello Steve,

thanks for this update. Today I found out that a new version 4.4.2.004-1 is available since January, 19th 2018. And in the release notes I see that you fixed the issue regarding the failover of the Gluster VIF. After updating my nodes I can confirm that it's now working well to move the Gluster VIF :-) But you have to delete and recreate this Interface at first for a working failover mechanism. It would be nice to add this important information to the release notes, too.

With 'crm configure show' on the CLI I can see that you changed the location parameter for the Interface gv<...> by replacing -inf to 100 and 150 for the preferred master. That's what I found out and that's what you fixed, right?

For now Gluster 3.7.15 is active (CLI command 'glusterd --version'). Why won't you upgrade Gluster to the actual long term release 3.12 (https://www.gluster.org/announcing-glusterfs-release-3-12-0-long-term-maintenance/)?

Kind Regards

Stefan

January 26, 2018 | Registered CommenterStefan Mössner

Hi Stefan,
You have it right, as I understand it we weighted the resource incorrectly so it couldn't move to the other appliance. Engineering made one of the weights 150 so it would have preference for the original node it was associated with and fixed the ones set to -∞.

>For now Gluster 3.7.15 is active (CLI command 'glusterd --version'). Why won't you upgrade Gluster to the actual long term release 3.12 (https://www.gluster.org/announcing-glusterfs-release-3-12-0-long-term-maintenance/)?

The latest release available on Ubuntu 14.04/trusty is 3.10.9 so we're moving up to that here in the next update. After that we will be upgrading QuantaStor to 16.04/xenial and that will open the way to moving up to 3.12 at the end of this quarter.

Best,
-Steve

January 26, 2018 | Registered CommenterSteve

Hi Steve,

thank you very much for this quick answer and clarification.

Your support is really great. You're doing a great job.

Kind Regards

Stefan

January 26, 2018 | Registered CommenterStefan Mössner