OVH Instance backup - cpanel - QEMU agent - Jailed Shell Print

  • 1

There is a known issue with cpanel and OVH cloud instance backups.

It relates to cpanel jailed shell and virtfs mounts that results in a kernel panic when OVH auto backup runs and makes an instance inaccessible. The OS locks up and OVH intervention is required to reboot an instance. The instance cannot be stopped or restarted from the OVH console.

OVH has a HowTo on how to avoid the situation here:
https://help.ovhcloud.com/csm/en-au-vps-cpanel-auto-backup?id=kb_article_view&sysparm_article=KB0047624

The solution they suggest are:

1. Disable QEMU Guest Agent

or

2. Switch from Jailed Shell to Normal Shell

or

3. Disable cPanel /tmp partition security

Option 3 is a no go by all accounts so the options are 1 or 2.

However in other OVH documentation they indicate that Instance backup relies on the QEMU Guest Agent for reliable backups.

This leaves the sysadmin with option 2 - disabling Jailed Shell.

The documentation for this option is not well documented though.

Here is what we found works:

1. Disable jailed shell for all users in WHM shell manager. Then double check it is done by viewing /etc/passwd

2. Unmount all active VirtFS mounts using this cpanel script:

/usr/local/cpanel/scripts/clear_orphaned_virtfs_mounts --clearall

3. Verify:

grep -i username /proc/mounts

4. Make sure the mounts are cleared:

find /home/virtfs/username/ -type f -links +1 -ls

5. If you are 100% that no users are using jailed then delete

rm -rf /home/virtfs  (careful here because if user symlinks are active in this directory they are hard links and will delete the users files!)

Here is what is missing in the documentation and helped us to get around OVH instance backups from causing a kernel panic:

6. If a user crontab runs then the /home/virtfs/username is recreated and the server will then lock up again on the next instance backup.

The only way we found to avoid this is to delete ALL user crons individually:

crontab -e -u username

and run user crons instead in the root crontab.

When the root cron runs it does re-create an empty /home/virtfs however this does not appear to be a problem.

We also remove "Cron Jobs" as a Feature in all WHM packages so that a user cron cannot be created. These need to be requested to be run in the root crontab instead.

 

This way one can continue to use QEMU Guest Agent for more reliable instance backups.

If you need users to be able to set their own crontabs then you will need to do more research on this.

And if you forget and accidentally create a user cron you will most likely be in deep kernel panic trouble the next time OVH Instance auto-backup runs.

Please contact us if you find a more elegant solution or if this "bug" is resolved. That would be most appreciated.

 

 

 


Was this answer helpful?

« Back
Local Call Cost: 1300 79 4441
Landline: +61 3 9310 2291

OUR TELEPHONE NUMBERS
Call Serversaurus Support on 1300 341 877