Developers Club geek daily blog

3 years, 3 months ago
In this article I will step by step describe setup of automatic backup and example of recovery of the virtual computers working at the ESX (i) platform by means of free scripts of ghettoVCB. I will be accented on the ESXi 5.x version, but the same means work and at versions 3.5-6.x, the truth for early versions of setup differ a little. The backup will be made on the NFS server. The report on work will be sent to mail. During backup the picture (snapshot) of the virtual computer becomes (including working), VMDK disks of the machine remain and the picture is removed.
The ghettoVCB project is perfectly documented, but in the course of implementation there were nuances which have poured out in this instruction. I hope, article will be useful to the beginning administrators.



  1. Backup setup
  2. Saving of configuration of the ESXi server
  3. Recovery of the machine from backup copy
  4. Links


Backup setup


First of all prepare the NFS server to taste where will be it is made backups. In my case it is FreeNAS (Freebsd 9.3) and ZFS dataset with the included dedublikation and compression that significantly saves place. Setup on ESXi the server is made on SSH from the root user. It is possible and under other user with the administrative rights, but then it will not turn out to start scripts for check from the console. Let's start.

1. Access to the server on SSH is necessary for setup, we include through vSphere client:
Configuration -> security profile -> properties -> SSH


2. We take scripts from github of repository and we place contents on the server. Surely we put execution bit, differently scripts will not work:
# chmod u+x /ghettoVCB-master/ghettoVCB.sh
# chmod u+x /ghettoVCB-master/ghettoVCB-restore.sh


3. Let's Bekapit weekly, with cycle of 4 weeks. Overdue backups will be removed. We register the corresponding config-file:
# cat /ghettoVCB-master/4week.conf
VM_BACKUP_VOLUME=/vmfs/volumes/backup
DISK_BACKUP_FORMAT=thin
#количество хранимых бэкапов
VM_BACKUP_ROTATION_COUNT=4
POWER_VM_DOWN_BEFORE_BACKUP=0
ENABLE_HARD_POWER_OFF=0
ITER_TO_WAIT_SHUTDOWN=3
POWER_DOWN_TIMEOUT=5
ENABLE_COMPRESSION=0
ALLOW_VMS_WITH_SNAPSHOTS_TO_BE_BACKEDUP=0
ENABLE_NON_PERSISTENT_NFS=1
UNMOUNT_NFS=1
# NFS сервер, куда будут лететь бэкапы
NFS_SERVER=10.1.3.101
NFS_MOUNT=/mnt/backup/vmware
NFS_LOCAL_NAME=backup
# папка на NFS сервере (для каждого ESXi своя)
NFS_VM_BACKUP_DIR=autobackup/vm01
SNAPSHOT_TIMEOUT=15
#настройки логирования в почту
EMAIL_LOG=1
EMAIL_SERVER=mail.core.local
EMAIL_SERVER_PORT=25
EMAIL_DELAY_INTERVAL=1
EMAIL_TO=admins@mail.local
EMAIL_FROM=ghettoVCB@vm01.core.local
WORKDIR_DEBUG=0
VM_SHUTDOWN_ORDER=
VM_STARTUP_ORDER=
Where keyword parameters:
  • VM_BACKUP_VOLUME — way on the ESXi server where nfs the section will be mounted;
  • DISK_BACKUP_FORMAT=thin — the VMDK format of disk which will be created at backup;
  • VM_BACKUP_ROTATION_COUNT — quantity of the stored backups;
  • POWER_VM_DOWN_BEFORE_BACKUP=0 — we do not switch off the machine before backup;
  • ENABLE_COMPRESSION=0 — we do not press data, we leave it on zfs;
  • ALLOW_VMS_WITH_SNAPSHOTS_TO_BE_BACKEDUP=0 – backup of machines from snapshotama (the latests version of scripts it are able, but it is not necessary to us);
  • ENABLE_NON_PERSISTENT_NFS=1 — nfs disk will be will be connected only for the period of backup;
  • UNMOUNT_NFS=1 — and after will be disconnected;
  • NFS_SERVER and NFS_MOUNT – disk nfs coordinates;
  • NFS_LOCAL_NAME — name which will be appropriated to the connected array (datastores identification);
  • NFS_VM_BACKUP_DIR — way where there will be copies (relatively VM_BACKUP_VOLUME);
  • EMAIL_LOG=1 — we include sending the report by mail;
  • EMAIL_* — settings of parameters of mail.

If on the ESXi server already there is disk connected by nfs with the same coordinates (server/way), the disk will not be connected.

The body of the letter forms during script operating time, and then is sent by the utility nc. It can cause failure from the e-mail server, with argument"Recipient address rejected: Improper use of SMTP command pipelining". It is necessary to exclude the corresponding check for the ESXi servers (at postfix-and it will be reject_unauth_pipelining).

4. We create the list of machines which need to be bekapit. It is possible to receive it by means of team esxcli vm process list. Each line of the list — one machine:
# cat /ghettoVCB-master/week.list
win7
win10
vCenterUpdate
If some plans of backups are necessary, with different frequency and parameters — we create necessary quantity of configurations.

5. We configure cron on execution of periodic task:
# cat /var/spool/cron/crontabs/root
#min hour day mon dow command
#……………………….

3    18   *   *   6     /ghettoVCB-master/ghettoVCB.sh -g /ghettoVCB-master/4week.conf -f /ghettoVCB-master/week.list > /var/log/ghettoVCB-backup-week-$((($(date +\%d)-1)/7+1)).log

The system time goes to UTC therefore it is necessary to do the correction on the current time zone. In my case of +7 hours — the backup will be started on Sunday at 1 o'clock in the morning. It is obligatory to write Log (or to transport in / dev/null), differently at buffer overflow, taken away for the user, the script can hang up. Construction $((($(date +\%d)-1)/7+1)) issues number of week in month, thus we will not produce garbage.

6. We restart cron:
# kill $(cat /var/run/crond.pid)
# crond


7. For sending mail it is necessary to add permission to the originating traffic to ESXi server firewall:
  • we create the file of setup with contents:
    # cat /etc/vmware/firewall/email.xml
    <ConfigRoot>
      <service>
        <id>email</id>
        <rule id="0000">
          <direction>outbound</direction>
          <protocol>tcp</protocol>
          <porttype>dst</porttype>
          <port>25</port>
        </rule>
        <enabled>true</enabled>
        <required>false</required>
      </service>
    </ConfigRoot>
    

  • we re-read rules of firewall it is checked:
    # esxcli network firewall refresh
    # esxcli network firewall ruleset list | grep email
    email                  true
    # nc mail.core.local 25
    220 mail.core.local SMTP Postfix
    quit
    221 2.0.0 Bye
    



It is possible to start test check of backup of the machine with the name "vCenterUpdate" as follows:
# /ghettoVCB-master/ghettoVCB.sh -g /ghettoVCB-master/4week.conf -d dryrun -m vCenterUpdate
Output:
# /ghettoVCB-master/ghettoVCB.sh -g /ghettoVCB-master/4week.conf -d dryrun -m vCenterUpdate
Logging output to "/tmp/ghettoVCB-2015-08-18_07-15-08-23516502.log" ...
2015-08-18 07:15:09 -- info: ============================== ghettoVCB LOG START ==============================

2015-08-18 07:15:09 -- info: CONFIG - USING GLOBAL GHETTOVCB CONFIGURATION FILE = /ghettoVCB-master/4week.conf
2015-08-18 07:15:09 -- info: CONFIG - VERSION = 2015_05_06_1
2015-08-18 07:15:09 -- info: CONFIG - GHETTOVCB_PID = 23516502
2015-08-18 07:15:09 -- info: CONFIG - VM_BACKUP_VOLUME = /vmfs/volumes/backup/autobackup/vm01
2015-08-18 07:15:09 -- info: CONFIG - ENABLE_NON_PERSISTENT_NFS = 1
2015-08-18 07:15:09 -- info: CONFIG - UNMOUNT_NFS = 1
2015-08-18 07:15:09 -- info: CONFIG - NFS_SERVER = 10.1.3.101
2015-08-18 07:15:09 -- info: CONFIG - NFS_VERSION = nfs
2015-08-18 07:15:09 -- info: CONFIG - NFS_MOUNT = /mnt/backup/vmware
2015-08-18 07:15:09 -- info: CONFIG - VM_BACKUP_ROTATION_COUNT = 4
2015-08-18 07:15:09 -- info: CONFIG - VM_BACKUP_DIR_NAMING_CONVENTION = 2015-08-18_07-15-08
2015-08-18 07:15:09 -- info: CONFIG - DISK_BACKUP_FORMAT = thin
2015-08-18 07:15:09 -- info: CONFIG - POWER_VM_DOWN_BEFORE_BACKUP = 0
2015-08-18 07:15:09 -- info: CONFIG - ENABLE_HARD_POWER_OFF = 0
2015-08-18 07:15:09 -- info: CONFIG - ITER_TO_WAIT_SHUTDOWN = 3
2015-08-18 07:15:09 -- info: CONFIG - POWER_DOWN_TIMEOUT = 5
2015-08-18 07:15:09 -- info: CONFIG - SNAPSHOT_TIMEOUT = 15
2015-08-18 07:15:09 -- info: CONFIG - LOG_LEVEL = dryrun
2015-08-18 07:15:09 -- info: CONFIG - BACKUP_LOG_OUTPUT = /tmp/ghettoVCB-2015-08-18_07-15-08-23516502.log
2015-08-18 07:15:09 -- info: CONFIG - ENABLE_COMPRESSION = 0
2015-08-18 07:15:09 -- info: CONFIG - VM_SNAPSHOT_MEMORY = 0
2015-08-18 07:15:09 -- info: CONFIG - VM_SNAPSHOT_QUIESCE = 0
2015-08-18 07:15:09 -- info: CONFIG - ALLOW_VMS_WITH_SNAPSHOTS_TO_BE_BACKEDUP = 0
2015-08-18 07:15:09 -- info: CONFIG - VMDK_FILES_TO_BACKUP = all
2015-08-18 07:15:09 -- info: CONFIG - VM_SHUTDOWN_ORDER =
2015-08-18 07:15:09 -- info: CONFIG - VM_STARTUP_ORDER =
2015-08-18 07:15:09 -- info: CONFIG - RSYNC_LINK = 0
2015-08-18 07:15:09 -- info: CONFIG - EMAIL_LOG = 1
2015-08-18 07:15:09 -- info: CONFIG - EMAIL_SERVER = mail.core.local
2015-08-18 07:15:09 -- info: CONFIG - EMAIL_SERVER_PORT = 25
2015-08-18 07:15:09 -- info: CONFIG - EMAIL_DELAY_INTERVAL = 2
2015-08-18 07:15:09 -- info: CONFIG - EMAIL_FROM = ghettoVCB@vm02.core.local
2015-08-18 07:15:09 -- info: CONFIG - EMAIL_TO = admins@mail.local
2015-08-18 07:15:09 -- info: CONFIG - WORKDIR_DEBUG = 0
2015-08-18 07:15:09 -- info:
2015-08-18 07:15:10 -- dryrun: ###############################################
2015-08-18 07:15:10 -- dryrun: Virtual Machine: vCenterUpdate
2015-08-18 07:15:10 -- dryrun: VM_ID: 588
2015-08-18 07:15:10 -- dryrun: VMX_PATH: /vmfs/volumes/ds3524_ds/vCenterUpdate/vCenterUpdate.vmx
2015-08-18 07:15:10 -- dryrun: VMX_DIR: /vmfs/volumes/ds3524_ds/vCenterUpdate
2015-08-18 07:15:10 -- dryrun: VMX_CONF: vCenterUpdate/vCenterUpdate.vmx
2015-08-18 07:15:10 -- dryrun: VMFS_VOLUME: ds3524_ds
2015-08-18 07:15:10 -- dryrun: VMDK(s):
2015-08-18 07:15:10 -- dryrun:  vCenterUpdate.vmdk      40 GB
2015-08-18 07:15:10 -- dryrun: INDEPENDENT VMDK(s):
2015-08-18 07:15:10 -- dryrun: TOTAL_VM_SIZE_TO_BACKUP: 40 GB
2015-08-18 07:15:10 -- dryrun: ###############################################

2015-08-18 07:15:10 -- info: ###### Final status: OK, only a dryrun. ######

2015-08-18 07:15:10 -- info: ============================== ghettoVCB LOG END ================================



To start in manual for the list of machines:
# /ghettoVCB-master/ghettoVCB.sh -g /ghettoVCB-master/4week.conf -f /ghettoVCB-master/week.list

Each copy of machines will be is stored in directories, look:
autobackup/vm01/VMNAME/VMNAME-FULL_DATE/

And to be machine disk in the *.vmdk format and the file of configuration of the machine *.vmx.

Saving of configuration of the ESXi server


The ESXi settings made above will live before the first reset. For saving of configuration it is necessary to make some more actions.

1. Let's add to boot script of command for change of settings cron-and:
# cat /etc/rc.local.d/local.sh
#...................
/bin/kill $(cat /var/run/crond.pid)
/bin/echo "3    18   *   *   6     /ghettoVCB-master/ghettoVCB.sh -g /ghettoVCB-master/4week.conf -f /ghettoVCB-master/week.list > /var/log/ghettoVCB-backup-week-$((($(date +\%d)-1)/7+1)).log" >> /var/spool/cron/crontabs/root
/bin/busybox crond


2. For saving of settings of firewall we will make own VIB packet and we will set it on the server. For this purpose we use the utility of VIB Author. Unfortunately, it only for 32-bit architecture therefore to me lxc was necessary to use the container. At installation can swear terribly on dependence of look:
# rpm -ivh vmware-esx-vib-author-5.0.0-0.0.847598.i386.rpm
error: Failed dependencies:
        libc.so.6()(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
        libc.so.6(GLIBC_2.2.5)(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
        libc.so.6(GLIBC_2.3)(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
        libc.so.6(GLIBC_2.4)(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
        libdl.so.2()(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
        libdl.so.2(GLIBC_2.2.5)(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
        libpthread.so.0()(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386
        libpthread.so.0(GLIBC_2.2.5)(64bit) is needed by vmware-esx-vib-author-5.0.0-0.0.847598.i386

But it it does not matter, and key rpm --nodeps will rescue us.

We prepare tree of directories for assembly of VIB of packet:
# mkdir -p stage/payloads/payload1/etc/vmware/firewall/ 

Also we create two files. The first — the description of our packet:
# cat stage/descriptor.xml
<vib version="5.0">
  <type>bootbank</type>
  <name>mailFirewall</name>
  <version>5.0.0-0.0.1</version>
  <vendor>Lelik.13a</vendor>
  <summary>Custom VIB from Lelik.13a</summary>
  <description>Adds custom firewall rule for mail sender to ESXi host</description>
  <relationships>
    <depends>
    </depends>
    <conflicts/>
    <replaces/>
    <provides/>
    <compatibleWith/>
  </relationships>
  <software-tags>
  </software-tags>
  <system-requires>
    <maintenance-mode>false</maintenance-mode>
  </system-requires>
  <file-list>
  </file-list>
  <acceptance-level>community</acceptance-level>
  <live-install-allowed>true</live-install-allowed>
  <live-remove-allowed>true</live-remove-allowed>
  <cimom-restart>false</cimom-restart>
  <stateless-ready>true</stateless-ready>
  <overlay>false</overlay>
  <payloads>
    <payload name="payload1" type="vgz"></payload>
  </payloads>
</vib>
It is possible to look at the detailed instruction to parameters on the utility site.

And the second file — email.xml which contents are stated above. And there is it will be on the way stage/payloads/payload1/etc/vmware/firewall/email.xml, where way after"payload1"– desirable way on the target server.

We collect VIB packet:
# vibauthor -C -t stage -v mailFirewall.vib

Also we check its contents
# vibauthor -i -v mailFirewall.vib
**** Info for VIB: mailFirewall.vib ****
VIB Format:             2.0.0
VIB ID:                 Lelik.13a_bootbank_mailFirewall_5.0.0-0.0.1
VIB Type:               bootbank
Name:                   mailFirewall
Version:                5.0.0-0.0.1
Vendor:                 Lelik.13a
Summary:                [Fling] Custom VIB from Lelik.13a
Description:            Adds custom firewall rule for mail sender to ESXi host
Creation Date:          2015-08-12 09:47:07.199735+00:00
Provides:
        mailFirewall = 5.0.0-0.0.1
Depends:
Conflicts:
Replaces:
        mailFirewall << 5.0.0-0.0.1
Software Tags:          []
MaintenanceMode:        remove/update: False, installation: False
Signed:                 False
AcceptanceLevel:        community
LiveInstallAllowed:     True
LiveRemoveAllowed:      True
CimomRestart:           False
StatelessReady:         True
Overlay:                False
Payloads:
  Name            Type        Boot Size        Checksums
  payload1        vgz         0    347         sha-256 69aa821faa4ccd5a5e34e487ecf6049aa6bf55652ffffbfaae1257e40610c405
                                               sha-1 4d77e529c8da74e82d4aa4e816bcf193e29ab8de


At need, can use my packet (at own risk).

3. We copy our packet on the ESXi server, we set and we check that has turned out:
# esxcli software vib install -v /tmp/mailFirewall.vib -f
# esxcli software vib list | grep mail
# esxcli network firewall refresh
As the packet adds the files to system, use of key is necessary"-f".
And just in case, we re-read rules of firewall.

Thus it is possible to collect and record also other useful settings of the server.

4. And in end, we start in manual backup of settings of the ESXi server:
# /sbin/auto-backup.sh


Recovery of the machine from backup copy


At once it is necessary to consider that the machine which reservation was made on hot, after recovery will be as after krash — the data loss in the machine is possible.

1. We connect storage with backup on target ESXi the server on NFS, or simply we copy data on ssh there.

2. We create configuration file of "vms_to_restore" of such look:
# cat /ghettoVCB-master/vms_to_restore
#"<DIRECTORY or .TGZ>;<DATASTORE_TO_RESTORE_TO>;<DISK_FORMAT_TO_RESTORE>"
# DISK_FORMATS
# 1 = zeroedthick
# 2 = 2gbsparse
# 3 = thin
# 4 = eagerzeroedthick
# e.g.
"/vmfs/volumes/restore/autobackup/vm01/vCenterUpdate/vCenterUpdate-2015-08-13_07-55-50/;/vmfs/volumes/local_vm01/;3;vCenterUpdate-restore"
Where one after another through ";":
  • way where the recovered machine lies;
  • way where to recover the machine (the directory under it will be created);
  • machine disk type for recovery;
  • new name of the machine (not necessarily).


3. We start test run:
# /ghettoVCB-master/ghettoVCB-restore.sh  -c /ghettoVCB-master/vms_to_restore -d 1

And fighting:
# /ghettoVCB-master/ghettoVCB-restore.sh  -c /ghettoVCB-master/vms_to_restore -l /var/log/vms-restore.log
################## Restoring VM: vCenterUpdate-restore  #####################
Start time: Fri Aug 14 06:05:06 UTC 2015
Restoring VM from: "/vmfs/volumes/restore/autobackup/vm01/vCenterUpdate/vCenterUpdate-2015-08-13_07-55-50/"
Restoring VM to Datastore: "/vmfs/volumes/local_vm01/" using Disk Format: "thin"
Creating VM directory: "/vmfs/volumes/local_vm01//vCenterUpdate-restore" ...
Copying "vCenterUpdate.vmx" file ...
Restoring VM's VMDK(s) ...
Updating VMDK entry in "vCenterUpdate-restore.vmx" file ...
Destination disk format: VMFS thin-provisioned
Cloning disk '/vmfs/volumes/restore/autobackup/vm01/vCenterUpdate/vCenterUpdate-2015-08-13_07-55-50//vCenterUpdate.vmdk'...
Clone: 100% done.
Registering vCenterUpdate-restore ...
12
End time: Fri Aug 14 06:11:19 UTC 2015

4. We rejoice.

Second option. As the backup represents machine dump with configuration file, it is possible simply:

1. To copy it somewhere on ESXi the server.

2. To correct the file of configuration of the machine (*.vmx), having changed name fields and locations of disk of the machine:
displayName = vCenterUpdate-restore
extendedConfigFile = "vCenterUpdate-restore.vmxf"
scsi0:0.fileName  = "vCenterUpdate-restore-0.vmdk"
sched.swap.derivedName = "vCenterUpdate-ff0c3749.vswp"
It is possible for way to files (and it is necessary) to specify concerning machine directory.

3. Through vSphereClient we go to storage:
Configuration -> storage -> ПКМ -> "browse datastore"

also we add the new machine to the list:
ПКМ -> "Add to inventory" на файле *.vmx

4. If the recovery server another, in settings of the machine we change "Network Connection".

5. At the first start it will ask, from where the machine was drawn, it is necessary to answer that have transferred.
If to answer that cloned — the address of the network interface card will change unique data, including mac.

Here in general and all. Scripts of ghettoVCB support still other interesting parameters and options of copying so it is useful to read documentation. The method is far from ideal but if there is a wish cheap but good, is quite efficient.

Thanks for attention.

Links



This article is a translation of the original post at habrahabr.ru/post/265043/
If you have any questions regarding the material covered in the article above, please, contact the original author of the post.
If you have any complaints about this article or you want this article to be deleted, please, drop an email here: sysmagazine.com@gmail.com.

We believe that the knowledge, which is available at the most popular Russian IT blog habrahabr.ru, should be accessed by everyone, even though it is poorly translated.
Shared knowledge makes the world better.
Best wishes.

comments powered by Disqus