SEP sesam backup client for VMware vStorage API

From SEPsesam
Jump to: navigation, search

Copyright © SEP AG 1999-2017. All rights reserved.

Any form of reproduction of the contents or parts of this manual is allowed only with the express written permission from SEP AG. When compiling and designing user documentation SEP AG uses great diligence and attempts to deliver accurate and correct information. However, SEP AG cannot issue a guarantee for the contents of this manual.


Contents

Introduction

This article is relevant for SEP sesam version 4.4.1/4.4.2 and describes SEP sesam extension for VMware vStorage APIs for Data Protection (VADP). Note that this is not the latest version of SEP sesam VMware documentation and, as such, does not provide information on features introduced in 4.4.3 and described in VMware Backup.

The SEP sesam extension for VMware vStorage APIs for Data Protection (VADP) provides hot backups of Virtual Machines (VMs) running on VMware ESXi servers. To be able use the vStorage API at least a VMware vSphere Essentials license must be purchased. Consistent backups are achieved by creating a snapshot of the virtual machine and transferring data of the virtual disk (VMDK) files directly to the SEP sesam Server or SEP sesam Remote Device Server. If a storage network environment is configured properly data can be transferred from ESXi server to SEP sesam Remote Device Server directly over SAN to avoid network traffic.


Advantages

  • No workload on the ESX Server

Disadvantages

  • Virtual machines must be on a SAN storage device to avoid data transfer over network

Known Issues

Restore of vSphere 6 does not work with 4.4.1
  • Backup of vSphere 6 works with capabilities of VDDK 5.5. But restore does not work at all with 4.4.1 due to check of vSphere version of SEP sesam GUI. There is no workaround available for version 4.4.1. Just with version 4.4.2 of SEP sesam a restore of vSphere 6 is possible.


Known issues
  • Backup of VMs using VDDK 5.1.1 on Linux can fail occasionally. These are known issues in VDDK and VMware is still working on a new version to fix them. We advise to use VDDK version 5.1.1 on Windows as long as the problems in VDDK for Linux exist. SEP will inform you, when a new version will be available here. For more information please refer on this VMware Knowledge Base article: http://kb.vmware.com/kb/2039931
  • The following characters are not supported in data center, data store and VM names: '$', '&', "'", ':', '%'


22.10.2013 Backup of virtual machines with two VMDK files with same file name does not work at the moment

A virtual machine with name "vmware" for example uses following VMDK files: [Datastore1]/vmware/vmware.vmdk and [Datastore1]/vmware_org/vmware.vmdk
Workaround: Rename one of the virtual disks.


The restore of a VM configuration uses a lot of harddisk space

If only the configuration of a VM containing thick provisioned VMDKs is restored, the new VMDKs will have the same size as the original VMDKs.
Reason: During the restore empty VMDKs are created. For thick provisioned disks always the full disk size is used. For thin provisioned disks 0 bytes will be reserved.

System requirements

An update to latest SEP sesam version 4.4.x is recommended by SEP


Using changed block tracking

To use CBT SEP sesam version 4.4.1.41 or higher is required


  • SEP sesam DataMover for VMware vSphere
    • Windows Server 2008 [R2] or Windows Server 2012 acting as SEP sesam DataMover for VMware vSphere. This could be the SEP sesam Server, the vCenter Server or any other SEP sesam Client.
      or
    • Linux server acting as SEP sesam DataMover for VMware vSphere. This could be the SEP sesam Server, the vCenter Server or any other SEP sesam Client.
      • SLES 11 64 bit SP1 and SP2
      • SLES 10 64 bit SP4
      • RHEL 5.9 64 bit
      • RHEL 6.2 64 bit
      • RHEL 6.3 64 bit
      • RHEL 6.4 64 bit


  • The following packages and modules have to be installed on the DataMover:
    • PowerShell 2.0 (Windows DataMover)
    • VMware Virtual Disk Development Kit 5.x (included by SEP sesam on Windows)
    • SEP sesam Client package version >= 4.4.1.41 or later (Server-, GUI- or RDS- package will work too).
    • Best Practice: Use a physical Windows system with vmware SAN connectivity as DataMover.

Usage of SAN/HOTADD with vSphere6

  • To use SAN or HOTADD with vSphere6, VDDK 5.5.4 is necessary. All VDDK versions below cannot esablish a SAN or HOTADD connection, just NBD and NBDSSL.


Version of VMware Virtual Disk Development Kit

Because SEP sesam extension is linked against a certain VDDK, the following version will be required with SEP sesam version 4.4.1:

VDDK 5.5.1:

  • Windows x64 (min. Server 2008)
  • SLES 11 64 bit SP1 and SP2
  • SLES 10 64 bit SP4
  • RHEL 5.9 64 bit
  • RHEL 6.2 64 bit
  • RHEL 6.3 64 bit
  • RHEL 6.4 64 bit


With SEP sesam version 4.4.2 following VDDK is used:

VDDK 5.5.3 respectivelly VDDK 5.5.4:

  • Windows x64 (min. Server 2008)
  • SLES 11 64 bit SP1 and SP2
  • SLES 10 64 bit SP4
  • RHEL 5.9 64 bit
  • RHEL 6.2 64 bit
  • RHEL 6.3 64 bit
  • RHEL 6.4 64 bit




Functional principle

To run a SEP sesam backup and restore in a vSphere environment with vStorage API a SEP sesam Client is necessary. This functions as a SEP sesam DataMover (central communication element) between SEP sesam Server and VMware vSphere Farm.

The arrows in the illustration show the direction of the connection establishment. After, data certainly flows in both directions.


VADP Principle 405.jpg


The initial communication goes from the SEP sesam Server to the vCenter Server. For this the vCenter Server has to be registered as a client in the SEP sesam environment. Communication for browsing starts over the SEP sesam control port 11301 (1) and goes to the vCenter Server. This means that when setting up a backup task you're always shown the file structure of the vCenter first. Now if the VMware vSphere branch is chosen, the https-connection (2) from the SEP sesam Server to the vCenter Server is started. Here, all virtual machines are identified and displayed.

When SEP sesam Server starts the backup it initiates a SEP sesam control connection (Port 11301) to the datamover (3). The datamover connects over the open source LIBVIRT library via https to the vCenter Server (4). There all parts of the virtual machine that are going to be backed up are identified. Now before a new SEP sesam backup snapshot is created all possibly still existing old SEP sesam backup snapshots are deleted. After taking the snapshot the datamover establishes a connection via the VMware VDDK on port 902 (6) to the ESXi server, where the VM that is being backed up, is hosted. Now the files of the VM are transferred from the ESXi server over the VDDK to the datamover and then further over the SEP sesam Transfer Protocol (7) to the SEP sesam Server where they are saved onto a medium of the chosen media pool.

For restores the way is the same except backwards.

vCenter Server, Data Mover and SEP sesam Server can run on the same machine.

If Data Mover runs virtualized on the Data Center from which VMs should be saved, SAN backups will not work.

Restrictions

Restrictions
  • For backups of VMs over SAN mode, the SEP sesam DataMover must operate as a physical machine with access to the Fibre Channel or iSCSI SAN, containing the virtual disks to be accessed!
  • For backups of VMs using HOTADD mode, the SEP sesam DataMover must operate as a virtual machine in the same VMware DataCenter as the VM!
  • Currently only ONE VM per backup job is possible in SEP sesam.
  • vCenter password must not include % sign.
  • Independent disks and raw devices are not supported

VDDK installation

The SEP sesam requires the VMware Virtual Disk Development Kit (VDDK) to be installed on SEP sesam Data Mover (SDM-VV). From SEP sesam version 4.4.1.x onwards, the VDDK is installed automatically during the SEP sesam installation.

Information sign.png Note
It is recommended to use VDDK 5.5.4 with SEP sesam 4.4.2, while SEP sesam 4.4.3 supports VDDK version 5.5.4 and 6.0.2.

The VDDK installation procedure differs according to the operating system.
For information on VMware VDDK release notes, known issues and workarounds, see Further Links/Literature at the end of this document.

On Windows

SEP sesam 4.4 installation package with VDDK 5.5.4

With SEP sesam v. 4.4 either Virtual Disk Development Kit (VDDK) 5.5.1, 5.5.2, 5.5.3, or 5.5.4 for Windows is included in the SEP sesam installation package depending on the SEP sesam version of the installation package. When upgrading to or installing the SEP sesam v. 4.4, the VDDK version 5.5.1/5.5.2/5.5.3/5.5.4 is automatically installed. If a VDDK is already present on the system to which the SEP sesam is going to be installed or upgraded, the previous VDDK version is uninstalled and the VDDK 5.5.1/5.5.2/5.5.3/5.5.4 is unpacked.

The VDDK 5.5.0 requirements with SEP sesam version 4.2.2.40 or newer

  • Supported operating systems are Windows Vista/Windows Server 2008 and higher.
  • If the VDDK is already installed on a system, manually uninstall it. Make sure that the registry path HKLM:\SOFTWARE\Wow6432Node\VMware, Inc.\VMware Virtual Disk Development Kit does not exist anymore.
  • The VDDK 5.5.0 zip file (starting with VMware-vix-disklib) should be present in the <sesam-root>\skel directory. If the VDDK 5.5.0 zip cannot be found, download it from the VMware homepage and copy it into the skel directory.
  • With the VDDK 5.5.x installation new parameters are introduced in the set_vddk.ps1 script:
    • -e install is used for the VDDK 5.5.x installation
    • -e update is used for updating the VDDK 5.5.x to the latest VDDK (i.e 5.5.1)
    • -e remove is used for removing the VDDK 5.5.x structure
    • If you want the SSL certificates to be validated, specify the -ssl 1 parameter. It is available with parameters -e install and -e update. The default value is 0.
    • Use the additional parameter -f to overwrite currently installed VDDK 5.5.x. The syntax is -f -e install.
    • The parameter -h shows the usage of the script.

After the VDDK 5.5.x installation, the necessary 64-bit libraries are located as a ZIP file in the VMware program directory. SEP sesam package includes a PowerShell script to install these libraries and set the registry key correctly.

With the SEP sesam v.4.2.2.40 and on SEP sesam clients, you need to perform additional steps.

  • Log in to the DataMover.
  • Open the <SESAM_ROOT>/bin/sesam/ directory.
  • Execute the set_vddk.ps1 script. Note that in the PowerShell the command line .\ hast to be prefixed.

To use VDDK 5.5 on Windows RDS/Clients, the PowerShell script <SEPsesam>\bin\sesam\set_vddk.ps1 is used during update. The script can be manually called with set_vddk.ps1 -e install. To force the installation, use the switch -f: set_vddk.ps1 -e install -f.

Attention

By default, the Powershell execution policy is set to Restricted. To execute the PowerShell scripts, set the execution policy to RemoteSigned by using the set-executionpolicy remotesigned command. Note that you must be logged in as an administrator or start the script with administrative privileges. For more information on PowerShell script execution policy, see Using the Set-ExecutionPolicy Cmdlet.

On Linux

  1. Download the appropriate VDDK for Linux from the VMware download center. Then unpack and install the VDDK, as shown in the following VMware VDDK installation example on Linux 64-bit:
  2. VDDK version 5.5.4

    tar xvzf VMware-vix-disklib-5.1.1-774844.x86_64.tar.gz
    cd vmware-vix-disklib-distrib/
    ./vmware-install.pl
    
    
    VDDK version >= 6.0.0
    rm -r /usr/lib/vmware-vix-disklib/ # Only neccessary if a previous VDDK installation exists
    tar xvzf VMware-vix-disklib-6.0.0-2498720.x86_64.tar.gz
    cp -r vmware-vix-disklib-distrib/ /usr/lib/vmware-vix-disklib
    

    The VMware VDDK will be installed to /usr/lib as default.

    Information sign.png Note
    SEP only supports installation of VDDK into the default path!
  3. On the system which acts as a data mover, the LD_LIBRARY_PATH has to be set to the VDDK library path.
    • For regular CTRL access to SEP sesam data mover, edit <SESAM_VAR>/var/ini/sm.ini and add/modify the following lines:
    • For 32-bit Linux:

       [ENVIRONMENT]
       LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib32:/usr/lib/vmware-vix-disklib/plugins32:$LD_LIBRARY_PATH
      

      For 64-bit Linux:

       [ENVIRONMENT]
       LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib64:/usr/lib/vmware-vix-disklib/plugins64:$LD_LIBRARY_PATH
      
    • For SSH access to SEP sesam data mover, edit /etc/sesam2000.ini and add/modify the LD_LIBRARY_PATH line:
    • For 32-bit Linux:

        VERSION=4.2.1.41
        SM_BIN_SESAM=/opt/sesam/bin/sesam/
        ...
        LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib32:/usr/lib/vmware-vix-disklib/plugins32:$LD_LIBRARY_PATH
      

      For 64-bit Linux:

        VERSION=4.2.1.41
        SM_BIN_SESAM=/opt/sesam/bin/sesam/
        ...
        LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib64:/usr/lib/vmware-vix-disklib/plugins64:$LD_LIBRARY_PATH
      
  4. Restart the SEP sesam service for the changes to take effect.

Best Practices

Scalable LAN-free Backup over SAN Transport

LAN-free Backup over SAN Transport is the fastest transport method for SEP sesam Backup on SAN connected ESXi hosts. The VM Backup performance can be scaled up with SEP sesam. This allows simultaneously backup of a large number of virtual machines.


The following diagram describes the structure and the operating principle for scalable LAN-free backup over SAN transport:
SEP sesam Best-Practice scalable VMware LAN-free-Backup 02.png
The modular structure of SEP sesam supports the "scale-out" principle. This allows to use multiple identical Remote Device Server (RDS).

Requirements

The following requirements apply to the "scalable LAN-free backup over SAN transport" scenario:

Generell requirements for SAN transport

  • The VMware datastore must support hardware acceleration. Therefore open vSphere Client, choose ESXi host and switch to tab “Configuration”. Then choose “Storage” and check, whether hardware acceleration is supported by the specified VMware datastore.
  • Also hardware acceleration must be enabled on each ESXi host. Therefore check tab “Configuration” and choose “Advanced Settings”. Select section “DataMover” and ensure, both values are set to 1. Also select section “VMFS3” and make sure “VMFS3.HardwareAcceleratedLocking” is set to 1.


Additional requirements for SAN transport with Linux proxy

On Linux a SCSI reservation conflict can occur during SAN access. This causes the backup/restore to fail. To prevent this, the following things need to be checked.

A terminal has to be started on each ESXi server to execute the following command:

vmkfstools -Ph -v1 /vmfs/volumes/<name of VMware datastore>

example:

vmkfstools -Ph -v1 /vmfs/volumes/SAN-test-volume/

The output could include this:

Mode: public
Partitions spanned (on "lvm"):
        naa.600000e00d280000002801dd00050000:1

If mode is public, then the problem can occur. In this case ATS (Atomic Test & Set) has to be enabled. No VM should run during this time on this VMware datastore (recommendation by VMware). This command needs to be executed on each ESXi server with mode public:

vmkfstools --configATSOnly 1 /vmfs/devices/disks/<naa-id]:[partition-number>

example:

vmkfstools --configATSOnly 1 /vmfs/devices/disks/naa.600000e00d280000002801dd00050000:1

The device name “naa.600000e00d280000002801dd00050000:1” was shown with first command above. After this step the output of the first command above should show this now:

Mode: public ATS-only


Up from this time ATS is used and then sporadically SAN errors on Linux proxy shouldn’t appear anymore.


SEP sesam Backup Server:

  • Hardware minimal requirements: Intel 4 Core Server CPU or AMD Server 6 Core CPU, min. 16GB RAM
  • Backup LAN: 10 Gbit (the VM backup data is transferred through the SAN)
  • SAN connectivity: For the backup server not required

SEP sesam Remote Device Server:

  • Hardware minimal requirements: Intel 4 Core Server CPU or AMD Server 6 Core CPU, min. 16GB RAM
  • Backup LAN: 10 Gbit (the VM backup data is transferred through the SAN)
  • SAN connectivity: All vmware LUN's must be connected via SAN and be visible. For iSCSI example see also: http://linux.die.net/man/8/iscsiadm
  • Backup Storage: Connection of the tape hardware and disk backup storage via FC, iSCSI, or SAS
  • VDDK: On all Remote Device Server the VMware VDDK 5.5.3 or 5.5.4 must be installed
  • vCenter: can be installed on a remote device server as vCenter for Windows, or the VMware vCenter appliance can be used as a VM
  • Datamover (Proxy)

Configuration

Activating Changed Block Tracking (CBT) globally

For using Changed Block Tracking (CBT), you must enable the CBT functionality globally. Afterwards, when configuring a backup task, you can decide whether you want to use the CBT in the respective backup task or not. For details, see the section Changed Block Tracking (CBT).

In the SEP sesam GUI, from the menu bar select Configuration -> Defaults -> tab Extras, and then select Globally activate Change Block Tracking (CBT).

Enable cbt global en.png


Configuration with vCenter

Depending upon your SEP sesam version, the procedure for configuring vCenter server may be slightly different. Refer to the relevant version-specific steps below.

1. Configure Virtual Center server in the SEP sesam GUI as a normal Windows or Linux client. In the Main selection -> Components -> Topology -> New client, select the appropriate operating system. You can select a location to which you want to add a new client from the drop-down list of configured locations.

2. Define the client as a vCenter server.

SEP sesam version 4.4.2

In the New client properties, from the VM Server type drop-down list, select the option vCenter.

Client vCenter.png


SEP sesam version 4.4.1 and older

In the New client properties, select the check box Client is a vCenter Server.

VADP vCenter-activate 403en.png



3. Click the tab vCenter Access and add a vCenter credentials. The following screenshots show version specific access dialog.

4. Select a data mover. The VMware backup performance is highly dependent on the transport mode. Please see the recommendations for the optimal Data Mover: Transport Modes

Configuration with ESX Server

In case no vCenter is available, the ESX Server itself can be added to SEP sesam. In the Main selection -> Components -> Topology -> New client, add the ESX Server with its DNS name, select ESX-Server as operating system and Proxy as access mode. Then select the tab ESX Server Access and enter the credentials.

Backup

In SEP sesam GUI create under the vCenter Client a new backup task with task type VMware vSphere. The VM name (backup source) can be selected using the client file system browser under VMware vSphere.

The backup source has the following format: //<display name of VM>

Example:

 /<data center>/<VM name>

VM SEP-DC01 in data center esxixfix:

 Source=/esxixfix/SEP-DC01


VADP Task-Config 403en.png


Exclude of VMDKs

One or several VMDKs can be excluded from backup by specify complete path or symbolic name in exclude option field. These values have to be entered manually. The symbolic names reference the order of VMDKs of virtual machine definition.

The 1st virtual disk is vmdk0, the 2nd is vmdk1, and so on.


VM backup task dialog

VM task offers several options, which influence backup process.


Changed block tracking 42140.jpg



  • Backup VM configuration only (no VMDK data)
Backup of OVF and CONF file. Used to create VM that contains virtual devices but no data or operating system
  • Backup VM config and VMDK in separate savesets
VM config (OVF-File) and each VMDK of VM are stored in its own backup files
  • Change block tracking active
New: Activate/deactivate change block tracking. Enable this option to reduce data size (see below)

Snapshot Parameters

  • Snapshot creation with quiescence (needs VMware tools installed and VSS on Windows):
Quiescing a file system is a process of bringing the data on disk to a consistent state. This process might include such operations as writing cache to disk.


Changed block tracking 42140 Options.jpg


Example in the tab Options 1 in Backup options:

-a  qui=<1|0>  (Default: '1')

Transport modes

The VMware backup performance is highly dependent on the transport mode. VMware offers several transport modes for transferring the data from VMware datastore to the backup device.

The table shows recommendations for the Data Mover, depending on the storage type and desired transport mode:

SAN Transport Mode (recomended) HotAdd Transport Mode Network Transport Mode
Performance: Very fast Fast Slow (It might works well with 10 Gbit)
FC/iSCSI SAN Storage: Data Mover: The physical SEP Sesam Server / RDS. Data Mover: A Windows VM with Sesam Client. Data Mover: The SEP Sesam Server.
NFS and local Storage: Not available.
Requirements for the Data Mover:
  1. The Data Mover must be a physical machine
    and needs direct access to the SAN LUNs (read only).
  2. Define the Client as Data Mover in the vCenter settings.
  1. Install the latest SEP Client on a Windows VM.
  2. Add the Client to the Sesam GUI.
  3. Define the Client as Data Mover in the vCenter settings.


SAN Transport Mode
To use SAN as transport mode during backup, VMDKs of VM must be on a SAN device accessible by the SEP sesam data mover. Data Mover must not be a VM.
HOTADD Transport Mode
Back up VMDKs by hot-add them to Data Mover VM. So the Data Mover must be a VM running in the same data center as VM which is to be saved.
NETWORK Transport Mode(NBD)
The Network Block Device mode transports the data over the regular LAN. This method will be always available.
NBDSSL
Same way like NBD, but the data will be encrypted by SSL.

By default, the following order is set in the vStorageAPI: san:hotadd:nbdssl:nbd. Using the backup option trans, the transport mechanisms that are to be used by the vStorage API can be forwarded with the backup task:

Example in the tab Options 1 in Backup options:

 -a trans=nbd:nbdssl

Here, the data stream is set to travel via LAN, regardless of encryption.

Which transport mode can be used or should be used depends on your virtualization environment and of the recommendations and restrictions of the VMware vStorage API. (Detailed Information see VMware VDDK virtual disk transport methods)


Attention: A Data Mover with linux is experimental and is not recommended!

Changed Block Tracking (CBT)

New in SEP sesam 4.2.1.40

  • CBT is a VMware feature to perform block level incremental and differential backups of VMDKs. VM's running on ESX/ESXi hosts 4.0 or later can track changed disk sectors and identify all sectors that are in use on VMFS partitions.
Technical information about CBT from VMware:

For best results, when using QueryChangedDiskAreas (this API function is used by a CBT backup) to gather information about snapshots, enable change tracking before taking snapshots. Attempts to collect information about changes to snapshots that occurred before change tracking was enabled result in a FileFault error. Enabling change tracking provides the additional benefit of saving disk space because it allows for the backup of only information that has changed.

If change tracking is not enabled, it is unknown what has changed, so the entire virtual machine must be backed up each time, rather than only backing up changed data. Be aware that entire disks are allocated, and therefore are returned when QueryChangedDiskAreas is used in the following scenarios:

  • A cloned thick disk is created.
  • Disks are formatted using the Long Format setting.
  • Disks are created with vmkfstools using the eagerzeroedthick format option.
  • A VMDK is stored on storage that does not support VMFS, such as NFS or iSCSI devices.


  • When SEP sesam starts a 'DIFF' or 'INCR' backup, it can request transmission of blocks changed since the last FULL backup only. In case of a 'FULL' backup only used blocks in a VMDK file are requested. The backup type 'COPY' stores all data of a VMDK including empty blocks (as provisioned).

Information about CBT

  • Supported with Sesam version '4.2.1.40' or above
  • VM CBT is necessary for VM FULL/DIFF/INCR backup.
  • Independent disks are not supported by CBT.
  • The host must be ESX/ESXi 4.0 or later.
  • CBT can be manually enabled/disabled at backup task dialog.

VM backup type 'COPY'

  • Backup VM with type 'COPY' will back up the whole VMDK including empty blocks (provisioned size = size of backup and size of VMDK equals).
  • CBT is not required and will not be activated.

VM backup type 'FULL'

  • Backup VM with type 'FULL' will only backup used blocks of VMDK files.
  • VMDK files must be located on a VMFS volume, backed up by SAN, iSCSI, or local disk. VMware RDM (Raw Device Mapping) is not VMFS.
  • Backup of used blocks will not be able for VMDKs stored on NFS volume. Nevertheless 'DIFF' and 'INCR' backups work.
  • CBT will be activated by Sesam for a VM, if a backup starts with level 'FULL'.

VM backup type 'INCR'/'DIFF'

  • This operation requires same conditions as 'FULL' backup. DIFF and INC typed CBT backups are working on the differential way as normal path backups but only on block level.
  • VM CBT must be activated and a FULL backup has to be executed successfully before.


Attention

Do not disable CBT, otherwise the whole VM FULL/DIFF/INC chain will be destroyed and further VM DIFF/INC backups are impossible and a new backup chain with type 'FULL' must be created.

Restore

The restore can be started with the regular SEP sesam restore wizard.

  • All vCenter Servers that are configured as SEP sesam Clients can be selected for restore.
  • The according Data Centers and ESXi servers can be selected.
  • If different login credentials are needed SEP sesam will ask for them.
  • If vCenter is not set, the restore will be done directly over the ESXi server

Select the according restore task and confirm the selection with <Next>.


VADP Restore-Task-Select 403en.png


On the page Save and Start of the restore wizard, the valid VMware infrastructure values for the backup are shown. If the VM is not supposed to be backed up to its original state, different existing values for VMware Datacenter, ESXi server and VMware DataStore can be selected. If necessary, you'll be asked to enter the valid users/passwords to authenticate to the vCenter and to the ESXi Server.

Additionally it is possible to verify the login information with the <Check>-button. If clicking the <Check>-button doesn't cause a login dialog to appear, the stored information is valid.


VADP Restore-ValueSet 403en.png


If the option Overwrite is set, an existing VM with the same name will be deleted before the restore starts

Troubleshooting

Message

SSL handshake error during browse of vSphere server

  • The name "VMware vSphere:" appears again after expanding it.
  • A pop up opens, which refers to this page

Reason

  • This issue is related to VMware vSphere 5.5 (may also 5.1) and IBM Java due to changes according OpenSSL.

Solution

There are two possible solutions to solve the issue:

  1. install Oracle Java instead of IBM Java, because Oracle Java is able to connect to vSphere 5.5. After installing Oracle Java, the java link in /opt/sesam/bin/sesam needs to be adjusted to redirect to Oracle Java.
  2. Change list of allowed cyphers on the vSphere server. Please have a look at http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2049143 for further information.


Message:

Error: VM Exception: [HostCommunication].

  • A VADP backup ends with the error message mentioned above.
  • In the event log of the hyper visor the following error message occurs at the same time if
    • a snapshot should be created or deleted on this host
    • a virtual machine should be migrated to or from this host
    • a virtual machine on this host should be cloned
An error occurred while communicating with the remote host.

Reason
One of the management services on the hypervisor is not or not correctly running.

Solution

  • Log on to the to the ESXi server via ssh
  • restart the vpxa Service
/etc/init.d/vpxa restart

See also: Restarting the Management agents on an ESXi or ESX host


Message:

Execution error of set_vddk.ps1 according vmware-vdiskmanager.exe

Reason

  • The depending msvcr90.dll for this application is not available.

Solution



Message:

GUI reports Login denied during browsing the vSphere farm

  • When vCenter user and password have been verified an d the error is still running, you can check the VMware farm access without a graphical user interface (GUI)

Reason

  • Occasionally, there are troubles with special characters in passwords. For a better analysis of these problems, it could help to check the VMware vSphere farm connection over command line.


Solution

  • In the following command you have to replace the vCenter server name, username and password with your own environment.
Pattern:
sbc_vadp -D -a username=<vCenter username>,password=<password>,url=https://<vCenter Hostname>/sdk,ignorecert=ignorecert "/VMware vSphere:"
Example:
sbc_vadp -D -a username=Administrator,password=mypassword,url=https://ws2008x64/sdk,ignorecert=ignorecert "/VMware vSphere:"

It should then be listed in the configured VMware Data Center. Is that not the case, then it is helpful first to set the password to be easier without special characters. Is the access test to the farm successful, then the original password contains a character that causes a problem accessing the VMware farm.


Message:

svp-1901: Error: VM Exception: [Exception=FileFault,dynamictype=null]. -> CBT-RESET needed

  • Error occurs if a snapshot is not deleted after a CBT backup

Reason

  • CBT is not in a clean state

Solution

  • Reset CBT (see next chapter)

Possible choices to reset CBT via command line or via GUI

Soft reset (On line) Generally available up from version 4.2.2.26 (up from version 4.4.1.x a soft reset is also possible via GUI and via CLI, too. This is described below the next command)

  • When executing the following command, the virtual maschine does not need to be shut down. The command needs to be executed in directory <sesam-root>\bin\sesam:
sbc_vadp -b -a "action=softreset,server=<vSphere Server>,username=<user>,password=<password>" "<datacenter>/<vm name>"

Example
sbc_vadp -b -a "action=softreset,server=ws2008x64,username=Administrator,password=secret" "DC/linuxdbserver"


Soft reset (On line) via GUI respectively via CLI (sm_cmd)

  • As described above, a soft reset of CBT is also possible via GUI and CLI up from version 4.4.1.x of SEP sesam.
    • Via GUI this can be done for each task. The button "Reset CBT" is available on the first tab for each task at "Tasks > by Clients" when opening the properties of a specified task.
    • The second possibility is to use CLI (sm_cmd):
sm_cmd resetcbt -S <vCenter or ESX> -d <data center> -V <Name of VM>

Example
sm_cmd resetcbt -S qsbox1 -d "SEP Cloud" -V "cosinus (SEP)"


Hard reset (Off line) (up from version 4.4.1.x a hard reset is also possible via CLI, too. This is described below the next command)

  • This command should be used, if a soft reset does not work or if it's not available. Look at requirements below for further information.
sbc_vadp -b -a "action=resetcbt,server=<vSphere Server>,username=<user>,password=<password>" "<datacenter>/<vm name>"

Example
sbc_vadp -b -a "action=resetcbt,server=ws2008x64,username=Administrator,password=secret" "DC/linuxdbserver"


  • As described above, a hard reset of CBT is also possible CLI up from version 4.4.1.x of SEP sesam. The syntax for this command is:
sm_cmd resetcbt -S <vCenter or ESX> -d <data center> -V <Name of VM> -m hard

Example
sm_cmd resetcbt -S qsbox1 -d "SEP Cloud" -V "cosinus (SEP)" -m hard


Requirements

  • The state of the virtual machine must be powered off for this action.

Manual CBT reset

Alternatively, via the vSphere client, a CBT-reset will be performed:


On line

With the following steps the virtual machine does not need a shut down to reset CBT:

1. Open the properties of the affected virtual machine at "Tasks > By Clients"
2. Remove the tick "Enable Change Block Tracking (CBT)" and save the task.
3. Create a snapshot via vSphere Client of the affected virtual machine (none of the ticks needs to be selected)
4. Delete the snapshot, which has been created before.
5. In SEP sesam --> activate the CBT option in the VM client settings --> start the VM backup
Off line

This step needs to be done, if the a reset of CBT in on line mode does not work. With this steps the virtual machine must be turned off:

1 Shutdown the VM
2 Open the VMware Snapshot-Manager --> delete all snapshots
3 Right click the VM --> "Edit settings" --> "Options" tab --> "General" -> "Configuration Parameters"
4 Set the value "ctkEnabled" to false
5 Set the value "scsi0:0.ctkEnabled" to false (NOTE: Set the value "false" for each disk)
6 Open the folder where the VM *.VMDK files exists --> delete all *-CTK.VMDK files
7 Start the VM
8 Shutdown the VM (This is for the CTK table update needed)
9 Start the VM
10 In SEP sesam --> activate the CBT option in the VM client settings --> start the VM backup

Message

Error: At least one snapshot exists before CBT was enabled, which is not possible

'Possible Cause'

  • VM is suspended
  • Manual snapshots available

Solution

  • Restart or shutdown VM
  • Delete manually created snapshots and then possibly perform a reset CBT

Message:

VIX_E_FAIL

  • Error occurs during backup of 2nd VMDK.

Reason

  • Timeout in vCenter connection

Solution

  • Update VDDK library to version 1.2.1


Messages:

VIX_E_FILE_NOT_FOUND:

or

VIX_E_FILE_ACCESS_ERROR:

or

You do not have access rights to this file:

or

Thin/TBZ/Sparse disks cannot be opened in multi writer mode.

Reason

  • Wrong VMDK file has been linked for backup

Solution

  • This problem will be fixed in a future version of Sesam (state Jul 2 2013).
    Otherwise start backup with option -a qui=0. As a result of this option the involved system boots in normal state and shows message regarding safe boot. See this screen shot as an example on how to add this option:

Qui.jpg


Backup fails with error on a 64 Bit system even though VDDK is already installed:

Error: Load VDDK library failed: Cannot load: vixDiskLib.dll

Reason

  • VDDK is only installed as a 32 Bit version

Solution


Backup stops with message:

sbc-1500: Error: VixDiskLib_Open() failed: [sesam] sesam_vm/sesam_vm.vmdk:Cannot connect to the host

Reason

  • The file vixDiskLib.dll does not exist in the directory <SESAM_ROOT>\bin\sesam of the VMware data mover.

Solution

  • Copy the file C:\Program Files\VMware\VMware Virtual Disk Development Kit\vixDiskLib.dll to <SESAM_ROOT>\bin\sesam of your VMware data mover.

VDDK/VIX side-by-side configuration is incorrect

Reason

  • Required C++ libraries are not installed.

Solution

Encrypted backup stops with Encryption/Decryption operation failed

Reason

  • Backup was done with bf64 (blowfish 64) encryption.

Solution

  • Encryption has to be changed to aes256 instead of bf64.


Message:

VM Exception: [InvalidRequest].

  • Error occurs during VM creation with OVF config file.

Reason

  • Will be thrown, if OVF file, that will be used for create VM during restore is invalid.
    • Example: virtual CD-ROM with ISO file and file is not more available

Solution For successful restore, you must edit OVF file of VM and remove all virtual devices, which prevent creating VM.


  • Restore only VM config files.

New Restore Task VM config.jpg

  • Remove virtual CD-ROM with ISO from OVF file.
    • You can find the file at the sesam directory: <SESAM_ROOT>/var/tmp/_vadp_/<vm name>/<saveset of backup>
CD-ROM specification part of ovf file
    <Item>
      <rasd:AddressOnParent>0</rasd:AddressOnParent>
      <rasd:AutomaticAllocation>true</rasd:AutomaticAllocation>
      <rasd:ElementName>CD/DVD Drive 1</rasd:ElementName>
      <rasd:HostResource>ovf:/file/file2</rasd:HostResource>
      <rasd:InstanceID>7</rasd:InstanceID>
      <rasd:Parent>4</rasd:Parent>
      <rasd:ResourceType>15</rasd:ResourceType>
    </Item>
  • Start same restoretask and restore VM with VMDK


New Restore Task VM VMDK data.jpg


For successful restore, you must edit OVF file of VM and remove all virtual devices, which prevent creating VM.

 svp-3980: Info:    Create VM from OVF file: 'C:\ProgramData\SEPsesam\var\tmp\_vadp_\EX-SERVER\SC20130423113643791@70RYUkBmDGz\EX-SERVER.ovf'
 svp-1901: Error:   VM Exception: [VM creation failed  (error)].


VM creation breaks with error:

svp-3980: Info:    Create VM from OVF file: 'C:\ProgramData\SEPsesam\var\tmp\_vadp_\EX-SERVER\SC20130423113643791@70RYUkBmDGz\EX-SERVER.ovf'
svp-1901: Error:   VM Exception: [VM creation failed  (error)].

Reason

  • Various possible errors

Solution

  • Analysis of the VMware Logs:
    • Use vCenter <Administration>, <export system logs> to select the affected ESX-Server as source. Choose <Logs> only from system logs and specify desired export location.
    • Once the export is complete, unzip the tgz file from the export directory.
    • Then locate the host.d file and search for errors.

Possible errors

  • Wrong video RAM size. Must be set to 4MB in "auto-detect" mode. The error message in host.d is:
opID=49F494FA-000001D6-6a-7b-3e-ec-f4] Video Ram size edit not supported when auto-detect is True ./var/log/hostd.log:2013-04-23T14:43:56.923Z [61B7EB90 info 'vm:/vmfs/volumes/5087d83e-bbbbd91c-4218-001999da71cf/testEX-SERVER/testEX-SERVER.vmx' 
opID=49F494FA-000001D6-6a-7b-3e-ec-f4] Reconfigure failed: vmodl.fault.NotSupported

Cannot display VMX file

Error occurs during download of VMX file.

Reason

Solution

  • Update version of vCenter. This problem is not related to Sesam, but to vCenter only.

Error: Could not load default plugins from /usr/lib/vmware-vix-disklib/lib64/libdiskLibPlugin.so

  • A VADP backup ends with: Error: VM Exception: [Exit code from sbc: [1] - warning].. In the log file you can see: Could not load default plugins from /usr/lib/vmware-vix-disklib/lib64/libdiskLibPlugin.so.
  • The following causes can trigger the error:
  • The correct LD_LIBRARY_PATH has to be set
  • Solution:

Edit <SESAM_VAR>/var/ini/sm.ini and add/modify the following lines:

For 32 bit Linux:

 [ENVIRONMENT]
 LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib32:/usr/lib/vmware-vix-disklib/plugins32:$LD_LIBRARY_PATH

For 64 bit Linux:

 [ENVIRONMENT]
 LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib64:/usr/lib/vmware-vix-disklib/plugins64:$LD_LIBRARY_PATH
  • For SSH access to Sesam data mover:

Edit /etc/sesam2000.ini and add/modify the LD_LIBRARY_PATH line:

For 32 bit Linux:

  VERSION=4.2.1.41
  SM_BIN_SESAM=/opt/sesam/bin/sesam/
  ...
  LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib32:/usr/lib/vmware-vix-disklib/plugins32:$LD_LIBRARY_PATH

For 64 bit Linux:

  VERSION=4.2.1.41
  SM_BIN_SESAM=/opt/sesam/bin/sesam/
  ...
  LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib64:/usr/lib/vmware-vix-disklib/plugins64:$LD_LIBRARY_PATH


  • If the problem still exists after modifying the sm.ini and even after restarting the SEP sesam Service, we have to check if the libexpat.so.0 really exists on the chosen system

vcenter:/usr/lib64 # ldd /usr/lib/vmware-vix-disklib/lib64/libdiskLibPlugin.so

       linux-vdso.so.1 =>  (0x00007fff539ff000)
       libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdab6937000)
       libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fdab662d000)
       libdl.so.2 => /lib64/libdl.so.2 (0x00007fdab6428000)
       libcrypto.so.0.9.8 => /usr/lib64/libcrypto.so.0.9.8 (0x00007fdab6089000)
       libssl.so.0.9.8 => /usr/lib64/libssl.so.0.9.8 (0x00007fdab5e35000)
        libexpat.so.0 => not found
        
       libz.so.1 => /lib64/libz.so.1 (0x00007fdab5c1e000)
       libtypes.so => not found
       libvmomi.so => not found
       libvmacore.so => not found
       libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdab5a07000)
       libc.so.6 => /lib64/libc.so.6 (0x00007fdab5692000)
       /lib64/ld-linux-x86-64.so.2 (0x00007fdab70b0000)
       libm.so.6 => /lib64/libm.so.6 (0x00007fdab5419000)
  • Search for the file libexpat.so.0
       vcenter:/usr/lib64 # ls -lart libexp*
        -rwxr-xr-x 1 root root 196944 May 29  2009 libexpect5.44.1.11.so
        -rwxr-xr-x 1 root root 170984 Feb 20  2010 libexpat.so.0.5.0
  • Result: The file libexpat.so.0 does not exist
  • Resolve the issue:
ln -s libexpat.so.0.5.0 libexpat.so.0
  • First control after generating the soft link:
vcenter:/usr/lib64 # ls -lart libexp*
-rwxr-xr-x 1 root root 196944 May 29  2009 libexpect5.44.1.11.so
-rwxr-xr-x 1 root root 170984 Feb 20  2010 libexpat.so.0.5.0
lrwxrwxrwx 1 root root     17 Jul  2  2012 libexpat.so.0 -> libexpat.so.0.5.0
  • Second control after generating the soft link:
       vcenter:/usr/lib64 # ldd /usr/lib/vmware-vix-disklib/lib64/libdiskLibPlugin.so
       linux-vdso.so.1 =>  (0x00007fff90f39000)
       libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f83497de000)
       libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f83494d3000)
       libdl.so.2 => /lib64/libdl.so.2 (0x00007f83492cf000)
       libcrypto.so.0.9.8 => /usr/lib/vmware-vix-disklib/lib64/libcrypto.so.0.9.8 (0x00007f8349053000)
       libssl.so.0.9.8 => /usr/lib/vmware-vix-disklib/lib64/libssl.so.0.9.8 (0x00007f8348f02000)
 libexpat.so.0 => /usr/lib64/libexpat.so.0 (0x00007f8348cd8000) 
       libz.so.1 => /lib64/libz.so.1 (0x00007f8348ac2000)
       libtypes.so => /usr/lib/vmware-vix-disklib/lib64/libtypes.so (0x00007f834401a000)
       libvmomi.so => /usr/lib/vmware-vix-disklib/lib64/libvmomi.so (0x00007f83438c6000)
       libvmacore.so => /usr/lib/vmware-vix-disklib/lib64/libvmacore.so (0x00007f8342e34000)
       libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8342c1d000)
       libc.so.6 => /lib64/libc.so.6 (0x00007f83428a9000)
       /lib64/ld-linux-x86-64.so.2 (0x00007f8349f88000)
       libm.so.6 => /lib64/libm.so.6 (0x00007f8342630000)
       libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f83423f4000)


Restore of VMDK with transport mode SAN fails with "Write to disk failed: Error 1365799600129 at 4110" or similar message

Error occurs during restore shortly before all data are written.

Reason

  • This issue is related to VMware, not to SEP sesam. This usually just occurs, if a VMDK has been created with capacity value MB (megabyte) instead of GB (gigabyte) or TB (terabyte) and in this case just, if the value of the size is not a integer number (i.e. the value 16.3MB is not a integer number, but 16MB is).

Solution

  • VMware fixed the problem in version 5.5 of ESXi. If it is not possible to upgrade the ESXi, the restore can be done via all other transport modes (NBD,NBDSSL,HOTADD). To do this open the expert options in the restore wizard and enter -a trans=hotadd:nbd:nbdssl in the options field

Restore of VMware machine via SAN with Windows data mover fails with "SAN transport error: I/O - Operation failed"

Restore fails with given message above right after starting data transfer.

Problem:

2014-10-13 13:32:45: sbc-3925: Info:    VDDK:    INFO: 2014-10-13T13:32:45.260+02:00 [02192 error 'Default'] Incomplete write to, Wanted 65536 Got 0, Error 2 (19)
2014-10-13 13:32:45: sbc-3925: Info:    VDDK:    INFO: 2014-10-13T13:32:45.262+02:00 [02192 error 'Default'] Incomplete write to, Wanted 65536 Got 0, Error 2 (19)
2014-10-13 13:32:45: sbc-3925: Info:    VDDK:    INFO: 2014-10-13T13:32:45.264+02:00 [11876 error 'Default'] San transport error: I/O Operation failed.
2014-10-13 13:32:45: sbc-1031: Error:   Cannot write to remote archive: [VixDiskLib: VixDiskLib_Write: Write to disk failed: Error 17592452332404352 at 4116.

Reason:

  • The SAN volume on Windows data mover is offline, which implies read only access to this volume.


Solution:

  • Change volume status from offline to online by using the Windows disk management tool. Just right click the affected volume and change its status, then try again.
  • Additional information: Check SAN settings via DISKPART. Use command SAN to check current policies. The policy should be OnlineAll. This can be changed with SAN POLICY=OnlineAll


Connect a SCSI device to a VM

  • Enable Passthrough at Configuration -> Advanced Settings and restart the ESX server.


Attention: *** IMPORTANT ***
If Passthrough is grayed out and not selectable, either the passthrough function of PCI devices in BIOS has to be enabled or the mainboard does not support this feature. The following should be checked in BIOS:
  • Intel VT for Directed I/O -> Enab.
  • Interrupt Remapping -> Enab.
  • Coherency Support -> Disab.
  • ATS Support -> Enab.
  • Pass-trough DMA-Support -> Enab.


Attention: *** IMPORTANT ***
!!Paravirtualization is not supported!!

Finally, you have to do the following steps:

  1. Add a new PCI device in config of the VM.
  2. The passthrough adapter should be selectable now and has to be chosen.
  3. Now the controller with all connected devices is going to be passed through to the VM.
  4. The native driver of the controller has to be installed at the guest OS.


Attention
  • The SCSI controller can only be used once with the passthrough method.
  • Additionally, be aware that no more snapshots my be created for this VM.

See also

VMware Backup 4.4.3

Single Item Restore from VMware vSphere Backup

Instant Recover from VMware vSphere Backup

Standard Backup Procedure

Further Links/Literature