Archive:SEP sesam backup client for VMware vStorage API 4.4.1/4.4.2
THE CONTENT OF THIS PAGE IS OUTDATED | |
SEP AG has discontinued support for obsolete SEP sesam versions. Instructions are still available for these SEP sesam products, however, SEP AG accepts no responsibility or liability for any errors or inaccuracies in the instructions or for the incorrect operation of obsolete SEP sesam software. It is strongly recommended that you update your SEP sesam software to the latest version. For the latest version of SEP sesam documentation, see documentation home. |
Copyright © SEP AG 1999-2024. All rights reserved.
Any form of reproduction of the contents or parts of this manual is allowed only with the express written permission from SEP AG. When compiling and designing user documentation SEP AG uses great diligence and attempts to deliver accurate and correct information. However, SEP AG cannot issue a guarantee for the contents of this manual.
Introduction
This article is relevant for SEP sesam version 4.4.1/4.4.2 and describes SEP sesam extension for VMware vStorage APIs for Data Protection (VADP). Note that this is not the latest version of SEP sesam VMware documentation and, as such, does not provide information on features introduced in 4.4.3 and described in VMware Backup.
The SEP sesam extension for VMware vStorage APIs for Data Protection (VADP) provides hot backups of Virtual Machines (VMs) running on VMware ESXi servers. To be able use the vStorage API at least a VMware vSphere Essentials license must be purchased. Consistent backups are achieved by creating a snapshot of the virtual machine and transferring data of the virtual disk (VMDK) files directly to the SEP sesam Server or SEP sesam Remote Device Server. If a storage network environment is configured properly data can be transferred from ESXi server to SEP sesam Remote Device Server directly over SAN to avoid network traffic.
Advantages
- No workload on the ESX Server
Disadvantages
- Virtual machines must be on a SAN storage device to avoid data transfer over network
Known Issues
Restore of vSphere 6 does not work with 4.4.1 |
|
Known issues |
|
22.10.2013 Backup of virtual machines with two VMDK files with same file name does not work at the moment |
A virtual machine with name "vmware" for example uses following VMDK files: [Datastore1]/vmware/vmware.vmdk and [Datastore1]/vmware_org/vmware.vmdk
|
The restore of a VM configuration uses a lot of harddisk space |
If only the configuration of a VM containing thick provisioned VMDKs is restored, the new VMDKs will have the same size as the original VMDKs.
|
System requirements
- Supported versions of VMware hypervisors
- SEP sesam Server version >= 4.4.1.41
- For backups of VMs over SAN transport mode: The SEP sesam DataMover must operate as a physical machine with access to the Fibre Channel or iSCSI LUN, containing the virtual disks to be accessed!
- For iSCSI LUN connection see also: http://linux.die.net/man/8/iscsiadm or http://technet.microsoft.com/de-de/library
An update to latest SEP sesam version 4.4.x is recommended by SEP
Using changed block tracking |
To use CBT SEP sesam version 4.4.1.41 or higher is required |
- SEP sesam DataMover for VMware vSphere
- Windows Server 2008 [R2] or Windows Server 2012 acting as SEP sesam DataMover for VMware vSphere. This could be the SEP sesam Server, the vCenter Server or any other SEP sesam Client.
or - Linux server acting as SEP sesam DataMover for VMware vSphere. This could be the SEP sesam Server, the vCenter Server or any other SEP sesam Client.
- SLES 11 64 bit SP1 and SP2
- SLES 10 64 bit SP4
- RHEL 5.9 64 bit
- RHEL 6.2 64 bit
- RHEL 6.3 64 bit
- RHEL 6.4 64 bit
- Windows Server 2008 [R2] or Windows Server 2012 acting as SEP sesam DataMover for VMware vSphere. This could be the SEP sesam Server, the vCenter Server or any other SEP sesam Client.
- The following packages and modules have to be installed on the DataMover:
- PowerShell 2.0 (Windows DataMover)
- VMware Virtual Disk Development Kit 5.x (included by SEP sesam on Windows)
- SEP sesam Client package version >= 4.4.1.41 or later (Server-, GUI- or RDS- package will work too).
- Best Practice: Use a physical Windows system with vmware SAN connectivity as DataMover.
Usage of SAN/HOTADD with vSphere6
- To use SAN or HOTADD with vSphere6, VDDK 5.5.4 is necessary. All VDDK versions below cannot esablish a SAN or HOTADD connection, just NBD and NBDSSL.
Version of VMware Virtual Disk Development Kit |
Because SEP sesam extension is linked against a certain VDDK, the following version will be required with SEP sesam version 4.4.1: VDDK 5.5.1:
VDDK 5.5.3 respectivelly VDDK 5.5.4:
|
Functional principle
To run a SEP sesam backup and restore in a vSphere environment with vStorage API a SEP sesam Client is necessary. This functions as a SEP sesam DataMover (central communication element) between SEP sesam Server and VMware vSphere Farm.
The arrows in the illustration show the direction of the connection establishment. After, data certainly flows in both directions.
The initial communication goes from the SEP sesam Server to the vCenter Server. For this the vCenter Server has to be registered as a client in the SEP sesam environment. Communication for browsing starts over the SEP sesam control port 11301 (1) and goes to the vCenter Server. This means that when setting up a backup task you're always shown the file structure of the vCenter first. Now if the VMware vSphere branch is chosen, the https-connection (2) from the SEP sesam Server to the vCenter Server is started. Here, all virtual machines are identified and displayed.
When SEP sesam Server starts the backup it initiates a SEP sesam control connection (Port 11301) to the datamover (3). The datamover connects over the open source LIBVIRT library via https to the vCenter Server (4). There all parts of the virtual machine that are going to be backed up are identified. Now before a new SEP sesam backup snapshot is created all possibly still existing old SEP sesam backup snapshots are deleted. After taking the snapshot the datamover establishes a connection via the VMware VDDK on port 902 (6) to the ESXi server, where the VM that is being backed up, is hosted. Now the files of the VM are transferred from the ESXi server over the VDDK to the datamover and then further over the SEP sesam Transfer Protocol (7) to the SEP sesam Server where they are saved onto a medium of the chosen media pool.
For restores the way is the same except backwards.
vCenter Server, Data Mover and SEP sesam Server can run on the same machine.
If Data Mover runs virtualized on the Data Center from which VMs should be saved, SAN backups will not work.
Restrictions
Restrictions |
|
VDDK installation
The SEP sesam requires the VMware Virtual Disk Development Kit (VDDK) to be installed on SEP sesam Data Mover (SDM-VV). From SEP sesam version 4.4.1.x onwards, the VDDK is installed automatically during the SEP sesam installation.
Note | |
It is recommended to use VDDK 5.5.4 with SEP sesam 4.4.2, while SEP sesam 4.4.3 supports VDDK version 5.5.4 and 6.0.2. |
The VDDK installation procedure differs according to the operating system.
For information on VMware VDDK release notes, known issues and workarounds, see Further Links/Literature at the end of this document.
On Windows
SEP sesam 4.4 installation package with VDDK 5.5.4 |
With SEP sesam v. 4.4 either Virtual Disk Development Kit (VDDK) 5.5.1, 5.5.2, 5.5.3, or 5.5.4 for Windows is included in the SEP sesam installation package depending on the SEP sesam version of the installation package. When upgrading to or installing the SEP sesam v. 4.4, the VDDK version 5.5.1/5.5.2/5.5.3/5.5.4 is automatically installed. If a VDDK is already present on the system to which the SEP sesam is going to be installed or upgraded, the previous VDDK version is uninstalled and the VDDK 5.5.1/5.5.2/5.5.3/5.5.4 is unpacked. |
The VDDK 5.5.0 requirements with SEP sesam version 4.2.2.40 or newer
- Supported operating systems are Windows Vista/Windows Server 2008 and higher.
- If the VDDK is already installed on a system, manually uninstall it. Make sure that the registry path HKLM:\SOFTWARE\Wow6432Node\VMware, Inc.\VMware Virtual Disk Development Kit does not exist anymore.
- The VDDK 5.5.0 zip file (starting with VMware-vix-disklib) should be present in the <sesam-root>\skel directory. If the VDDK 5.5.0 zip cannot be found, download it from the VMware homepage and copy it into the skel directory.
- With the VDDK 5.5.x installation new parameters are introduced in the set_vddk.ps1 script:
- -e install is used for the VDDK 5.5.x installation
- -e update is used for updating the VDDK 5.5.x to the latest VDDK (i.e 5.5.1)
- -e remove is used for removing the VDDK 5.5.x structure
- If you want the SSL certificates to be validated, specify the -ssl 1 parameter. It is available with parameters -e install and -e update. The default value is 0.
- Use the additional parameter -f to overwrite currently installed VDDK 5.5.x. The syntax is -f -e install.
- The parameter -h shows the usage of the script.
After the VDDK 5.5.x installation, the necessary 64-bit libraries are located as a ZIP file in the VMware program directory. SEP sesam package includes a PowerShell script to install these libraries and set the registry key correctly.
With the SEP sesam v.4.2.2.40 and on SEP sesam clients, you need to perform additional steps.
- Log in to the DataMover.
- Open the
<SESAM_ROOT>/bin/sesam/
directory. - Execute the set_vddk.ps1 script. Note that in the PowerShell the command line .\ hast to be prefixed.
To use VDDK 5.5 on Windows RDS/Clients, the PowerShell script <SEPsesam>\bin\sesam\set_vddk.ps1 is used during update. The script can be manually called with set_vddk.ps1 -e install. To force the installation, use the switch -f: set_vddk.ps1 -e install -f.
Attention |
By default, the Powershell execution policy is set to Restricted. To execute the PowerShell scripts, set the execution policy to RemoteSigned by using the set-executionpolicy remotesigned command. Note that you must be logged in as an administrator or start the script with administrative privileges. For more information on PowerShell script execution policy, see Using the Set-ExecutionPolicy Cmdlet. |
On Linux
- Download the appropriate VDDK for Linux from the VMware download center. Then unpack and install the VDDK, as shown in the following VMware VDDK installation example on Linux 64-bit: VDDK version 5.5.4 tar xvzf VMware-vix-disklib-5.1.1-774844.x86_64.tar.gz cd vmware-vix-disklib-distrib/ ./vmware-install.pl
- VDDK version >= 6.0.0
- On the system which acts as a data mover, the LD_LIBRARY_PATH has to be set to the VDDK library path.
- For regular CTRL access to SEP sesam data mover, edit
<SESAM_VAR>/var/ini/sm.ini
and add/modify the following lines:
For 32-bit Linux:
[ENVIRONMENT]
LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib32:/usr/lib/vmware-vix-disklib/plugins32:$LD_LIBRARY_PATH
For 64-bit Linux:
[ENVIRONMENT]
LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib64:/usr/lib/vmware-vix-disklib/plugins64:$LD_LIBRARY_PATH
- For SSH access to SEP sesam data mover, edit
/etc/sesam2000.ini
and add/modify the LD_LIBRARY_PATH line:
For 32-bit Linux:
VERSION=4.2.1.41
SM_BIN_SESAM=/opt/sesam/bin/sesam/
...
LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib32:/usr/lib/vmware-vix-disklib/plugins32:$LD_LIBRARY_PATH
For 64-bit Linux:
VERSION=4.2.1.41
SM_BIN_SESAM=/opt/sesam/bin/sesam/
...
LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib64:/usr/lib/vmware-vix-disklib/plugins64:$LD_LIBRARY_PATH
- For regular CTRL access to SEP sesam data mover, edit
- Restart the SEP sesam service for the changes to take effect.
Note | |
SEP only supports installation of VDDK into the default path! |
Best Practices
Scalable LAN-free Backup over SAN Transport
LAN-free Backup over SAN Transport is the fastest transport method for SEP sesam Backup on SAN connected ESXi hosts. The VM Backup performance can be scaled up with SEP sesam. This allows simultaneously backup of a large number of virtual machines.
The following diagram describes the structure and the operating principle for scalable LAN-free backup over SAN transport:
The modular structure of SEP sesam supports the "scale-out" principle. This allows to use multiple identical Remote Device Server (RDS).
Requirements
The following requirements apply to the "scalable LAN-free backup over SAN transport" scenario:
Generell requirements for SAN transport
- The VMware datastore must support hardware acceleration. Therefore open vSphere Client, choose ESXi host and switch to tab “Configuration”. Then choose “Storage” and check, whether hardware acceleration is supported by the specified VMware datastore.
- Also hardware acceleration must be enabled on each ESXi host. Therefore check tab “Configuration” and choose “Advanced Settings”. Select section “DataMover” and ensure, both values are set to 1. Also select section “VMFS3” and make sure “VMFS3.HardwareAcceleratedLocking” is set to 1.
Additional requirements for SAN transport with Linux proxy
On Linux a SCSI reservation conflict can occur during SAN access. This causes the backup/restore to fail. To prevent this, the following things need to be checked.
A terminal has to be started on each ESXi server to execute the following command:
vmkfstools -Ph -v1 /vmfs/volumes/<name of VMware datastore>
example:
vmkfstools -Ph -v1 /vmfs/volumes/SAN-test-volume/
The output could include this:
Mode: public Partitions spanned (on "lvm"): naa.600000e00d280000002801dd00050000:1
If mode is public, then the problem can occur. In this case ATS (Atomic Test & Set) has to be enabled. No VM should run during this time on this VMware datastore (recommendation by VMware). This command needs to be executed on each ESXi server with mode public:
vmkfstools --configATSOnly 1 /vmfs/devices/disks/<naa-id]:[partition-number>
example:
vmkfstools --configATSOnly 1 /vmfs/devices/disks/naa.600000e00d280000002801dd00050000:1
The device name “naa.600000e00d280000002801dd00050000:1” was shown with first command above. After this step the output of the first command above should show this now:
Mode: public ATS-only
Up from this time ATS is used and then sporadically SAN errors on Linux proxy shouldn’t appear anymore.
SEP sesam Backup Server:
- Hardware minimal requirements: Intel 4 Core Server CPU or AMD Server 6 Core CPU, min. 16GB RAM
- Backup LAN: 10 Gbit (the VM backup data is transferred through the SAN)
- SAN connectivity: For the backup server not required
SEP sesam Remote Device Server:
- Hardware minimal requirements: Intel 4 Core Server CPU or AMD Server 6 Core CPU, min. 16GB RAM
- Backup LAN: 10 Gbit (the VM backup data is transferred through the SAN)
- SAN connectivity: All vmware LUN's must be connected via SAN and be visible. For iSCSI example see also: http://linux.die.net/man/8/iscsiadm
- Backup Storage: Connection of the tape hardware and disk backup storage via FC, iSCSI, or SAS
- VDDK: On all Remote Device Server the VMware VDDK 5.5.3 or 5.5.4 must be installed
- vCenter: can be installed on a remote device server as vCenter for Windows, or the VMware vCenter appliance can be used as a VM
- Datamover (Proxy)
Configuration
Activating Changed Block Tracking (CBT) globally
For using Changed Block Tracking (CBT), you must enable the CBT functionality globally. Afterwards, when configuring a backup task, you can decide whether you want to use the CBT in the respective backup task or not. For details, see the section Changed Block Tracking (CBT).
In the SEP sesam GUI, from the menu bar select Configuration -> Defaults -> tab Extras, and then select Globally activate Change Block Tracking (CBT).
Configuration with vCenter
Depending upon your SEP sesam version, the procedure for configuring vCenter server may be slightly different. Refer to the relevant version-specific steps below.
1. Configure Virtual Center server in the SEP sesam GUI as a normal Windows or Linux client. In the Main selection -> Components -> Topology -> New client, select the appropriate operating system. You can select a location to which you want to add a new client from the drop-down list of configured locations.
2. Define the client as a vCenter server.
SEP sesam version 4.4.2
In the New client properties, from the VM Server type drop-down list, select the option vCenter.
SEP sesam version 4.4.1 and older
In the New client properties, select the check box Client is a vCenter Server.
3. Click the tab vCenter Access and add a vCenter credentials. The following screenshots show version specific access dialog.
4. Select a data mover. The VMware backup performance is highly dependent on the transport mode. Please see the recommendations for the optimal Data Mover: Transport Modes
Configuration with ESX Server
In case no vCenter is available, the ESX Server itself can be added to SEP sesam. In the Main selection -> Components -> Topology -> New client, add the ESX Server with its DNS name, select ESX-Server as operating system and Proxy as access mode. Then select the tab ESX Server Access and enter the credentials.
Backup
In SEP sesam GUI create under the vCenter Client a new backup task with task type VMware vSphere. The VM name (backup source) can be selected using the client file system browser under VMware vSphere.
The backup source has the following format: //<display name of VM>
Example:
//<VM name>
VM SEP-DC01 in data center esxixfix:
Source=/esxixfix/SEP-DC01
Exclude of VMDKs
One or several VMDKs can be excluded from backup by specify complete path or symbolic name in exclude option field. These values have to be entered manually. The symbolic names reference the order of VMDKs of virtual machine definition.
The 1st virtual disk is vmdk0, the 2nd is vmdk1, and so on.
VM backup task dialog
VM task offers several options, which influence backup process.
- Backup VM configuration only (no VMDK data)
- Backup of OVF and CONF file. Used to create VM that contains virtual devices but no data or operating system
- Backup VM config and VMDK in separate savesets
- VM config (OVF-File) and each VMDK of VM are stored in its own backup files
- Change block tracking active
- New: Activate/deactivate change block tracking. Enable this option to reduce data size (see below)
Snapshot Parameters
- Snapshot creation with quiescence (needs VMware tools installed and VSS on Windows otherwise an error such as 'failed to quiesce' may occur):
- Quiescing a file system is a process of bringing the data on disk to a consistent state. This process might include such operations as writing cache to disk.
Example in the tab Options 1 in Backup options:
-a qui=<1|0> (Default: '1')
Transport modes
The VMware backup performance is highly dependent on the transport mode. VMware offers several transport modes for transferring the data from VMware datastore to the backup device.
The table shows recommendations for the Data Mover, depending on the storage type and desired transport mode:
SAN Transport Mode (recomended) | HotAdd Transport Mode | Network Transport Mode | |
---|---|---|---|
Performance: | Very fast | Fast | Slow (It might works well with 10 Gbit) |
FC/iSCSI SAN Storage: | Data Mover: The physical SEP Sesam Server / RDS. | Data Mover: A Windows VM with Sesam Client. | Data Mover: The SEP Sesam Server. |
NFS and local Storage: | Not available. | ||
Requirements for the Data Mover: |
|
|
- SAN Transport Mode
- To use SAN as transport mode during backup, VMDKs of VM must be on a SAN device accessible by the SEP sesam data mover. Data Mover must not be a VM.
- HOTADD Transport Mode
- Back up VMDKs by hot-add them to Data Mover VM. So the Data Mover must be a VM running in the same data center as VM which is to be saved.
- NETWORK Transport Mode(NBD)
- The Network Block Device mode transports the data over the regular LAN. This method will be always available.
- NBDSSL
- Same way like NBD, but the data will be encrypted by SSL.
By default, the following order is set in the vStorageAPI: san:hotadd:nbdssl:nbd. Using the backup option trans, the transport mechanisms that are to be used by the vStorage API can be forwarded with the backup task:
Example in the tab Options 1 in Backup options:
-a trans=nbd:nbdssl
Here, the data stream is set to travel via LAN, regardless of encryption.
Which transport mode can be used or should be used depends on your virtualization environment and of the recommendations and restrictions of the VMware vStorage API. (Detailed Information see VMware VDDK virtual disk transport methods)
Attention: A Data Mover with linux is experimental and is not recommended!
Changed Block Tracking (CBT)
New in SEP sesam 4.2.1.40
- CBT is a VMware feature to perform block level incremental and differential backups of VMDKs. VM's running on ESX/ESXi hosts 4.0 or later can track changed disk sectors and identify all sectors that are in use on VMFS partitions.
Technical information about CBT from VMware: |
For best results, when using QueryChangedDiskAreas (this API function is used by a CBT backup) to gather information about snapshots, enable change tracking before taking snapshots. Attempts to collect information about changes to snapshots that occurred before change tracking was enabled result in a FileFault error. Enabling change tracking provides the additional benefit of saving disk space because it allows for the backup of only information that has changed. If change tracking is not enabled, it is unknown what has changed, so the entire virtual machine must be backed up each time, rather than only backing up changed data. Be aware that entire disks are allocated, and therefore are returned when QueryChangedDiskAreas is used in the following scenarios:
|
- When SEP sesam starts a 'DIFF' or 'INCR' backup, it can request transmission of blocks changed since the last FULL backup only. In case of a 'FULL' backup only used blocks in a VMDK file are requested. The backup type 'COPY' stores all data of a VMDK including empty blocks (as provisioned).
Information about CBT
- Supported with Sesam version '4.2.1.40' or above
- VM CBT is necessary for VM FULL/DIFF/INCR backup.
- Independent disks are not supported by CBT.
- The host must be ESX/ESXi 4.0 or later.
- CBT can be manually enabled/disabled at backup task dialog.
VM backup type 'COPY'
- Backup VM with type 'COPY' will back up the whole VMDK including empty blocks (provisioned size = size of backup and size of VMDK equals).
- CBT is not required and will not be activated.
VM backup type 'FULL'
- Backup VM with type 'FULL' will only backup used blocks of VMDK files.
- VMDK files must be located on a VMFS volume, backed up by SAN, iSCSI, or local disk. VMware RDM (Raw Device Mapping) is not VMFS.
- Backup of used blocks will not be able for VMDKs stored on NFS volume. Nevertheless 'DIFF' and 'INCR' backups work.
- CBT will be activated by Sesam for a VM, if a backup starts with level 'FULL'.
VM backup type 'INCR'/'DIFF'
- This operation requires same conditions as 'FULL' backup. DIFF and INC typed CBT backups are working on the differential way as normal path backups but only on block level.
- VM CBT must be activated and a FULL backup has to be executed successfully before.
Attention |
Do not disable CBT, otherwise the whole VM FULL/DIFF/INC chain will be destroyed and further VM DIFF/INC backups are impossible and a new backup chain with type 'FULL' must be created. |
Restore
The restore can be started with the regular SEP sesam restore wizard.
- All vCenter Servers that are configured as SEP sesam Clients can be selected for restore.
- The according Data Centers and ESXi servers can be selected.
- If different login credentials are needed SEP sesam will ask for them.
- If vCenter is not set, the restore will be done directly over the ESXi server
Select the according restore task and confirm the selection with <Next>.
On the page Save and Start of the restore wizard, the valid VMware infrastructure values for the backup are shown. If the VM is not supposed to be backed up to its original state, different existing values for VMware Datacenter, ESXi server and VMware DataStore can be selected. If necessary, you'll be asked to enter the valid users/passwords to authenticate to the vCenter and to the ESXi Server.
Additionally it is possible to verify the login information with the <Check>-button. If clicking the <Check>-button doesn't cause a login dialog to appear, the stored information is valid.
If the option Overwrite is set, an existing VM with the same name will be deleted before the restore starts
Troubleshooting
Message
SSL handshake error during browse of vSphere server
- The name "VMware vSphere:" appears again after expanding it.
- A pop up opens, which refers to this page
Reason
- This issue is related to VMware vSphere 5.5 (may also 5.1) and IBM Java due to changes according OpenSSL.
Solution
There are two possible solutions to solve the issue:
- install Oracle Java instead of IBM Java, because Oracle Java is able to connect to vSphere 5.5. After installing Oracle Java, the java link in /opt/sesam/bin/sesam needs to be adjusted to redirect to Oracle Java.
- Change list of allowed cyphers on the vSphere server. Please have a look at http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2049143 for further information.
Message:
Error: VM Exception: [HostCommunication].
- A VADP backup ends with the error message mentioned above.
- In the event log of the hyper visor the following error message occurs at the same time if
- a snapshot should be created or deleted on this host
- a virtual machine should be migrated to or from this host
- a virtual machine on this host should be cloned
An error occurred while communicating with the remote host.
Reason
One of the management services on the hypervisor is not or not correctly running.
Solution
- Log on to the to the ESXi server via ssh
- restart the vpxa Service
/etc/init.d/vpxa restart
See also: Restarting the Management agents on an ESXi or ESX host
Message:
Execution error of set_vddk.ps1 according vmware-vdiskmanager.exe
Reason
- The depending msvcr90.dll for this application is not available.
Solution
- Install the "Microsoft Visual C++ 2008 Service Pack 1 Redistributable Package ATL Security Update", which includes the library: http://www.microsoft.com/en-us/download/details.aspx?id=11895
Message:
GUI reports Login denied during browsing the vSphere farm
- When vCenter user and password have been verified an d the error is still running, you can check the VMware farm access without a graphical user interface (GUI)
Reason
- Occasionally, there are troubles with special characters in passwords. For a better analysis of these problems, it could help to check the VMware vSphere farm connection over command line.
Solution
- In the following command you have to replace the vCenter server name, username and password with your own environment.
Pattern:
sbc_vadp -D -a username=<vCenter username>,password=<password>,url=https://<vCenter Hostname>/sdk,ignorecert=ignorecert "/VMware vSphere:"
Example:
sbc_vadp -D -a username=Administrator,password=mypassword,url=https://ws2008x64/sdk,ignorecert=ignorecert "/VMware vSphere:"
It should then be listed in the configured VMware Data Center. Is that not the case, then it is helpful first to set the password to be easier without special characters. Is the access test to the farm successful, then the original password contains a character that causes a problem accessing the VMware farm.
Message:
svp-1901: Error: VM Exception: [Exception=FileFault,dynamictype=null]. -> CBT-RESET needed
- Error occurs if a snapshot is not deleted after a CBT backup
Reason
- CBT is not in a clean state
Solution
- Reset CBT (see next chapter)
Possible choices to reset CBT via command line or via GUI
Soft reset (On line) Generally available up from version 4.2.2.26 (up from version 4.4.1.x a soft reset is also possible via GUI and via CLI, too. This is described below the next command)
- When executing the following command, the virtual maschine does not need to be shut down. The command needs to be executed in directory <sesam-root>\bin\sesam:
- sbc_vadp -b -a "action=softreset,server=<vSphere Server>,username=<user>,password=<password>" "<datacenter>/<vm name>"
Example
sbc_vadp -b -a "action=softreset,server=ws2008x64,username=Administrator,password=secret" "DC/linuxdbserver"
Soft reset (On line) via GUI respectively via CLI (sm_cmd)
- As described above, a soft reset of CBT is also possible via GUI and CLI up from version 4.4.1.x of SEP sesam.
- Via GUI this can be done for each task. The button "Reset CBT" is available on the first tab for each task at "Tasks > by Clients" when opening the properties of a specified task.
- The second possibility is to use CLI (sm_cmd):
- sm_cmd resetcbt -S <vCenter or ESX> -d -V <Name of VM>
Example
sm_cmd resetcbt -S qsbox1 -d "SEP Cloud" -V "cosinus (SEP)"
Hard reset (Off line) (up from version 4.4.1.x a hard reset is also possible via CLI, too. This is described below the next command)
- This command should be used, if a soft reset does not work or if it's not available. Look at requirements below for further information.
- sbc_vadp -b -a "action=resetcbt,server=<vSphere Server>,username=<user>,password=<password>" "<datacenter>/<vm name>"
Example
sbc_vadp -b -a "action=resetcbt,server=ws2008x64,username=Administrator,password=secret" "DC/linuxdbserver"
- As described above, a hard reset of CBT is also possible CLI up from version 4.4.1.x of SEP sesam. The syntax for this command is:
- sm_cmd resetcbt -S <vCenter or ESX> -d -V <Name of VM> -m hard
Example
sm_cmd resetcbt -S qsbox1 -d "SEP Cloud" -V "cosinus (SEP)" -m hard
Requirements
- The state of the virtual machine must be powered off for this action.
Manual CBT reset
Alternatively, via the vSphere client, a CBT-reset will be performed:
On line
With the following steps the virtual machine does not need a shut down to reset CBT:
1. Open the properties of the affected virtual machine at "Tasks > By Clients" 2. Remove the tick "Enable Change Block Tracking (CBT)" and save the task. 3. Create a snapshot via vSphere Client of the affected virtual machine (none of the ticks needs to be selected) 4. Delete the snapshot, which has been created before. 5. In SEP sesam --> activate the CBT option in the VM client settings --> start the VM backup
Off line
This step needs to be done, if the a reset of CBT in on line mode does not work. With this steps the virtual machine must be turned off:
1 Shutdown the VM 2 Open the VMware Snapshot-Manager --> delete all snapshots 3 Right click the VM --> "Edit settings" --> "Options" tab --> "General" -> "Configuration Parameters" 4 Set the value "ctkEnabled" to false 5 Set the value "scsi0:0.ctkEnabled" to false (NOTE: Set the value "false" for each disk) 6 Open the folder where the VM *.VMDK files exists --> delete all *-CTK.VMDK files 7 Start the VM 8 Shutdown the VM (This is for the CTK table update needed) 9 Start the VM 10 In SEP sesam --> activate the CBT option in the VM client settings --> start the VM backup
Message
Error: At least one snapshot exists before CBT was enabled, which is not possible
'Possible Cause'
- VM is suspended
- Manual snapshots available
Solution
- Restart or shutdown VM
- Delete manually created snapshots and then possibly perform a reset CBT
Message:
VIX_E_FAIL
- Error occurs during backup of 2nd VMDK.
Reason
- Timeout in vCenter connection
Solution
- Update VDDK library to version 1.2.1
Messages:
VIX_E_FILE_NOT_FOUND:
or
VIX_E_FILE_ACCESS_ERROR:
or
You do not have access rights to this file:
or
Thin/TBZ/Sparse disks cannot be opened in multi writer mode.
Reason
- Wrong VMDK file has been linked for backup
Solution
- This problem will be fixed in a future version of Sesam (state Jul 2 2013).
Otherwise start backup with option -a qui=0. As a result of this option the involved system boots in normal state and shows message regarding safe boot. See this screen shot as an example on how to add this option:
Backup fails with error on a 64 Bit system even though VDDK is already installed:
Error: Load VDDK library failed: Cannot load: vixDiskLib.dll
Reason
- VDDK is only installed as a 32 Bit version
Solution
- execute the set_vddk64.ps1 script to install the 64 Bit version. See also VDDK installation on 64 Bit Windows.
Backup stops with message:
sbc-1500: Error: VixDiskLib_Open() failed: [sesam] sesam_vm/sesam_vm.vmdk:Cannot connect to the host
Reason
- The file vixDiskLib.dll does not exist in the directory
<SESAM_ROOT>\bin\sesam
of the VMware data mover.
Solution
- Copy the file
C:\Program Files\VMware\VMware Virtual Disk Development Kit\vixDiskLib.dll
to<SESAM_ROOT>\bin\sesam
of your VMware data mover.
VDDK/VIX side-by-side configuration is incorrect
Reason
- Required C++ libraries are not installed.
Solution
- Install tools application of the same version as vddk https://packages.vmware.com/tools/esx/index.html
Encrypted backup stops with Encryption/Decryption operation failed
Reason
- Backup was done with bf64 (blowfish 64) encryption.
Solution
- Encryption has to be changed to aes256 instead of bf64.
Message:
VM Exception: [InvalidRequest].
- Error occurs during VM creation with OVF config file.
Reason
- Will be thrown, if OVF file, that will be used for create VM during restore is invalid.
- Example: virtual CD-ROM with ISO file and file is not more available
Solution For successful restore, you must edit OVF file of VM and remove all virtual devices, which prevent creating VM.
- Restore only VM config files.
- Remove virtual CD-ROM with ISO from OVF file.
- You can find the file at the sesam directory:
<SESAM_ROOT>/var/tmp/_vadp_/<vm name>/<saveset of backup>
- You can find the file at the sesam directory:
CD-ROM specification part of ovf file <Item> <rasd:AddressOnParent>0</rasd:AddressOnParent> <rasd:AutomaticAllocation>true</rasd:AutomaticAllocation> <rasd:ElementName>CD/DVD Drive 1</rasd:ElementName> <rasd:HostResource>ovf:/file/file2</rasd:HostResource> <rasd:InstanceID>7</rasd:InstanceID> <rasd:Parent>4</rasd:Parent> <rasd:ResourceType>15</rasd:ResourceType> </Item>
- Start same restoretask and restore VM with VMDK
For successful restore, you must edit OVF file of VM and remove all virtual devices, which prevent creating VM.
svp-3980: Info: Create VM from OVF file: 'C:\ProgramData\SEPsesam\var\tmp\_vadp_\EX-SERVER\SC20130423113643791@70RYUkBmDGz\EX-SERVER.ovf' svp-1901: Error: VM Exception: [VM creation failed (error)].
VM creation breaks with error:
svp-3980: Info: Create VM from OVF file: 'C:\ProgramData\SEPsesam\var\tmp\_vadp_\EX-SERVER\SC20130423113643791@70RYUkBmDGz\EX-SERVER.ovf' svp-1901: Error: VM Exception: [VM creation failed (error)].
Reason
- Various possible errors
Solution
- Analysis of the VMware Logs:
- Use vCenter <Administration>, <export system logs> to select the affected ESX-Server as source. Choose <Logs> only from system logs and specify desired export location.
- Once the export is complete, unzip the tgz file from the export directory.
- Then locate the host.d file and search for errors.
Possible errors
- Wrong video RAM size. Must be set to 4MB in "auto-detect" mode. The error message in host.d is:
opID=49F494FA-000001D6-6a-7b-3e-ec-f4] Video Ram size edit not supported when auto-detect is True ./var/log/hostd.log:2013-04-23T14:43:56.923Z [61B7EB90 info 'vm:/vmfs/volumes/5087d83e-bbbbd91c-4218-001999da71cf/testEX-SERVER/testEX-SERVER.vmx' opID=49F494FA-000001D6-6a-7b-3e-ec-f4] Reconfigure failed: vmodl.fault.NotSupported
Cannot display VMX file
Error occurs during download of VMX file.
Reason
- vCenter is not able to download the file due to known issues in older vCenter versions. Problem is described here: Using the vCenter Server datastore browser to download or copy a powered-on virtual machine's .vmx and .nvram files fails
Solution
- Update version of vCenter. This problem is not related to Sesam, but to vCenter only.
Error: Could not load default plugins from /usr/lib/vmware-vix-disklib/lib64/libdiskLibPlugin.so
- A VADP backup ends with: Error: VM Exception: [Exit code from sbc: [1] - warning].. In the log file you can see: Could not load default plugins from /usr/lib/vmware-vix-disklib/lib64/libdiskLibPlugin.so.
- The following causes can trigger the error:
- The correct LD_LIBRARY_PATH has to be set
- Solution:
Edit <SESAM_VAR>/var/ini/sm.ini
and add/modify the following lines:
For 32 bit Linux:
[ENVIRONMENT] LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib32:/usr/lib/vmware-vix-disklib/plugins32:$LD_LIBRARY_PATH
For 64 bit Linux:
[ENVIRONMENT] LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib64:/usr/lib/vmware-vix-disklib/plugins64:$LD_LIBRARY_PATH
- For SSH access to Sesam data mover:
Edit /etc/sesam2000.ini
and add/modify the LD_LIBRARY_PATH line:
For 32 bit Linux:
VERSION=4.2.1.41 SM_BIN_SESAM=/opt/sesam/bin/sesam/ ... LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib32:/usr/lib/vmware-vix-disklib/plugins32:$LD_LIBRARY_PATH
For 64 bit Linux:
VERSION=4.2.1.41 SM_BIN_SESAM=/opt/sesam/bin/sesam/ ... LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib64:/usr/lib/vmware-vix-disklib/plugins64:$LD_LIBRARY_PATH
- If the problem still exists after modifying the sm.ini and even after restarting the SEP sesam Service, we have to check if the libexpat.so.0 really exists on the chosen system
vcenter:/usr/lib64 # ldd /usr/lib/vmware-vix-disklib/lib64/libdiskLibPlugin.so
linux-vdso.so.1 => (0x00007fff539ff000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdab6937000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fdab662d000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fdab6428000) libcrypto.so.0.9.8 => /usr/lib64/libcrypto.so.0.9.8 (0x00007fdab6089000) libssl.so.0.9.8 => /usr/lib64/libssl.so.0.9.8 (0x00007fdab5e35000)
libexpat.so.0 => not found
libz.so.1 => /lib64/libz.so.1 (0x00007fdab5c1e000) libtypes.so => not found libvmomi.so => not found libvmacore.so => not found libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdab5a07000) libc.so.6 => /lib64/libc.so.6 (0x00007fdab5692000) /lib64/ld-linux-x86-64.so.2 (0x00007fdab70b0000) libm.so.6 => /lib64/libm.so.6 (0x00007fdab5419000)
- Search for the file libexpat.so.0
vcenter:/usr/lib64 # ls -lart libexp* -rwxr-xr-x 1 root root 196944 May 29 2009 libexpect5.44.1.11.so -rwxr-xr-x 1 root root 170984 Feb 20 2010 libexpat.so.0.5.0
- Result: The file libexpat.so.0 does not exist
- Resolve the issue:
ln -s libexpat.so.0.5.0 libexpat.so.0
- First control after generating the soft link:
vcenter:/usr/lib64 # ls -lart libexp* -rwxr-xr-x 1 root root 196944 May 29 2009 libexpect5.44.1.11.so -rwxr-xr-x 1 root root 170984 Feb 20 2010 libexpat.so.0.5.0 lrwxrwxrwx 1 root root 17 Jul 2 2012 libexpat.so.0 -> libexpat.so.0.5.0
- Second control after generating the soft link:
vcenter:/usr/lib64 # ldd /usr/lib/vmware-vix-disklib/lib64/libdiskLibPlugin.so linux-vdso.so.1 => (0x00007fff90f39000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f83497de000) libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f83494d3000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f83492cf000) libcrypto.so.0.9.8 => /usr/lib/vmware-vix-disklib/lib64/libcrypto.so.0.9.8 (0x00007f8349053000) libssl.so.0.9.8 => /usr/lib/vmware-vix-disklib/lib64/libssl.so.0.9.8 (0x00007f8348f02000)
libexpat.so.0 => /usr/lib64/libexpat.so.0 (0x00007f8348cd8000)
libz.so.1 => /lib64/libz.so.1 (0x00007f8348ac2000) libtypes.so => /usr/lib/vmware-vix-disklib/lib64/libtypes.so (0x00007f834401a000) libvmomi.so => /usr/lib/vmware-vix-disklib/lib64/libvmomi.so (0x00007f83438c6000) libvmacore.so => /usr/lib/vmware-vix-disklib/lib64/libvmacore.so (0x00007f8342e34000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8342c1d000) libc.so.6 => /lib64/libc.so.6 (0x00007f83428a9000) /lib64/ld-linux-x86-64.so.2 (0x00007f8349f88000) libm.so.6 => /lib64/libm.so.6 (0x00007f8342630000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f83423f4000)
Restore of VMDK with transport mode SAN fails with "Write to disk failed: Error 1365799600129 at 4110" or similar message
Error occurs during restore shortly before all data are written.
Reason
- This issue is related to VMware, not to SEP sesam. This usually just occurs, if a VMDK has been created with capacity value MB (megabyte) instead of GB (gigabyte) or TB (terabyte) and in this case just, if the value of the size is not a integer number (i.e. the value 16.3MB is not a integer number, but 16MB is).
Solution
- VMware fixed the problem in version 5.5 of ESXi. If it is not possible to upgrade the ESXi, the restore can be done via all other transport modes (NBD,NBDSSL,HOTADD). To do this open the expert options in the restore wizard and enter -a trans=hotadd:nbd:nbdssl in the options field
Restore of VMware machine via SAN with Windows data mover fails with "SAN transport error: I/O - Operation failed"
Restore fails with given message above right after starting data transfer.
Problem:
2014-10-13 13:32:45: sbc-3925: Info: VDDK: INFO: 2014-10-13T13:32:45.260+02:00 [02192 error 'Default'] Incomplete write to, Wanted 65536 Got 0, Error 2 (19) 2014-10-13 13:32:45: sbc-3925: Info: VDDK: INFO: 2014-10-13T13:32:45.262+02:00 [02192 error 'Default'] Incomplete write to, Wanted 65536 Got 0, Error 2 (19) 2014-10-13 13:32:45: sbc-3925: Info: VDDK: INFO: 2014-10-13T13:32:45.264+02:00 [11876 error 'Default'] San transport error: I/O Operation failed. 2014-10-13 13:32:45: sbc-1031: Error: Cannot write to remote archive: [VixDiskLib: VixDiskLib_Write: Write to disk failed: Error 17592452332404352 at 4116.
Reason:
- The SAN volume on Windows data mover is offline, which implies read only access to this volume.
Solution:
- Change volume status from offline to online by using the Windows disk management tool. Just right click the affected volume and change its status, then try again.
- Additional information: Check SAN settings via DISKPART. Use command SAN to check current policies. The policy should be OnlineAll. This can be changed with SAN POLICY=OnlineAll
Connect a SCSI device to a VM
- Enable Passthrough at Configuration -> Advanced Settings and restart the ESX server.
Attention: *** IMPORTANT *** |
If Passthrough is grayed out and not selectable, either the passthrough function of PCI devices in BIOS has to be enabled or the mainboard does not support this feature. The following should be checked in BIOS:
|
Attention: *** IMPORTANT *** |
!!Paravirtualization is not supported!! |
Finally, you have to do the following steps:
- Add a new PCI device in config of the VM.
- The passthrough adapter should be selectable now and has to be chosen.
- Now the controller with all connected devices is going to be passed through to the VM.
- The native driver of the controller has to be installed at the guest OS.
Attention |
|
See also
Item Restore from VMware vSphere Backup
Instant Recover from VMware vSphere Backup