5 0 0:Automating Backup Process

From SEPsesam
Jump to: navigation, search
Other languages:
Deutsch • ‎English
Draft.png WORK IN PROGRESS
This page is a draft. Treat the information on this page with caution as it may be incomplete.

Copyright © SEP AG 1999-2021. All rights reserved.

Any form of reproduction of the contents or parts of this manual is allowed only with the express written permission from SEP AG. When compiling and designing user documentation SEP AG uses great diligence and attempts to deliver accurate and correct information. However, SEP AG cannot issue a guarantee for the contents of this manual.

Docs latest icon.png Welcome to the latest SEP sesam documentation version 5.0.0 Jaglion. For previous documentation version(s), check Documentation archive.


Overview

This article describes automating backup process in v. ≥ 5.0.0 Jaglion. For previous documentation versions, see documentation archive.
With SEP sesam, you can back up all kinds of data in different virtual environments. But as the number of virtual servers increases - new virtual machines (VMs) are created and deleted every day in the dynamic environment - manual configuration of backup tasks becomes quite tedious and time-consuming.

To simplify the backup of multiple VMs, you can automate the process by automatically generating tasks for VMs associated to the same host; additionally, you can also automatically create clients for VMs to which the generated tasks and ACLs can be assigned. It is recommended to use these features whenever a new hypervisor is added and there are many new VMs that need to be backed up.

CLI commands and pre-backup script

SEP sesam provides specific CLI commands that can be used in the GUI as a pre-backup script that is executed before the backup job for the task group and automatically adds new tasks for VMs. As SEP sesam uses the default auto-generated names for backup tasks, you can customize the task name creation by using the task name template, as described in Task Name Template – Backup.

GUI VM task management

After the tasks are initially created, the administrator can use the GUI option Manage VM Tasks (recommended) or Customize Task List to create the task group based on the specified filters and include the task generation command in the pre-backup script of a task group. Alternatively, a command event can be used to run the task generation command on a schedule.

Restricting access

Tasks and virtual clients can be created under a specific location, and depending on your security requirements, access can be based on user types. For example, tasks with the auto_generated flag are read-only for a user with the backup role. For more details on SEP sesam user types, see User Roles and Permissions.

Scenario for automatic task creation with generating clients, task customization, and access restriction

You could use the following scenario to create the tasks, generate clients in different locations, customize the task naming convention to your needs, allow specific user types to start the tasks but exclude them from modifying already created tasks.

  1. Configure automatic generation of tasks.
  2. Configure automatic generation of clients.
  3. Use the Rythm template engine to customize the auto-generated task names.
  4. Use the GUI Manage VM or Customize Task List option to customize the task list.

Automatic task generation

The SEP sesam command line tool taskgen is used to create tasks according to the specified task type and can also be used to generate VM clients.

The following parameters are required for task generation:

sm_cmd add taskgen <task name> -c <client or VM server> -t <task type> -G <task group>

If the client is a VM server, the tasks are created for all VMs of the VM server.

To specify the source for multiple tasks and filter the list of VMs for the task to be created, use the –f parameter, which can be defined by the following two variants:

  • The file must contain a list of VMs or source notes in JSON format that can be created with the command and manually edited. For details, see the section Generating VM tasks.
  • sm_cmd add taskgen <task name> -c <client or vm server> -t <task type> -G <task group> -f <file path> 
    
  • The sm_cmd dir client command can be used directly as source input for task generation. For vSphere clients, the sm_cmd dir vsphere command can be used instead.
  • sm_cmd dir -F JSON client <client or vm server> [filter options] | sm_cmd add taskgen <task name>  -c <client or vm server> -t <task type> -G <task group> -f - 
    
Generating source input: sm_cmd dir client
  1. Use the sm_cmd dir client command to browse the client or client path. Refer to the command help (sm_cmd help dir client) for a description and usage of all parameters.
  2. SEP Tip.png Tip
    To verify that the correct tasks are generated, the input should first be executed without the task generation command and without -F JSON. Such output is more readable and therefore more verifiable.
  3. The output is a list of path nodes that can be converted to JSON format. Depending on the client, select the appropriate syntax:
    • For clients that are not hypervisors, use:
    • sm_cmd dir -F JSON client <client or path> [filter options]
      
    • Use the additional parameter -l (force flat mode) to output all VMs of a virtualization server:
    • sm_cmd dir -F JSON client –l 1 <virtualization server> [filter options]
      
SEP Tip.png Tip
For vSphere clients
  • Use the sm_cmd dir vsphere command as an alias to the command dir client. See sm_cmd help dir vsphere.
  • You can also use additional parameters to filter the output, for example, -P <on|off> to filter by powerstate.

If the output of any of the described commands is passed as input for task creation, a backup task is created for each node of the output. This applies to VMs of a virtualization server, to path nodes of a normal client, and also to subnodes of a database server.

Generating VM tasks

The following syntax is used for generating VM tasks:

sm_cmd dir -F JSON client –l 1 <client or vm server> [filter options] | sm_cmd add taskgen <task name>  -t <task type> -G <task group> -O <option> -f -

where

  • <client or vm server> is the case-sensitive name of the client or VM server as defined in the SEP sesam environment
  • <task name> is the desired task name, ideally generated with the Rythm template, see the section Customizing auto-generated names by templates
  • <task type> is the task type according to which the data is backed up, e.g., VMware vSphere, Hyper-V... (default is Path)
  • <task group> is the task group to which the tasks should be assigned

The following options can be set:

-G <group name>             #refers to the task group and is not mandatory (unless the -O option is used) 
-g group1[,group2]          #additional groups to which tasks should be added
-k                          #(vSphere specific) search in the view of all VMs and see quick sorting based on attributes Backup required=yes
-O <option>                 #specifies the options to use for task generation regarding the existing tasks; these options can be combined, see below

The possible values of the option parameter are:

-O remove_non_existing      #delete all tasks from the task group for VMs that no longer exist
-O remove_all               #delete all tasks belonging to the task group before creating new tasks
-O add_non_exiting          #create a task even if the VM no longer exists
-O remove_only              #delete all tasks for VMs that no longer exist but do not create new tasks
-O source_not_found         #create tasks only if no other task in the same task group is associated with the VM
  • When a task is generated, the flag auto_generated is set.
  • If a task is deleted due to the options remove_non_existing, remove_all or remove_only, the associated backup data is not deleted.

Automatic generation of VM clients

To automatically generate virtual clients for VMs with SEP sesam, the following parameters must be specified:

-n client name: If set, a client is generated for each virtual machine passed. The following placeholders are allowed:
  _AUTO_: use the guest host name of the virtual machine
  _IP_: use the guest IP address of the virtual machine
  _AUTO-DNS_: resolve the virtual machine guest IP address to the host name
Information sign.png Note
Placeholders only work if the required attributes, such as the host name or IP address, are available. Currently, _AUTO-DNS_ and _IP_ only work with VMware vSphere.
-L location for generated VM client: If the location or location tree is not found, it is automatically generated. If the location is not specified, the root location is used.
-w operating system of generated virtual machine client: If this value is set to _AUTO_, the operating system is guessed from the virtual machine. 
Information sign.png Note
Placeholders only work if the required attributes such as the guest operating system are present.

The following attributes are automatically set for the generated client:

a client is a virtual machine
access mode = VIRTUAL
the virtual server name is set by the client of the task
the virtual machine name, if available
operating system, if available (default Linux)
Auto_generated

Tasks are assigned to the client. In the Topology view they are displayed twice: once under generated client and once under VM server.

By using sm_cmd dir client ID you can browse the client's directory (check sm_cmd help dir for details).

Information sign.png Note
  • If the task generation options remove_non_existing, remove_all, or remove_only are set, the generated clients will not be deleted.
  • If task generation is invoked with the virtual client generation parameter and the corresponding task exists but not the client, only the client is generated and the task remains untouched.

Customizing auto-generated names by templates

You can use the Rythm template to automatically generate task, location or VM client names. For details on how to use task name templates with the Rythm syntax, see Task Name Template – Backup.

The following table lists useful placeholders that can be used when generating tasks names.

Name Description
{{client}} Get the name of the client associated with the backup task object.
{{client_short}} Get client without domain.
{{source}} Get the source property of the backup task object.
{{source_last_item}} Get last source segment separated by slash.
{{source_item[<pos>]}} Get segment position separated by slash (negative position as position counted from the end).
{{count}} Get a number between 1 and 10000. Evaluates to the first number in this range, resulting in a unique task name.
{{count_optional}} Almost the same as above {{count}} , but the count is only added if the generated task name is not unique.
{{task_type_short}} Get the short name of the task type.
{{source_pool}} Get the source media pool name from a migration or replication task (not applicable when generating backup tasks for virtual machines).
{{target_pool}} Get the target media pool name from a migration or replication task (not applicable when generating backup tasks for virtual machines).
{{<field>}} All description fields (uuid, folder, guest_os, etc.) from output of list (sm_cmd dir client –l 1 …) can be referenced.

When using placeholders in name generation, the placeholder can be adapted with one of the defined conversion functions, for example, proxmox_ {{toUpper client}}_{{source}}_{{count}}. Note that conversion functions cannot be combined and only one can be used per placeholder.

Configuration in GUI

You can automate the backup of new VMs in the SEP sesam GUI by adding tasks to a task group and including the task creation command in the pre-backup script of a task group. If configured, the script is executed before the backup job and automatically adds new tasks for VMs. For more information, see Using Pre and Post Scripts.

You can add all tasks for a backup target or other tasks with similar characteristics to a task group and trigger all tasks with a single event. For a step-by-step procedure, see Adding a Task to the Task Group.

Task group Beefalo V2.jpg

Information sign.png Note
  • A task group must always be linked to a schedule, to which a corresponding event is attached, if the tasks in the group should be backed up regularly. For details on creating the whole scheduled event setup, see Standard Backup Procedure.
  • If you already have a configured schedule, you can link your task group to it by clicking the Schedules tab when you create a task group or later in the task group properties window. Once you have configured a schedule, you have to create a new event for it.

After creating a task group, include the task generation command in the pre-backup script of a task group in one of the following ways:

  • By using the option Manage VM Tasks (right-click the client and select Manage VM Tasks). In this case, the task group, client and task type are automatically inserted. For details, see VMware example in Automating Backup of Virtual Machines.
  • Manage VM tasks window Beefalo V2.jpg

  • By using the option Customize Task List when creating a task group or in the task group properties, see Generating VM tasks.
  • Customize task list Jaglion.jpg

  • Alternatively, by creating a command event that runs the task generation command on a schedule. For details, see Creating a Command Event.

When a backup job is started for the group, the tasks on the drives will run simultaneously if possible. Other tasks are dynamically assigned to backup devices by SEP sesam. You can view the status of the entire task group by selecting Last backup state in the Main selection window. The backup status overview provides detailed information on the last run of backup jobs, including the task name, start and stop time of the last backup, backup level, data size, throughput, assigned media pool, etc.

SEP Tip.png Tip
You can also check the details of your backups online by using the Web UI. For details, see SEP sesam Web UI.

Workflow scenario and examples

Workflow scenario

The following is an example of auto-generation of tasks and clients.

  • For example, we have an administrator with superuser role configuring:
  • sm_cmd dir -F JSON client -l 1 vm_hypervisor -P on  <some more filter options> | sm_cmd add taskgen "gen_{{client}}_{{source}}_{{count}}" -O "source_not_found,remove_non_existing" -t "VMware vSphere" -G vm_taskgen_vmware -L Generated-Clients -n "_AUTO_" -w "_AUTO_" -f –
    
  • If the superuser wants to create the clients in different locations, he/she can filter the VMs with sm_cmd dir client and filter option, e.g., powerState=Running, and create different locations with sm_cmd taskgen.
  • The superuser does not want other users to be able to edit the clients and tasks that were auto-created earlier, but he/she does want some users to be able to access those clients and tasks to perform the backup and monitor progress. The superuser can configure ACLs for the location or client to allow access to a user with a backup role. For details, see Using Access Control Lists.
  • Backup user permissions Jaglion.jpg

  • The superuser can also configure a pre-interface for the task group to automate the generation of tasks and clients if the configuration and state of the VMs change. See the section Configuration in GUI for more information.
  • Once set, a user with the backup role can check generated clients and tasks for which permissions (ACLs) have been granted. A backup user can start a backup of the generated tasks, schedule the tasks but not modify them, and create new tasks to back up the client.
  • A user with the backup (or restore) role can also restore backups of generated tasks for which permissions (ACLs) have been assigned, but without changing the target settings; only the option Restore to original location is selectable.

Examples

vSphere task and clients

To auto-generate tasks and clients for vSphere VMs that do not have any other task associated with them in the same task group and simultaneously delete all tasks from the task group for VMs that no longer exist, do the following:

sm_cmd dir -F JSON vsphere vcenter6fix -P on  -i '.*swyxsrv.*' | sm_cmd add taskgen "vsphere_gen_{{client}}_{{source}}_{{count}}" -O "source_not_found,remove_non_existing" -t "VMware vSphere" -G test_vm_taskgen_vmware -T _IMG -V vss -a "refresh=true" -L Generated-Clients -n "_AUTO_" -w "_AUTO_" -f -
Hyper-V tasks

To auto-generate tasks for Hyper-V VMs that do not have any other task associated with them in the same task group (test_vm_taskgen_hyperv) and simultaneously delete all tasks from the task group for VMs that no longer exist, do the following:

sm_cmd dir -F JSON client -l 1 "hypervdev" "-i" "Windows.*" | sm_cmd add taskgen "hyperv_gen_{{client}}_{{source}}_{{count}}" -O "source_not_found,remove_non_existing" -t "Hyper-V" -G "test_vm_taskgen_hyperv" -f –
Citrix tasks

To auto-generate tasks for Citrix VMs that do not have any other task associated with them in the same task group (test_vm_taskgen_citrix), do the following:

sm_cmd dir -F JSON client -l 1 "xenserver62b.sep.de" -k "powerState=Running" | sm_cmd add taskgen "citrix_gen_{{client}}_{{source}}_{{count}}" -O "source_not_found" -t "Citrix XenServer" -G "test_vm_taskgen_citrix" -f –
Proxmox VE tasks

To auto-generate tasks for Proxmox VE VMs that do not have any other task associated with them in the same task group (test_vm_taskgen_proxmox), do the following:

sm_cmd dir -F JSON client -l 1 "pxmx-node1.srv.qa.hk.sep.de" -k "description=ct-on-node2" | sm_cmd add taskgen "proxmox_gen_{{client}}_{{source}}_{{count}}" -O "source_not_found" -t "Proxmox-VE" -G "test_vm_taskgen_proxmox" -f –
RHV tasks

To auto-generate tasks for RHV VMs that do not have any other task associated with them in the same task group (test_vm_taskgen_RHV), do the following:

sm_cmd dir -F JSON client -l 1 "rhev43-mgmt.srv.qa.hk.sep.de" | sm_cmd add taskgen "rhv_gen_{{client}}_{{source}}_{{count}}" -O "source_not_found" -t "RHEV" -G "test_vm_taskgen_rhv" -f –
OpenNebula tasks

To auto-generate tasks for OpenNebula VMs that do not have any other task associated with them in the same task group (test_vm_taskgen_opennebula), do the following:

sm_cmd dir -F JSON client -l 1 "one-hv.srv.dev.hk.sep.de" -k "description=QA-debian9" | sm_cmd add taskgen "nebula_gen_{{client}}_{{source}}_{{count}}" -O "source_not_found" -t "OpenNebula" -G "test_vm_taskgen_opennebula" -f -
MySQL tasks

To auto-generate tasks for MySQL databases in the same task group (test_mysql_gen), do the following:

sm_cmd -p 11409 dir -F json client "/qs2ws2008x64sql/MySQL:" | sm_cmd -A java -p 11409 add taskgen "mysql_gen_{{client}}_{{source}}_{{count}}" -t "MySQL" -G "test_mysql_gen" -f –