Archive:SEP sesam extension for Lotus Domino Server

Introduction

The SEP sesam Client SBCLN is a program library for backups of Lotus Notes/Domino databases. To backup Lotus Notes Server, transaction logging has to be activated in Archived Mode on the Lotus Notes Server. Transaction logging is part of the Lotus Notes system. Notes databases can also be backed up without transaction logging although the databases are always saved as full backups this way.

Another important point in planning a backup strategy is considering a valid DBIID (Database Instance ID). Transactions only refer to information with an identical DBIID of a database. If tools, such as fixup or compact are used on databases, the database usually gets a new DBIID and all transaction information that was previously created for it is then turned invalid. We therefore strongly recommend to run a full backup of the database immediately after using these Notes tools. Neither planning nor operating these Lotus Notes tools is part of SEP sesam.

Additional information on the topics transaction logging and DBIID can be found in the IBM's Lotus Notes documentation, e.g., Transaction Logging on Domino Servers and Notes/Domino Best Practices: Transaction Logging (more references are provided in section External references).

The backup API of Lotus Notes does not offer any direct backup or restore options for single emails or documents. The API is designed more like a relational database and set up for disaster recovery. A single mail recovery is possible but requires profound knowledge of Notes administration. The section Single mail restore briefly describes the practical procedure for a single mail restore.

Prerequisites

• Check the support matrix for the list of supported versions.

	Note
	Windows 2012 is supported with its regular file system functions. There is no support if the Notes data were held in a NTFS deduplication partition.

• To use the Lotus Domino extension, download SEP sesam Server and SEP sesam Client. It is highly recommended to use the latest SEP sesam version.
• Verify that your system meets the requirements.

Configuration

Linux

• A fully installed SEP sesam Linux client that can be backed up from the SEP sesam Server via path backup. For details on creating a backup, see Standard Backup Procedure.
• Copy the content of the SEP sesam Extension for Lotus Notes into the directory <SESAM_ROOT>/bin/sesam/.
• To work with Notes and to load the library LOTUS, NOTES_DATA_DIR, Notes_ExecDirectory and PATH have to be set according to the Notes environment and the LD_LIBRARY_PATH has to direct to the libraries directories libsbcln.so (located in <SESAM_BIN>/bin/sesam) and libnotes.so (in the binary directory of the Notes installation).

The [ENVIRONMENT] section has to be added in <SESAM ROOT>/var/ini/sm.ini at the client. The values listed below depend on the Notes installation. Please use values that apply to your Notes environment.

 [ENVIRONMENT]
 ...
 LOTUS=/opt/ibm/lotus
 NOTES_DATA_DIR=/srv/notesdata
 Notes_ExecDirectory=$LOTUS/notes/latest/linux
 Notes_Resources=$Notes_ExecDirectory/res
 SESAM_LIB_PATH=/opt/sesam/bin/sesam
 NOTES_LIB_PATH=$NOTES_DATA_DIR:$Notes_ExecDirectory:$Notes_Resources
 PATH=$PATH:$NOTES_DATA_DIR:$Notes_ExecDirectory:$Notes_Resources
 LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SESAM_LIB_PATH:$NOTES_LIB_PATH
 ...

If there is no [ENVIRONMENT] section in sm.ini, it has to be created.

Afterwards, the SEP sesam daemon has to be restarted.

 /etc/init.d/sesam restart

Windows

• A fully installed SEP sesam Windows Client that can be backed up from the SEP sesam Server via path backup. For details on creating a backup, see Standard Backup Procedure.
• The Lotus Notes Client module SBCLN.DLL is already included in the regular SEP sesam Client for Windows. However, to use it a separate license for the online extension of Lotus Notes is required.
• Ensure that the NNOTES.DLL file is not located in the <SESAM_BIN>\bin\sesam directory or the Windows System32 directory!
• The system variable Path must include the path of the library NNOTES.DLL. If this path is not included, the variable has to be set accordingly under environment – system variables, e.g.:

 PATH=...;C:\Program Files\Lotus\Domino

Then the SEP sesam service has to be stopped and restarted:

net stop sm_main
net start sm_main

For details on adding the path, see the IBM article the path to nnotes.dll in the environment path variable.

Backup

All further steps of the setup are performed using the SEP sesam GUI. When performing a Lotus Notes online backup, files with Notes-relevant filename extensions (e.g., ".nsf", ".ntf", ".dic", ".box", ".dsk", ".id", etc.) are backed up over the Notes API at the operating time of the Notes Server.

Note

notes.ini is not directly part of the Lotus Notes data files but it contains information and parameters about the Notes installation and configuration. SEP sesam online Notes backup tries to detect the location of notes.ini, first in the binary path of the Notes installation, then in the Notes data root. Once notes.ini is detected, it is backed up automatically with the Notes files. Otherwise a warning is issued in the backup protocol. In this case the administrator has to make sure that notes.ini is backed up with a path backup.

How does a Notes online backup work?

If a backup task with backup type Lotus Notes, backup type Full and backup source all is started, SEP sesam client connects with the Notes API and requests a backup of all files relevant to Lotus Notes. Notes takes the first database, copies it to the temporary SEP sesam path (gv_rw_tmp in PATHES section in the sm.ini) to the file system and then checks if any transactions are still in queue. If they are, first the transactions are executed and then the database is backed up by SEP sesam client. This way all Lotus Notes databases are transferred from Notes to the file system and backed up sequentially by SEP sesam client. To enable successful backup, make sure that the temporary storage space is at least as big as the largest Notes database.

	Note
	If your Lotus Domino databases use DAOS, check backup considerations for DAOS in section DAOS backup.

When restoring over the Notes API, the database that is being restored travels the same way back. The database file is initially put into the temporary SEP sesam space in the file system by SEP sesam and then transferred to the Notes API.

Backup type

FULL

The Notes databases that are selected as a backup source are backed up completely. The file source all means that all files in the Notes data root, which are identified as Notes files over the Notes API are backed up. Full backup includes database files and all backup-relevant transaction logs (.txn files).

DIFF

DIFF saves all new and changed databases since the last FULL backup. It does not backup the transaction logs, only the database files!

INC

When executing an incremental backup, only backup-relevant transaction logs are saved. Only all can be specified as a backup source for incremental backups.

COPY

Same as full backup without consideration of the transaction logs.

Note

You have to run a full database backup after the failure of an incremental job or if an incremental backup is corrupted or deleted, because this will break your backup chain. You need to start a new chain by performing a full backup. It is recommended to immediately perform a full database backup whenever a database receives a new DBIID to avoid data loss. If the DBIID changes and no current backup exists, you will not be able to successfully restore databases.

Entering the backup source

All backup tasks for Notes are set as Lotus Notes type in the GUI (except for DAOS backup, see backup considerations for DAOS in section DAOS backup). The following patterns can be entered as a backup source:

Full backup

The placeholder all has to be entered in the field Source to backup all Notes relevant files of the Notes Server (supported backup types: FULL, INC, COPY).

Selective backup

Using the file wizard the backup source can be selected. As an example, we will take the Notes file directory E:\notesdata for Windows and /srv/notesdata for Linux.

• Backup with path statement with one or two directories. Here all Notes-relevant files are backed up with their subdirectories. Lotus Notes backup in Linux is different from the normal path backup of Linux clients, as it also follows the directories beyond file system boundaries and symbolic links.
• Examples Windows:
a) E:/notesdata/mail b) E:/notesdata/mail1,E:/notedata/help
• Examples Linux:
a) /srv/notesdata/mail b) /srv/notesdata/mail1,/srv/notesdata/help
• Backup with file pattern:
  e.g., all databases from directory mail in Windows:
E:/notesdata/mail/*.nsf or all templates from the directory help in Linux: /srv/notesdata/help/*.ntf
• Backup of single files (1st line in Linux / 2nd line in Windows):
1. /srv/notesdata/mail/user1.nsf 2. E:/notesdata/mail/user1.nsf

Using Exclude list

Optionally, specify the filenames or patterns which you do not want to back up in the Exclude list. For details on various exclude options, see Creating Exclude List.

	Note
	In SEP sesam versions < 4.4.3.45, the exclude list, defined on Linux/UNIX in a Lotus Notes backup task, does not work if specified with \.. For versions < 4.4.3.45, specify the exclude list for Notes without the preceding backslash and a dot, e.g., use /srv/notesdata/help$ instead of \./srv/notesdata/help$'.

Options for Lotus Notes in Linux

By default the values for the user that runs the Notes service in Linux and the path of notes.ini server can be stored in the section [ENVIRONMENT] in sm.ini on the SEP sesam Client. If the Notes user is not notes or differs from the standard value, it can be entered in the task properties under the tab Options in backup- and restore options with

 -a USER={ihr-notes-user}

Likewise, if notes.ini is not located in /local/notesdata, its location must be specified in the backup- and restore options with

 -a NOTESINI={complete path to the notes.ini file}

Example for backup and restore options:

 -a USER=nadmin -a NOTESINI=/srv/notesdata/notes.ini

This information has to be entered for every backup task that is designated to backup files on the respective Notes Server with the backup type Lotus Notes.

DAOS backup

DAOS (Domino attachment and object service) was introduced in Lotus Domino v. 8.5.0. It allows the mail attachments to be stored outside the NSF file in Notes Large Object (.NLO) files. By transferring attachments to a file system in your server and allowing to preserve a single copy rather than multiple copies of the same attachment, it preserves disk space. This means that the NFS database is no longer self-contained, therefore an additional backup of the .NLO data is required. For more details on DAOS, see IBM documentation, e.g., IBM Lotus Domino Attachment and Object Service (DAOS) deployment guide and The new Lotus Domino attachment and object service.

Requirements

• For databases on a Lotus Domino Server that use DAOS, this must be activated and the consolidation must be enabled on the DAOS tab in the Server document.
• Additionally, every Notes database must have the Use Domino Attachment and Object Service advanced database property selected.

Backup considerations

SEP sesam automatically backs up DAOS configuration files daos.cfg and daoscat.nsf by regular Lotus Notes backup type. However, you should be aware that IBM strongly advises against backing up the daos.cfg and daoscat.nsf files as restoring old copies of these index files that would not match the existing NSF and NLO environment can cause discrepancies and lead to data loss. The preferred method is to re-create these two files from the DAOS repository and the NSFs participating in DAOS if required. For more details, see the IBM article DAOS Backup and Restore.

A separate backup is required to back up the NLO data. Consider the following for the .NLO backup:

• If the DAOS path is set up under the Notes data path (e.g., E:\notesdata\DAOS), all DAOS files (.NLO) are backed up together with other Notes files.
• If the DAOS repository is configured outside of Domino's data visibility, you have to create an additional path backup and select the DAOS data location as a backup source. For details on creating a path backup, see Standard Backup Procedure.

	Note
	It is recommended that you set up a SEP sesam path backup to backup .NLO files in your DAOS repository regularly, e.g., together with FULL/DIFF/INCR Lotus Notes task. You should back up all the NSF data before backing up the .NLO files to ensure that the referenced NLO files exist.

For details on backup recommendations, see IBM articles Backing up servers that participate in attachment consolidation (DAOS) and DAOS Backup and Restore.

Domino partitioned server backup

A Domino partitioned server is a Notes Server where several Notes instances are running. All running instances use the same (shared) program files, but every instance has its own data path and notes.ini. From the network view every Domino instance can run on a separate IP address; when they run on the same IP address they can use different ports, depending of the Domino configuration. For further information on the installation and configuration of a Lotus Domino partitioned server, see the relevant IBM documentation.

To back up a Domino partitioned server environment, you have to create a backup task for each partition. SEP sesam backup references the notes.ini of the corresponding partition directly. For each partition-related backup task you have to specify the following:

• Task Name: The name of the task, for example notus05_notes_DataA.
• Backup type: Lotus Notes
• Client: The name of the client (not the partition) where the Domino program is running, for example notus05.
• Source: all
• Switch to the tab Options. In the Save options field, enter the corresponding notes.ini, e.g.:

-a NOTESINI="D:\Lotus Domino\Data A\notes.ini"

For every partition you want to backup, you have to specify the client name where the Domino program is running. Even though every Domino partition has its own IP address and DNS name, the backup job will reference the native program host. For instance, in the example above the Domino Server itself has the DNS name notus05 and the partitions have the names notus05a and notus05b, but you always reference the notus05.

Restore

A restore of Lotus Notes databases begins with starting the restore wizard in the SEP sesam GUI. First, select the backup task from which you want to restore the database and confirm with Next. Then browse through the backup tree to select the file and click Next again.

The restore of Lotus Notes databases can be done in two different ways.

> To the original location: The database is restored over Notes API to its original location.

> To a different location: The database is restored outside the visibility of the Notes Server into the file system to be processed further there.

To the original location

The restore over the Notes API can be executed selectively (single or several databases) or fully (typically required only for disaster recovery).

The following Execution Options are available:

First field:

do not overwrite existing files

The database chosen for the restore must not exist in the Notes file directory or the restore will fail.

overwrite existing files

If the original database still exists in the Notes system, overwrite existing files has to be selected to substitute the existing database. Deleting the original and substituting it with the database from the save set is done completely from the Notes API. The restored file can be accessed immediately.

Second field:

No recover after restore

If there are any existing transaction logs, they are not applied to the database after the restore, i.e. the databases are just stored at the target system.

Auto recover and online after restore

Applies any existing transaction logs to every restored database.

With the additional restore option -a logs only the transaction logs from the save set are stored on the client! This option can be entered under the Expert Options in the restore wizard.

To a different location

When restoring a database (e.g., user1.nsf), Lotus Notes automatically detects if it is going to be restored into a path inside the Notes data path. If a different path (e.g., e:/temp/restore) is used (the restore path can be chosen via the file wizard by clicking the Browser button), the restore is performed as a path restore without transfer to the Notes API. For example, the database is stored in e:/temp/restore/<notespath>/user1.nsf.

From this point on the admin has to copy the database file to the target location in the Notes file path manually according to the requirements of the file restoration.

Point-in-time recovery

The Lotus Notes API offers point-in-time recovery thus enabling to recover a database to a specific point in time – back to the state it was in at some point in time between the last full backup and now. This is done by the program dbrecover which is already included in the SEP sesam Client (dbrecover, originally from the Notes backup SDK, was slightly modified by SEP). For details on dbrecover options, see section dbrecover.

The following prerequisites must be met for the point-in-time recovery:

1. A successful full backup of the database.
2. Lotus Notes must be set to use archived transaction logging.
3. All transaction logs (.txn files) since the last full backup up until the time of recovery must be available.

To perform point-in-time recovery of Lotus Notes databases, proceed as follows:

1. Restore the database at a different location (e.g., C:\temp). Choose the values Do not overwrite and No recover after restore under the Execution options.
2. Make sure that all required transaction logs exist in the LogDir of the Lotus Notes Server (if necessary, through separate restores) because dbrecover gets the required recover information over the Notes backup API.
3. Start the program dbrecover via the command line. For example, to restore the databaserbogner.nsf from 11/02/2017 04:00 am, the following command has to be executed from the directory <SESAM_BIN>\bin\sesam:

 dbrecover recover c:\temp\rbogner.nsf "11/02/17 04:00 am"

After that the following output appears in the console:

 $Id: dbrecover.c,v 1.13 2014/04/16 17:01:09 miraculix Exp $
 
  Recovering backup file ...
 
 Media Recovery Replay (60 MB): 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
 11.02.2017 10:08:52   Recovery Manager: Media Recovery complete for c:\temp\rbogner.nsf, last update applied 11.02.2017 03:48:23.
 
  Backup file c:\temp\rbogner.nsf recovered.
 
 Program completed successfully.

Then the database has to be copied into the file space of Lotus Notes.

Single mail restore

	Note
	A direct restore of single mails or documents from Lotus Notes databases is not possible because of the special functionality of the IBM Notes backup API. Single emails can only be restored from backups with the profound knowledge of the Lotus Notes administrator.

The following simplified scenario shows how the Lotus Notes administrator can restore single documents/mails.

Today, the user of the mail database user1.nsf has accidentally deleted a few days old email. This email must exist in the last night’s backup. Lotus Notes cannot restore an existing database into a different directory or with a different name on the same server directly over the Notes backup API, because Notes will notice that a database with the same replica-ID already exists.

The only solution is to restore the database into the file system, create a copy of this database and then copy the required email from it to the restore destination.

The administrator will follow the steps below to restore the single email:

1. Restore the database user1.nsf on the Notes Server into the file system (not in Notes data root!).
2. Open user1.nsf locally with the Notes Client or the Notes Administrator and create a copy with the name user1a.nsf on the Notes Server. Creating this new copy assigns a different replica-ID to the database user1a.nsf than the original user1.nsf has. Therefore Lotus Notes sees them as two completely different databases.
3. Given the necessary access rights both databases can now be opened and the required email document can be copied from user1a.nsf to user1.nsf.
4. After restoring the email document, delete the database user1a.nsf.

Disaster recovery

Prerequisites

To prepare a Lotus Notes Server for a possible disaster recovery, the following backups are required:

• Operating system
  • Windows
    • system partition with VSS
    • backup of system data (system_state)
    • backup of the Notes application (directory with the Notes binaries)
  • Linux
    • complete Linux system without Notes data (all-Backup)
    • backup of partition data (disk_info)
• Lotus Notes online backup of all Notes data with the source all

Execution

1. In case of a disk crash, first the OS and the Lotus Notes program files are restored. Do not start the Notes service at this point!
2. Next, the Lotus Notes data is restored as a path backup to the original Notes data path – with overwrite. Hereby all databases, transaction logs, ID-files and the notes.ini are copied to the server.
3. Before starting, the file(s) nlogctrl.* has to be deleted in the log directory.
4. After the successful restore of all Notes data, the Notes Server service can be started. During startup, the Notes Server automatically checks the consistency of the databases and recovers their state. The Notes recovery procedure may last some time. This depends on the number of the restored databases and their volume.

Lotus Domino dbbackup and dbrecover tools

SEP provides the Lotus Domino SDK tools dbbackup tool and dbrecover with some adaptations.

dbbackup

The dbbackup tool allows backing up database by storing the database copy into a file.

The dbbackup arguments are displayed with the dbbackup usage:

$Id: dbbackup.c,v 1.11 2013/11/13 10:44:35 miraculix Exp $
Usage:  dbbackup  <database filename> <output filename>
Hint:   To specify an .ini file, add an argument preceded by the equal sign ('='), e.g., '=../notes.ini'

dbrecover

The dbrecover tool allows to archive Lotus Domino logs and to perform database recovery and restore. It also allows point-in-time recovery, as described above in the section Point-in-time recovery.

	Note
	Do not use dbrecover ARCHIVE together with SEP sesam INCR backups because the archived logs may not be backed up with SEP sesam if already backed up into files. Therefore the files may be missing during SEP sesam Lotus Domino database recovery.

The dbrecover arguments are displayed with the dbrecover usage:

#>dbrecover
$Id: dbrecover.c,v 1.13 2014/04/16 17:01:09 miraculix Exp $

Usage: dbrecover <option> [input file] [restore db|datetime]

option: ARCHIVE - Archive system logs.   (Input file NOT REQUIRED)
        CHECK   - Check a database or backup file.    (Input file REQUIRED)
        RECOVER - Recover a backup file. (Input file REQUIRED, DateTime optional)
        RECOVER_ALT - Recover with archive log extends in TRANSLOG_RECOVER_PATH. (Input file REQUIRED)
        NOTE    - Recover a backup file and get Note info. (Input file REQUIRED)
        NOTE_ALT    - Recover get Note info with log extends in TRANSLOG_RECOVER_PATH (Input file REQUIRED)
        RESTORE - Restore a database.    (Input file & restore db REQUIRED)

input file: - Path to a backup file.   (If REQUIRED)

restore db: - Path to the database to be restored.   (If REQUIRED)

datetime  : - Date and time in system format, e.g., '01/01/96 12:01 AM' (optional)

Hint:   To specify an .ini file, add an argument preceded by the equal sign ('='), e.g., '=../notes.ini'

Option RECOVER

If the recover option is selected, the DBIID of the recovered file is refreshed, allowing access to the original DB and to the recovered version. This option creates a new instance of the original database, therefore a separate backup is required for this new instance in order to enable it to be also recovered at the time of need.

Option RESTORE

If the restore option is selected, the original DB's DBIID is not refreshed and a new instance is not created. This option deletes the original DB when it is taken offline and replaces it with the recovered backup file.

Troubleshooting

For troubleshooting Lotus Domino Server issues, see Troubleshooting Lotus Domino Server.

External references

• SEP sesam Extension for Lotus Domino Server
• IBM Notes Backup & Recovery API
• Transaction Logging on Domino Servers
• Tranactional Logging and DBIID
• IBM Lotus Domino going green: The new Lotus Domino attachment and object service http://www.ibm.com/developerworks/lotus/library/domino-green/
• DAOS Backup and Restore http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daos-backup-and-restore