Source:Analyzing SEP sesam Log Files

From SEPsesam
Revision as of 08:36, 12 August 2020 by Sta (talk | contribs) (Fixed formatting.)
Other languages:

Copyright © SEP AG 1999-2024. All rights reserved.

Any form of reproduction of the contents or parts of this manual is allowed only with the express written permission from SEP AG. When compiling and designing user documentation SEP AG uses great diligence and attempts to deliver accurate and correct information. However, SEP AG cannot issue a guarantee for the contents of this manual.

Docs latest icon.png Welcome to the latest SEP sesam documentation version 4.4.3/4.4.3 Beefalo V2. For previous documentation version(s), check Documentation archive.


Overview

Analyzing SEP sesam log files is very useful for detecting operations that caused errors or malfunctions, for example, in case of a failed backup.

SEP sesam creates two protocols or log files for each backup day: the status file and the day log. An error log is the subset of the entire day log, where only error messages are recorded. Log files can be printed or sent by email. The default location (main directory) for the log files is SESAM_VAR/log. You can check backup logs (state, day or error) in the GUI (Main Selection -> Logging -> State/Day/Error Log).

As of 4.4.3 Beefalo V2, you can also check your system logs online by using new Web UI. For details, see SEP sesam Web UI.

Log files creation order during a backup

When a scheduled backup is performed, the log files are generated in a certain order. Note that when you are analyzing a problem and you see that the corresponding log files are missing from a certain point in the past, a cause of the problem is most likely positioned just before that point. An example of how a log file is generated is given below.

  1. The SEPuler creates the log file sm_sepul_event_xxx.log, for example sm_sepul_event_20181004.log.
  2. The queue manager writes the sm_qm_main_xxx.log, for example sm_qm_main_20180913.log.
  3. If the backup was able to start, a bck_*.log is created.
  4. When the backup starts, a backup .not log (notification) is created in the SEP sesam's SESAM_VAR/lis directory, e.g., smhg00_all- 20181004_001_SF20131004090011986@YlyxvqJCsHm.not.
  5. In case of optional media init, a sm_init_X_20180915.log is created. The X stands for the SEP sesam drive number.
  6. The information for monitoring drives and performance data is written in the sms log (sm_sms_watch_X_20181004.log).
  7. The files which are backed up are first written to a *.lis file (list of the backed up files and directories) and to sgm file (segment-file of used segment markers on the used tapes) on the device server in the SESAM_VAR/work/smslis directory. Once the backup is finished, these files are copied from the device server to the SEP sesam SESAM_VAR/lis directory. This data is needed for a selective restore.

Course of action for log file analysis

The recommended course of action depends on the failure. For example, if a scheduled backup did not run or have failed, proceed as follows.

Check the backup log in the GUI/Web UI

From Main Selection -> Backups, double-click the relevant Failed backup and open the Main Log tab to check the backup log.

As of 4.4.3 Beefalo V2, you can also check your failed backups online by using new Web UI. For details, see SEP sesam Web UI.

Check if a medium init error happened

If a medium init error occured (bck_*.log), it is the cause a failed backup.

Check if a backup does not have a log or have failed

If the backup does not have a log yet or have failed:

  1. Check the status and daily logs in the SESAM_VAR/prot directory for events and errors at that particular time.
  2. Check if a .not log exists in the SESAM_VAR/lis directory. If not, then the possible causes for the error are as follows:
    • Client is not accessible (DNS, ping).
    • There is no media available.
    • The backup did not start yet.
  3. The process logs are stored in the SESAM_VAR/log/lgc directory. The logs should be listed chronologically in the terminal, e.g., Linux ls -lart).
    • The bck_*.log is created by the program sm_backup; name convention: bck_<job_name>_<save_set_ID>_<sesam_day>.log. Note that unlike most other logs, the bck_*.log must be read from the beginning to find the first error message that may reveal the cause of the failure.
      • Check the license.
      • Set time range: Check if the backup is within the starting time frame.
      • Alive test: Check if the client is active and reachable.
      • CHECK_MEDIUM: Check the availability of the media.
        • iGET_PREPARED_MEDIA: Check the media pool. If msg=0 appears, there is no media available.
        • GET_BACKUP_MEDIUM: The sm_sms_interface is doing something on the tape (getlabel, init, etc.). For tasks started simultaneously, search for the largest backup file which contains the log files of the executed media init.
        • In case of tape media, the following Options may be set as described in Configuring a Media Pool. They help you control which media will be used for a backup. For example, for devices which load media in sequential order, or if you do not want an unattended backup to fail because the specified media are not available, you may choose to use empty media policy.
          • empty: If this option is selected and there is no EOL-free media available in the requested media pool, SEP sesam uses any suitable media for backup – empty media and media that are unrecognized by SEP sesam.
          • spare: If this option is selected and there is no EOL-free media available in the requested media pool, then media from the SPARE_ pool are used for backup.
          • other: If this option is selected and there is no EOL-free media available in the requested media pool, then any EOL-free media from other media pools are used.
        • Note: The EOL defines how long the backed up data on media remains protected after the data is written to the medium (see Managing EOL). When the protection expires, SEP sesam can re-use the media for backups again. You can check the life cycle of a tape in the daily log; for details on media initialization, checked the *.sms log files in the <SESAM_VAR>/log/sms directory.
      • sm_notify is delivered.
      • With the search pattern "Cmd= sbc" you jump directly to the command given in the log which calls up the backup.
      • If the backup has been started successfully, then the .not log is created in the SESAM_VAR/lis directory.
      • During an active backup all log information are appended to .not log.
    • If there are any problems with the media init, the errors will be recorded in the sm_init_<drive>_<sesam_day>.log. If a log includes error: all media with eol restriction, then no media in the requested pool is available. There may be further attempts to get a backup media according to the media pool options you have specified (see above Options.
    • If there are any problems with the subsequent tapes after writing backup data until the End Of Media (EOM) is reached, they will be written in the sm_sms_watch_<drive>_<sesam_day>.log. If this log includes error: all media with eol restriction, then no media in the requested pool is available.
    • To check the communication between the client and the SEP sesam Server either over SMSSH (default, via port 11322) or over CTRL (via port 11301), look for sm_sshd_<sesam_day>.log or sm_ctrld_<PID>.lgc in the SESAM_VAR>/log/lgc directory on the client. The logs on the client are generated when the SEP sesam Server performs a task on the client, for example:
      • Execution of a backup.
      • Execution of a command.
      • Browsing the GUI file wizard (when creating a task).
    • Whenever the SEPuler finds a task that has to be executed, the operation is recorded in (sm_sepul_event_<schedule_identifier>_<sesam_day>.log). With QUE_SUBMIT a job is put into a queue; sm_backup shows that the backup is being put into the queue.
    • On a SEP sesam Server, search the log sm_sms_watch_<drive>_<sesam_day>.log from bottom upwards to check the drive information. This includes:
      • Monitoring of a drive. This log is generated only on a SEP sesam Server (also for the drives on RDS, the corresponding watch logs are located on the SEP sesam Server).
      • Data throughput of a drive.
      • The search string "‘+++ EOM"‘ shows media changes related to end of media (tape is full). The sm_sms_interface init command:
        • STATUS=SUCCESS: Successful media initialization.
        • STATUS=IO-ERROR: There is a problem with media or a drive. If necessary, check the SESAM_VAR>/log/messages on Linux or the event log on Windows for any hardware problems. To confirm that there is a hardware problem, check the sms log in the <SESAM_VAR>/log/sms directory.
    • The log sm_sms_watch_0_<sesam_day>.log regularly displays the process status of the sm_main processes and the processes in the sm_qm_main queues, in addition to regular check of available space GET_FREE_SPACE_OF_DIR.