Debugging for Domino

Before proceeding, let me say that starting debugging should only be done per instructions from IBM. Also note that debugging can, by it's very nature, impact system performance. Although, I have not noticed any performance impacts when I have enabled debugging. This post will only deal with configuring the server for debug mode and not with the analysis of debug data. Many of these commands are also useful for the Notes Client. I will update this document from time to time. As I do so, I will maintain a change log at the bottom of this document. See also this Lotus Technote for information on debugging.

Let's begin with some of the DEBUG_* options by describing their usage along with some information on how to enable them.

1.) DEBUG_THREADID=1
Per Lotus Technote 1089976, "This prefixes the console output with the process and threadid information in the format [ProcessID:Virtual Thread ID-Native Thread ID]. This can be helpful in identifying the process or thread holding a semaphore."

2.) DEBUG_NTI_DRVR=1 I cannot find any information on this via technotes. My understanding is that this will enable debugging of the TCP/IP NTI Driver.

3.) DEBUG_TCP_ALL=1 This turns on debugging for all TCP functions.

4.) DEBUG_TCP_ERRORS=1 This turns on debugging for TCP errors.

5.) DEBUG_DBSERVER=1 This turns on debugging for the DB Server calls that may be made.

6.) DEBUG_SHOW_TIMEOUT=1 This turns on debugging for semaphores. See this Lotus Technote 1094630 for general information on semaphores.

7.) DEBUG_CAPTURE_TIMEOUT=10 This adds timestamps to the semaphore debugging.

8.) Debug_outfile=debug.txt This option specifies the location of the debug file. As a side note, I have noticed that after a couple of days on my iSeries that this file stops growing at 4GB. Lotus could not confirm this, but it appears that at 4GB this file becomes circular just like transaction logging. The following information is from the Lotus Administrator help database:

9.) DEBUG_HTTP_DBCLOSE=1 Anytime there is an http close of an NSF file it dumps to the console.

10.) DEBUG_TRACK_FREEHANDLE=1 I could not find anything specific to Domino debugging. From this DB2 page it "frees resources associated with a specific environment, connection, statement, or descriptor handle." Information on the freehandles in your Domino environment can be seen through the SHOW STAT DATABASE.FREEHANDLE*:
> Database.FreeHandleStack.FreeHandleStackHits = 3352913
> Database.FreeHandleStack.HandleAllocations = 3356284
> Database.FreeHandleStack.MissRate = 0
> 3 statistics found

11.) DEBUG_CHECKMARKERS=1 - This will debug memory checkmarkers and can be used with value #12 below to add this information to the NSD.

12.) NSD_RUN_MEMCHECK=1 - Per Lotus Technote 1086330, "Memcheck runs a scan of all shared memory segments in use by Notes processes, as well as the individual stack memory allocated to each process. Memcheck determines if the memory segment format matches the expected format for the memory segment. If Memcheck does not find what it expects in the memory segment, it dumps the appropriate memory segment pool into a file with the following syntax:

memcheck.__mm_dd@hh_mm.dump"

13.) DEBUG_NOTEOPEN=5 - This will log open activities for databases. In researching this, I found Lotus Technote 1086649 on how to use to track deletions in databases. The information logged will be in the following format:

Server Note Open:
Database Path: /local/notesdata/log.nsf,log.nsf
User: CN=Chris Whisonant/O=ORG
Title: tpdomino1's Log
#2StdNotesLog

From my understanding, these are all dynamic options (except for #8, 11, and 12) - meaning that you can type "set config DEBUG_THREADID=1" (or any of the others) at the console and begin seeing this output. If you have the debug outfile or console logging active (see below), then the will be reflected in those files as well. Please see this Lotus Technote for more information on setting this dynamically.

Syntax: Debug_Outfile=filename
Description: Specifies the file name for the Console Log file. If both this setting and the LogFile_Dir setting exist and Debug_Outfile contains a fully qualified path name, then LogFile_Dir is not used.
If only the Debug_Outfile setting exists and it contains only a file name, then the default path \DATADIRECTORY\IBM_TECHNICAL_SUPPORT is used. If neither Debug_Outfile or LogFile_Dir exist, then the default path is \DATADIRECTORY\IBM_TECHNICAL_SUPPORT and the default path is CONSOLE.LOG.

A side note on analysis, with options such as DEBUG_THREADID, you will be able to see a Process ID number along with Thread ID's. You can then, by possibly having agent print statements, be able to associate the Process/Thread ID with a particular agent, task, or user. This can potentially go a long way in helping to determine the cause of an issue.

Change Log:
05/16/2006: Added debug parameters, numbers 11 and 12 regarding technote 1086330.
Added debug parameter, number 13.
Shifted some information around and removed the section containing the debug parameters I had in my notes.ini for Domino 7.0 troubleshooting at one point.
Added link to my detailed Show-n-Tell post about how DEBUG_THREADID can assist in finding the problematic thread.