Installed Fixpack 1 for Sametime 6.5.1

Just installed FP1 for Sametime 6.5.1 on iSeries. After having uploaded the 500MB PTF save file, I just had to run 4 commands:

1) End IM server
2) Load PTF
3) Apply PTF
4) Start IM server

Want to know how long it took me?

1) 7:31 - 7:35 PM
2) 7:36 - 7:37 PM
3) 7:38 - 7:41 PM
4) Server back online at 7:44 PM

Not bad 10 minute total downtime to load a big fixpack. Has anyone else upgraded?

Debugging for Domino

Before proceeding, let me say that starting debugging should only be done per instructions from IBM. Also note that debugging can, by it's very nature, impact system performance. Although, I have not noticed any performance impacts when I have enabled debugging. This post will only deal with configuring the server for debug mode and not with the analysis of debug data. Many of these commands are also useful for the Notes Client. I will update this document from time to time. As I do so, I will maintain a change log at the bottom of this document. See also this Lotus Technote for information on debugging.

Let's begin with some of the DEBUG_* options by describing their usage along with some information on how to enable them.

1.) DEBUG_THREADID=1
Per Lotus Technote 1089976, "This prefixes the console output with the process and threadid information in the format [ProcessID:Virtual Thread ID-Native Thread ID]. This can be helpful in identifying the process or thread holding a semaphore."

2.) DEBUG_NTI_DRVR=1 I cannot find any information on this via technotes. My understanding is that this will enable debugging of the TCP/IP NTI Driver.

3.) DEBUG_TCP_ALL=1 This turns on debugging for all TCP functions.

4.) DEBUG_TCP_ERRORS=1 This turns on debugging for TCP errors.

5.) DEBUG_DBSERVER=1 This turns on debugging for the DB Server calls that may be made.

6.) DEBUG_SHOW_TIMEOUT=1 This turns on debugging for semaphores. See this Lotus Technote 1094630 for general information on semaphores.

7.) DEBUG_CAPTURE_TIMEOUT=10 This adds timestamps to the semaphore debugging.

8.) Debug_outfile=debug.txt This option specifies the location of the debug file. As a side note, I have noticed that after a couple of days on my iSeries that this file stops growing at 4GB. Lotus could not confirm this, but it appears that at 4GB this file becomes circular just like transaction logging. The following information is from the Lotus Administrator help database:

9.) DEBUG_HTTP_DBCLOSE=1 Anytime there is an http close of an NSF file it dumps to the console.

10.) DEBUG_TRACK_FREEHANDLE=1 I could not find anything specific to Domino debugging. From this DB2 page it "frees resources associated with a specific environment, connection, statement, or descriptor handle." Information on the freehandles in your Domino environment can be seen through the SHOW STAT DATABASE.FREEHANDLE*:
> Database.FreeHandleStack.FreeHandleStackHits = 3352913
> Database.FreeHandleStack.HandleAllocations = 3356284
> Database.FreeHandleStack.MissRate = 0
> 3 statistics found

11.) DEBUG_CHECKMARKERS=1 - This will debug memory checkmarkers and can be used with value #12 below to add this information to the NSD.

12.) NSD_RUN_MEMCHECK=1 - Per Lotus Technote 1086330, "Memcheck runs a scan of all shared memory segments in use by Notes processes, as well as the individual stack memory allocated to each process. Memcheck determines if the memory segment format matches the expected format for the memory segment. If Memcheck does not find what it expects in the memory segment, it dumps the appropriate memory segment pool into a file with the following syntax:

memcheck.__mm_dd@hh_mm.dump"

13.) DEBUG_NOTEOPEN=5 - This will log open activities for databases. In researching this, I found Lotus Technote 1086649 on how to use to track deletions in databases. The information logged will be in the following format:

Server Note Open:
Database Path: /local/notesdata/log.nsf,log.nsf
User: CN=Chris Whisonant/O=ORG
Title: tpdomino1's Log
#2StdNotesLog

From my understanding, these are all dynamic options (except for #8, 11, and 12) - meaning that you can type "set config DEBUG_THREADID=1" (or any of the others) at the console and begin seeing this output. If you have the debug outfile or console logging active (see below), then the will be reflected in those files as well. Please see this Lotus Technote for more information on setting this dynamically.

Syntax: Debug_Outfile=filename
Description: Specifies the file name for the Console Log file. If both this setting and the LogFile_Dir setting exist and Debug_Outfile contains a fully qualified path name, then LogFile_Dir is not used.
If only the Debug_Outfile setting exists and it contains only a file name, then the default path \DATADIRECTORY\IBM_TECHNICAL_SUPPORT is used. If neither Debug_Outfile or LogFile_Dir exist, then the default path is \DATADIRECTORY\IBM_TECHNICAL_SUPPORT and the default path is CONSOLE.LOG.

A side note on analysis, with options such as DEBUG_THREADID, you will be able to see a Process ID number along with Thread ID's. You can then, by possibly having agent print statements, be able to associate the Process/Thread ID with a particular agent, task, or user. This can potentially go a long way in helping to determine the cause of an issue.

Change Log:
05/16/2006: Added debug parameters, numbers 11 and 12 regarding technote 1086330.
Added debug parameter, number 13.
Shifted some information around and removed the section containing the debug parameters I had in my notes.ini for Domino 7.0 troubleshooting at one point.
Added link to my detailed Show-n-Tell post about how DEBUG_THREADID can assist in finding the problematic thread.

Installing Fixpack 1 for Sametime 6.5.1

For all you waiting with baited breath for this to happen, I was just informed by IBM that I need to install the Sametime 6.5.1 Fixpack 1 to try and fix my problems I've been having with Sametime 6.5.1 on iSeries.

I plan on installing it between now and Monday sometime. The installer save file is 500MB (actually only 50MB less than the original 6.5.1 installer...). Will post with my results!

Installed: Lotus Domino 6.5.4 Fix Pack 1

This past Friday night I installed FP1 on my production iSeries. I had loaded it last Wednesday on my cluster iSeries. With both cases, the install was a breeze. It was just a matter of getting the file to the box (FTP) and then using the LODPTF and APYPTF commmands after ending my servers.

The servers came up without a problem and have been OK since Friday. Once I've ran it for about 2 weeks without having a problem I will be apt to think that it has corrected my stability issues with 6.5.4 that I have previously blogged about.

If this doesn't appear to correct the problem I will likely be applying the Sametime 6.5.1 Fixpack 1 since the issues possibly appear to be with integration between Notes Integrated IM and Sametime server.

Foot In Mouth?

Just finished loading 6.5.4 FP1 and came across this news article about Howard Dean before I was getting off my system.

Dean states:

"I think that we must be absolutely firm in being the party of individual freedom and personal freedom, which means that in the end the government doesn't get to decide, we do."

I haven't seen him crying out against the "individual freedom and personal freedom" that the left has supported being taken from us via the Supreme Court's case on Eminent Domain. That case means that in the end the government does get to decide, we don't!

NSD Troubleshooting

This is another one from my internal FAQ. I've been dealing with some tough NSD issues with Lotus Notes crashes. The following suggestions are from Lotus support. One or all of them may need to be done and it still may not correct the problem.

- Delete the temporary folder c.lotus.notes.data in Windows Temporary drive
- Rename cache.ndk, bookmark.nsf, and/or desktop6.ndk
- Try to narrow it down to what database is causing the Client to crash. Work in one database at a time and see what database might be causing the crash.
- Is the user sending encrypted mail to anyone at the time of the crash?
- Please bring another Users ID file to this users machine and do the same actions the user did to cause the crash, and see if the crash still occurs.
- Create a local replica of the database(s) suspected to be causing the problem
- Reinstall Notes
- Get the default ID file from the server
- Run nightly utilities on the database(s) - fixup, updall, compact
- Is there one customized database that is in common that crashes for both users or are these 6 different databases?
- Do other users have access to this database, and do other users also click on the "Match invoice" button in these databases? Do those users workstations crash as well?
- What happens if you bring one of these two user ID's to your own local workstation, and try to access these customized databases on your OWN machine. Will it cause your client to crash? If so please go ahead and send me the NSD's.
- Please give yourself access to all 6 (?) of these customized databases and click on the same button that forces the Client to crash. Does Notes crash when using your own ID?
- Check to see if Sign mail and Encrypt mail is checked off for this user.
- Remove printer/scanner drivers from the system

Lotus Domino 6.5.4 Fix Pack 1 Released

Here's a link to the downloads for the various server platforms:

Lotus Domino 6.5.4 Fix Pack 1 Installers

With all the iSeries connection issues I've been having, I'm going to put this fix on my cluster box to make sure things seem to work alright...

The Windows installer is 13MB and the iSeries installer is 142MB - what a difference!

5 Words Developers Like To Hear

Paul and I are finishing up a project to replace a database application (Access backend) being used by our HR department. Paul has done a great job with this and I'm just lending a helping hand as needed and will probably be doing the data import next week.

Anyway, we were demoing it to some HR folks today and one of the ladies said:

"That is cool as crap!"

Always good to hear those words!!

Domino 6.5.4 Fixpack 1

I was just informed by Lotus Support today that FP1 of Domino 6.5.4 should be released early next week. Hopefully this will be correcting some of my issues.

6.5.4 FP1 Fix List Status

SPRs fixed

Lotus Notes and Domino 7.0 Beta 4

I have a test mule PC (600 MHz PIII) running RedHat Linux 9 (Shrike) and Domino 6.5.4. I downloaded the 7.0 Beta 4 today and attempted a straight upgrade (didn't read instructions...). It went fairly well, and I noticed that the installer now has a text-based "status bar" showing the completion percent - which was a pretty handy addition.

I was told by the installer that Domino does not support Shrike. Guess I'll have to peruse the readme and see what distro I need. Hopefully it's at LinuxISO.org.

As a side note, I had G Simmons (not gene!) upgrade from Notes 6.5.4 to 7.0 Beta 4. He instantly noticed a performance improvement in navigating his 6.5.4 mail file. Hopefully we'll see others giving the same feedback. I'm considering SmartUpgrading our IT department although I can't personally upgrade because of Development. I'm curious to see if I get any more positive feedback. Just today our PC Support manager was asking me if there had been improvements made to the DWA client in 6.5.4. I told him that there were many improvements and he said that he could definitely tell when accessing via his cable modem.

Scalability and ROI is what we've been seeing with the latest releases of Notes and Domino. There are supposed to be some performance enhancements with the 7.0 Domino iSeries code as well. I love a product that will allow you to leverage current hardware and software investments. I won't mention Exchange upgrades (they call it Exchange because you exchange out your current server and OS platforms for a newer investment...)

Just Hanging Out

OK, so it's Sunday morning, 9:30, and my team has been at work since 5. We're doing a system save on our production iSeries, loading fixes, adding a controller card for shared tape drive with our Domino iSeries (the NEXT save should be really fast!), and installing about 175 GB more disk storage.

Over the past few months Chris Moody has been migrating data from our IBM 3995 Optical Library to disk. We have about 150 GB of archived customer bills, financials, etc... on about 35 5.2GB Optical Platters. So, we're adding the disk drives to compensate for that new data.

Storing the data on the drives will definitely help performance and we will also be able to end maintenance on that 3995. The 3995 is driven by an attached workstation that runs OS/2! We will be glad to get rid of this dog because even though IBM still supports the hardware, we're out of luck if we have any OS/2 issues!

Is Sametime 6.5.1 FP1 NOW Ready for Primetime?

Saw that the FP1 for Sametime 6.5.1 is available again. IBM had previously pulled it from the download area due to issues.

Has anyone upgraded? Should we even upgrade since Sametime 7.0 may be coming out within the next 6 months (this is just a guess because I know a 7.0 release will be out after ND7 is released)?

Continued 6.5.4 for iSeries Issues

If you've been keeping up with my blog, we've had our share of issues with Domino 6.5.4 on iSeries. We've been experiencing hangs on our mail subsystem and we've also seen some issues with our Sametime subsystem as well. We have loaded two sets of iSeries fixes for those along with 2 system restarts (IPL). On 6/24/05, we performed an IPL.

For the Sametime issue, we have loaded not one, but two hotfixes that did not correct the problem. We finally determined that the Sametime issue was with the way the Notes 6.5.2+ Clients handle integrated presence awareness. We haven't had a Sametime problem (yet) since our IPL on 6/24, so it may turn out that one of the fixes may correct this issue yet... but I won't hold my breath.

We also went nearly 2 weeks without a problem with our mail server. But on Wednesday we had another hang on our mail subsystem and then we had another today. I've sent IBM more sets of debug data just for them to tell me again that it's just a network problem.

On the upside, I have learned a lot over the past month(s) in regards to some system troubleshooting. I have learned that Sametime for iSeries creates some pretty slick trace files that can be sent to IBM without having to collect the data yourself. I've worked a good bit with notes.ini settings for enabling debug parameters for TCP as well. I think one of the biggest lessons may be that it might not be so wise to stay on top of things so much. At the same time, though, some of my other Domino for iSeries colleagues have not been experiencing any issues with 6.5.4. I guess it goes to show that each shop is really unique and you never know if you'll have a good or bad experience with an upgrade. I still believe my philosophy is that you should try to stay really current because even though strange things can break, there are still many other fixes that can be helpful with upgrades!

Mensa

Feeling pretty good about myself - "Your score was 23 out of 30. That is a very good score, you would have a good chance of passing the Mensa test."

How would you do?

http://www.mensa.org/workout2.php

Mensa Website: "Mensa was founded in England in 1946 by Roland Berrill, a barrister, and Dr. Lance Ware, a scientist and lawyer. They had the idea of forming a society for bright people, the only qualification for membership of which was a high IQ. The original aims were, as they are today, to create a society that is non-political and free from all racial or religious distinctions. The society welcomes people from every walk of life whose IQ is in the top 2% of the population, with the objective of enjoying each other"s company and participating in a wide range of social and cultural activities."

Another Reason to Like Firefox

OK, so my coworker is putting up a form on our Intranet. Since our corporate supported browser is IE, that's where he was going. He kept getting the following error:

"The page cannot be displayed
There is a problem with the page you are trying to reach and it cannot be displayed.
Please try the following:
Open the hqapp01.comporium.com home page, and then look for links to the information you want.
Click the Refresh button, or try again later.
Click Search to look for information on the Internet.
You can also see a list of related sites.
HTTP 500 - Internal server error
Internet Explorer"

He sent me the link and my default browser (FF) opened it and I get:

"Error 500
HTTP Web Server: Lotus Notes Exception - Incorrect data type for operator or @Function: Number expected"

We could have spent hours trying to find the problem in IE but FF just told me the actual error! I say we could have spent hours, but my next reasonable step would probably be to try opening it and then looking at Notes log where this error is logged.

Pretty cool that the error is passed to FF!