Random observations of a very experienced software artist.

    Domino/Notes fix pack installations fail on Temp directory

    John McCann  May 21 2018 03:02:30 PM
    In October 2015, we opened a PMR with IBM concerning the failure of Domino 9.0.1 Fix Pack 4 installation failing.  The log was this:

    Image:Domino/Notes fix pack installations fail on Temp directory
    The failure was attempting to launch an executable, in this case, java.exe from a temporary folder, i.e., one that had "temp" in its name.  Blocking such attempts stops many attack vectors that try to drop an executable in a safe folder and launch it.  What is sad is that the java.exe was put there by copying it from the <notes program directory>\jvm\bin folder the step before.  If IBM would change their script to launch it from where they just copied it, all would be copasetic.

    We started the conversation with FP4.  Then came FP5, FP5.1, FP5.2, and finally FP6.   You might not remember at the time, but Notes client install would ask you where to unpack the temporary files, but fix packs did not.  They just used the specified temporary directory.

    With FP6, SEVEN months after we opened the original problem, IBM finally told us about the hidden /P switch for the installation.  This solved our problem.  IBM opens an APAR, but does not provide the solution in the APAR: https://www-01.ibm.com/support/entdocview.wss?uid=swg1LO86833   They document an untenable solution as the "local fix", remove the security feature - ha!

    Now come to the present

    We have to install Notes 9.0.1 Fix Pack 10 on a system with only Notes 9.0.1 gold installed because it had been reimaged.  Notes is not on the base image.  Notes 9 install works fine, Fix Pack 10 fails.  Turns out to be the same error in the log from the 2015 problem - launching java.exe from a temporary directory.  Again, we are not disabling security.  Time to open another PMR.  First response back from IBM pointed us to the APAR that had been opened in 2016.  We had a good laugh.

    The IBM rep came back the same day with the suggestion to change the temporary folder location with the environment variables.  We had some miscommunication and issues because there are two variables that affect the install.

    The TMP environment variable is use to provide the default "where to unpack" similar to the following:
    Image:Domino/Notes fix pack installations fail on Temp directory

    IBM had suggested directly editing the TMP environment variable to change the location of the java execution.  All it did was change the default on the above dialog.  So, being the fool that I am, I decided to change the TEMP environment variable. to point to d:\ibm\installit.    

    Amazingly, it worked.  

    Then, I sheepishly looked at the very first line of the first log image above:  Get environment variable TEMP.    The clues had been in the log all along.  None of us involved in solving this problem were able to succinctly articulate the solution because of so many red herrings of where to unpack files and blocking software.  We could have saved more than NINE months of work if we had read a little more in the log.

    I also learned a neat trick from IBM along the way.   Take the Interim fix, that might not have a where to unpack; change the .exe to a .zip; open it with 7Zip and you can do the install unpack to where you want yourself.    Thank you Nelson C. DiƱo Jr..

      Unknown LotusScript Error means you forgot Java use statements

      John M McCann  March 4 2014 01:52:44 PM
      I received this error message attempting to run a LotusScript agent that connects to a database using ODBC and calls Java routines for password encryption and base 64 encoding routines..   It works great when testing on my Designer client when database was Microsoft Access.    All is good.

      But, the real reason for this agent is to handle inbound mail, i.e., run on a Domino server.   I merrily switch the agent to be triggered by schedule and run on the server.  The agent was smart enough to handle unprocessed documents or look in a view so it could be run manually, scheduled, or on various events.  

      I was ready for first time success, yet got the following error message on the console:

      AMgr: Agent 'foo' in 'bar.nsf' did not process all documents successfully.  Check the Agent Log for more information: Unknown LotusScript Error.

      What!   Not a new error.  Google even found an entry from 1996 where someone had encountered this problem.   No, I am not using NoteUIWorkspace in a scheduled agent.   Grr.

      Time for my favorite rant about how the poor attitude towards diagnostics by Notes/Domino development has contributed to Domino's current marketplace situation.  You know which view wasn't found, why not tell us.

      After a few hours of having to rebuild my agent a few lines of code at a time, I finally tracked it down.   Somebody (I will remain nameless) forgot to include the following two statements:

      UseLSX "*javacon "
      Use "javaroutine"   '<< name of java script library

      I hope this saves some reader the hours I had to waste on yet another obscure, worthless error message.

        Instr sensitivity training

        John McCann  November 30 2013 07:32:07 AM
        I have been parsing a rather large CSV file using LotusScript.    The Instr function was getting used extensively.  For example,  to count the number of quotes in a string, the following code is used:

                lngPos = InStr(1, strData, strQuote,k)
                lngQuotes = 0
                Do While (lngPos > 0)
                        lngQuotes = lngQuotes + 1
                        lngPos = InStr(lngPos + 1, strData, strQuote,k)

        The 4th parameter of the Instr function is compMethod, a number designating the comparison method.   I wondered what effect the compMethod would have.   For your reference, the options are:

        0        case-sensitive, pitch-sensitive
        1        case-insensitive, pitch-sensitive
        4        case-sensitive, pitch-insensitive
        5        case-insensitive, pitch-insensitive

        If you omit compMethod, the default comparison mode is the mode set by the Option Compare statement for the module. If there is none, the default is 0 - case-sensitive and pitch-sensitive.

        I took a line of data that I was handling.  It is 1321 characters long.  It contains 658 quotes (") to delineate text strings for the 329 fields the record contains.   Most of the fields are zero length strings.  The record contains only 355 characters of 'real' data.  I ran the above code segment 1000 times on my core i7 system using Domino 9.0.1.  I received  these results in seconds.:


        I was only searching for pairs of quotes and delimiters.   If you are searching for lots  text strings and are trying to ignore case, it may be more efficient to lowercase your strings before using instr if you have repeated searches for the same patter or against the same string.  I suspect it will be highly data dependent.

        What if I just counted the number of quotes.

                For i = 1 To Len(strData)
                        If Mid$(strData,i,1) = strQuote Then
                                lngQuotes = lngQuotes + 1
                        End If
                Next i

        It turns out, this was even faster.  For my sample data, it took only 2.76 to iterate 1000 times over this code segment.   But, would that be true for all data?  

        I constructed another test case.  It was the same length, 1321 characters, but it only contained 3 fields; 6 quotes.   I had to increase the iterations 10-fold to get usable results.  10,000 iterations yields 27.18 seconds for the raw count, very inefficient.   I got the following for Instr:

        Without a doubt, this confirms my wife's argument for me to be more sensitive.

          How not to perform a Select Case

          John McCann  September 20 2012 05:30:18 PM
          This is a LotusScript example of how NOT to code Select Case.   I found this in an application I had to change.

          The code was in a function that tries to find the proper document  in the Domino Directory based on having a full name, an email address, and/or the name from the x.509 certificate.  The routine takes a parameter on how to search:
          ' Parms:    rintType ......... flag that indicates what type of search to do
          '                    ......... 1 = Attempt to match on NameFull        
          '                    ......... 2 = Attempt to match on EMail
          '                    ......... 3 = Attempt to match on EMail Address if NameFull fails
          '                    ......... 4 = Attempt to match on PKI Name
          '                    ......... 5 = Attempt to match on PKI Name if NameFull and EMail fail

          Note that types 3 and 5 instruct the routine to use multiple methods until success is achieved.  The code was basically this:

           ' view we need Domino Directory
                  Set vwNAB = dbNAB.GetView("($Users)")
          ' Assume NOT found
                  rintFound = 0
          ' Processing depends on Request Type
                  Select Case rintType
                  Case Is = 1, 3, 5
                          '.. see if we can match on FullName
                          Set docPerson = vwNAB.GetDocumentByKey(rstrFullName, True)
                          If Not (docPerson Is Nothing) Then
                                  rintFound = 1
                          End If
                  Case Is = 2, 3, 5        
                          '.. if no match already or Email only .. see if we can match on EMail Address
                          If rintFound = 0 Then                  
                                  If rstrEMail <> "" Then
                                          If vwNAB.FTSearch(rstrEMail) > 0 Then
                                                  Set docPerson = vwNAB.GetFirstDocument()
                                                  rintFound = 2
                                          End If
                                  End If
                          End If
                  Case Is = 4, 5                
                          '.. see if we can match on PKI Name
                          If rintFound = 0 Then                  
                                  If rstrNamePKI <> "" Then
                                          Set docPerson = vwNAB.GetDocumentByKey(rstrPKIName, True)
                                          If Not (docPerson Is Nothing) Then
                                                  rintFound = 3
                                          End If
                                  End If
                          End If                
                  End Select

          Select case only takes a single "Case".  If rintType is 3 or 5, the 2nd and 3rd Cases never get chosen, only the first.   The routine fails to find the person document, even when there is a match on email or PKI name.   I can only guess that the original hacker thought Select Case evaluated each case in turn.

          By the way, there was another gotcha here.  The variable for the view, vwNAB, is established outside the routine that actually contains this code.  The routine fails if multiple techniques were used to call the function in a single execution of an agent, say Type=2 followed by Type=1.  The FTSearch is never cleared and any non-FTSearch based lookup into the view probably fails.  

          Yes, I did rewrite this routine.

          Lesson for the day: Select Case executes at most ONE case.

            Fun with @DbLookup

            John M McCann  August 24 2012 09:45:54 AM
            This is a re-post of an earlier item to fix a bug in the code, add more diagnostic information, and recreate the item after an update exceed the 32K limit on a field.

            After spending considerable wasted time on XPages Server Side JavaScript @DbLookup results, I thought I would share my findings. My results are specific to 8.5.3, though I suspect they are applicable to all 8.5.x versions.

            The official IBM documentation indicates that @DbLookup (JavaScript) "Returns view column or field values that correspond to matched keys in a sorted view column". The "Return value" is "any" and the description is "An array containing the column or item values". Most of us now know this is incorrect. If one or zero records matched the key, the value returned is a string, not an array.

            What many don't know are the circumstances where the result is "undefined". I found two. If you have specified the database incorrectly, e.g. made a typo, the result will be "undefined". The second situation is where you don't have access to the database. Mine was a keyword lookup type of database that had an ACL with maximum Internet name and password set to "No Access". The parameter [FAILSILENT] seems to have no affect on this.

            In the days attempting to debug my code, I also confirmed the multiple undocumented database specification techniques available, not just the server name array. All the following worked (once I increased maximum Internet access):

            // the defined way, an array of two elements
            var db = @DbName();

            // arrays pointing to another database on the same server
            var db = [@DbName()[0], "folder\\filename.nsf"]; // note double slashes
            var db = new Array(session.getServerName(), "folder/filename.nsf"); // slash the other way not doubled

            // C API at lowest level still uses old Notes 2 conventions of double bang
            var db = @DbName()[0], + "!!" + "folder\\filename.nsf";

            // And, you can specify replicaId either as a string or single element array
            var db = ["85256FF7:12345678"];
            var db = "85256FF7:12345678";

            // all work in this lookup
            @DbLookup(db, viewname, "key", column, "[FAILSILENT");

            With all this additional information about DbLookup that I uncovered the hard way, I thought it only appropriate to share.

            I also decided to update Tom Steenbergen's excellent wrapper routine for DbLookup with this information. While doing so, I identified that I really needed control over where I wanted the caching to occur. This lead to the discovery that @Unique affected the cache, and other issues. Therefore, I came up with the following derivation:

            /* *****************************************************************
             * Returns @DbLookup results as array and allows for cache  
             * Author: John McCann - derived from work by Tom Steenbergen
             * @param server -name of the server the database is on (only used if dbname not empty, if omitted, the server of the current database is used)  
             * @param dbname -name of the database (if omitted the current database is used)  
             * @param cache -empty for nocache, otherwise scope at which to cache  (application, request, view, session)
             * @param unique -"unique" for returning only unique values, empty or anything for all results  
             * @param sortit -"sort" for returning the values sorted alphabetically  
             * @param viewname -name of the view  
             * @param keyname -key value to use in lookup  
             * @param field -field name in the document or column number to retrieve
             * @param keywords - one or more comma separate strings containing [FAILSILENT], [PARTIALMATCH], or [RETURNDOCUMENTUNIQUEID]  
             * @return array with requested results  
              ****************************************************************** */
            function DbLookupArray(server, dbname, cache, unique, sortit, viewname, keyname, field, keywords) {  
                    var result;
                    try {
                        var cachekey = "dblookup_"+dbname+"_"+@ReplaceSubstring(viewname," ","_")+"_"+@ReplaceSubstring(keyname," ","_")+"-"+@ReplaceSubstring(field," ","_");
                            // if cache is specified, try to retrieve the cache from the appropriate scope
                        switch (cache.toLowerCase()) {
                        case "application":
                             result = applicationScope.get(cachekey);  
                        case "request":
                             result = requestScope.get(cachekey);  
                        case "view":
                             result = viewScope.get(cachekey);  
                        case "session":
                             result = sessionScope.get(cachekey);  

                        // if the result is empty, no cache was available or not requested,  
                        //  do the dblookup, convert to array if not, cache it when requested  
                        if (!result) {
                             // determine database to run against  
                             var db = "";  
                             if (!dbname.equals("")) { // if a database name is passed, build server, dbname array  
                                if (server.equals("")){
                                        db = new Array(@DbName()[0],dbname); // no server specified, use server of current database  
                                } else if (dbname.indexOf("!!")!=-1 || dbname.indexOf(":")!=-1){
                                        db = dbname;  // string value if double bang or replicaID spec
                                } else {
                                                    db = new Array(server, dbname);
                             var result = @DbLookup(db, viewname, keyname, field, keywords);
                             if (result==undefined){
                                     // if Mark Leusink's debug toolbar installed, put out diagnostics there
                                     if (dBar) {
                                             dBar.error("DbLookupArray returned undefined, cachekey=" + cachekey);
                                             // additional diagnostics
                                             var dbCheck:Notesdatabase;
                                             if (!db.equals("")){
                                                 if (server.equals("")){
                                                        dbCheck = session.getDatabase(@DbName()[0],dbname,false);  
                                                } else if (dbname.indexOf("!!")!=-1 || dbname.indexOf(":")!=-1){
                                                        dbCheck = session.getDatabase("",dbname,false)
                                                } else {
                                                                    db = session.getDatabase(server, dbname, false);
                                                 if (dbCheck == null || dbCheck==undefined){
                                                         dBar.debug ("DbLookupArray unable to access database, server=" + server + ", name=" + dbname);
                                             } else {
                                                     dbCheck = session.getCurrentDatabase()
                                             if (dbCheck.isOpen()){
                                                            var vwCheck:NotesView = dbCheck.getView(viewname);
                                                     if (vwCheck==null || vwCheck==undefined){
                                                             dBar.debug ("DBLookupArray Unable to find view, name=" + viewname);
                                             dBar.debug("DbLookupArray key value=" + keyname + ", field=" + field + ", keywords=" + keywords);

                            // have result, process it
                             } else {
                                     if (result) {
                                             // cache before manipulating
                                                  switch (cache.toLowerCase()) {
                                                             case "application":
                                                        case "request":
                                                        case "view":
                                                        case "session":
                                             if (typeof result == "string") {
                                                     result new Array(result);  
                                             } else {
                                                     // sort and unique only apply if multiple results
                                                     if (unique.toLowerCase()=="unique") result = @Unique(result);  
                                                     if (sortit.toLowerCase()=="sort") result.sort();
                        // we cached before operations on result set performed, so redo these if necessary
                        }  else {
                             if (typeof result == "string") {
                                     result new Array(result);  
                             } else {
                                             // sort and unique
                                    if (unique.toLowerCase()=="unique") result = @Unique(result);  
                                    if (sortit.toLowerCase()=="sort") result.sort();
                    } catch(e){
                            // this is our own error capture routine
                            if (dBar) {
                                    dBar.error("DbLookupArray Error " + result);
                    } finally {
                            return result;

              Arial is not my favorite

              John McCann  August 8 2012 01:13:33 PM
              I did not think an application rendered well using the Arial font in Internet Explorer 9.   So, I wanted to change it to Verdana.  It took me a while to chase down all the references to font-family in the multitude of disjoint places.  I was not impressed by the redundant specifications instead of letting inheritance work.    I tried using some recommendations to specify the dojo claro theme instead of the default tundra.  I didn't get claro's fonts either.

              Since it took me a while, I thought I would share what I did.  Hopefully, I can save someone else's time.  If you know of a better technique, please share.

              The application has its own theme, specified once on the application properties, XPages tab.

              Image:Arial is not my favorite

              In the theme, which is extending the oneUIV2.1 theme, we specify our own style sheet, MSE.css.   Make sure it is the last, even after any dojo theme override.
              <theme extends="oneuiv2_1_gen1">

                  <resource dojoTheme="true">


              Then our theme.  For your reading enjoyment, I also included the clearfix hack for clearing floats.
              /* clearfix hack from Jeff Staff: http://perishablepress.com/new-clearfix-hack/ */

              .clearfix:after {
                  visibility: hidden;
                  display: block;
                  font-size: 0;
                  content: "";
                  clear: both;
                  height: 0;
              * html .clearfix             { zoom: 1; } /* IE6 */
              *:first-child+html .clearfix { zoom: 1; } /* IE7 */

              /* override all the places the oneui thinks it needs to specify the font-family */
              .lotusui button, .lotusui input, .lotusui .lotusSymbol, .lotusui select, .lotusui textarea,
              .xspDataTableFileDownload table table td,
              .xspDataTableViewPanel table table td,
              .xspText, .xspTextComputedField, .xspTextLabel, .xspTextViewTitle, .xspTextViewColumn, .xspTextViewColumnComputed, .xspTextViewColumnHeader
                  {font-family: Verdana, Arial, Helvetica, sans-serif;}

                Duplicate EMail Elimination

                John McCann  May 17 2012 10:56:57 AM
                Argh, ran into a problem with my inbox filled with duplicates while I was having problems with an IMAP source.   I wrote a duplicate email eliminator that I thought others might be able to use to save themselves some time.

                ' Agent Duplicate Deleter
                ' Purpose:  Delete duplicates emails from selected list

                ' Change History:
                ' May 17, 2012 - John McCann
                ' - Initial Creation

                Option Public
                Option Declare

                ' Class Msg
                ' Description: Information to compare and find the email
                Class Msg
                        Public strUNID As String
                        Public strMsgID As String
                        Public strOther        As String
                        Public strSubject As String

                End Class
                Sub Initialize
                        Dim session                 As New NotesSession
                        Dim dbThis                        As NotesDatabase
                        Dim dcThis                        As NotesDocumentCollection
                        Dim docThis                        As NotesDocument
                        Dim itmMessageID        As NotesItem
                        Dim itmOther                As NotesItem
                        Dim strUNID                        As String
                        Dim fRemoved                As Boolean
                        Dim lstMsgs                        List As Msg
                        Dim lstIDs                        List As String
                        Dim vntIDs                        As Variant
                        Dim msgThis                        As Msg
                        Dim msgBase                        As Msg
                        Dim i                                As Long
                        On Error GoTo This_Error
                        Set dbThis = session.Currentdatabase
                        Set dcThis = dbThis.Unprocesseddocuments
                        Set docThis = dcThis.Getfirstdocument()
                        While Not docThis Is Nothing
                                strUNID = docThis.UniversalID
                                ' going to match on one of the message IDs
                                Set itmMessageID = docThis.GetFirstItem("$MessageID")
                                If itmMessageID Is Nothing Then
                                        Set itmMessageID = docThis.GetFirstItem("$IMAPUID")
                                End If
                                ' Need at least another field for uniqueness
                                Set itmOther = docThis.GetFirstItem("$INetOrig")
                                If itmOther Is Nothing Then
                                        Set itmOther = docThis.Getfirstitem("$Orig")
                                        If itmOther Is Nothing Then
                                                Set itmOther = docThis.Getfirstitem("$Abstract")
                                                If itmOther Is Nothing Then
                                                        Set itmOther = docThis.GetFirstitem("DomainKey_Signature")
                                                End If
                                        End If
                                End If
                                ' create the message for our list
                                Set msgThis = New Msg
                                With msgThis
                                        .strMsgID = itmMessageID.Text
                                        .strSubject = docThis.Subject(0)
                                        .strOther = itmOther.Text
                                        .strUNID = strUNID
                                End With
                                ' save the message
                                Set lstMsgs(strUNID) = msgThis
                                ' create a list by IDs for dup elimination
                                If IsElement(lstIDs(msgThis.strMsgID)) THen
                                        lstIDS(msgThis.strMsgID) = lstIDS(msgThis.strMsgID) & ";" & docThis.UniversalID
                                        lstIDS(msgThis.strMsgID) = docThis.UniversalID
                                End if
                                Set docThis = dcThis.Getnextdocument(docThis)
                        ' now, figure out which ones to remove
                        ForAll msgID In lstIDs
                                vntIDs = Split(msgID,";")
                                ' only if more than 1
                                If UBound(vntIDs) > 0 Then
                                        Set msgBase = lstMsgs(vntIDs(0))
                                        ' compare each to the first
                                         For i = 1 To UBound(vntIDs)
                                                 strUNID = vntIDs(i)
                                                 If strUNID <> "" Then
                                                         Set msgThis = lstMsgs(strUNID)
                                                         ' if all three items match, then remove
                                                         If msgThis.strSubject = msgBase.strSubject Then
                                                                 If msgThis.strOther = msgBase.strOther Then
                                                                         If msgThis.strMsgID = msgBase.strMsgID Then
                                                                                 Set docThis = dbThis.Getdocumentbyunid(strUNID)
                                                                                 Call docThis.Remove(True)
                                                                                 Erase lstMsgs(strUNID)
                                                                         End If
                                                                 End If
                                                         End If
                                                 End If
                                End If
                        End ForAll

                        Exit Sub
                        MsgBox "Error " & Error & ", Subject=" & docThis.Subject(0) & ", Time=" & CStr(docThis.Created)
                        Resume this_Exit
                End Sub

                  Day of Week

                  John McCann  February 29 2012 11:20:27 AM
                  I picked up someone else's code today and saw a technique that struck me.   The application was set up to need a three character day of week.

                  I look at the code and see the following construct:

                         intDayOfWeek = Weekday(Now)
                         Select Case intDayOfWeek
                                 Case Is = 1
                                         strDayOfWeek = "sun"
                                 Case Is = 2
                                         strDayOfWeek = "mon"
                                 Case Is = 3
                                         strDayOfWeek = "tue"
                                 Case Is = 4
                                         strDayOfWeek = "wed"
                                 Case Is = 5
                                         strDayOfWeek = "thu"
                                 Case Is = 6
                                         strDayOfWeek = "fri"
                                 Case Is = 7
                                         strDayOfWeek = "sat"
                         End Select

                  Besides the unnecessary "Is = ", something just nagged at me as there has got to be a better way.  Scratched around for a few minutes and came up with what I think is a simpler solution:

                         strDayOfWeek = lcase$(format$(Now(),"ddd"))

                  The found construct was only in one time initialization code, so performance effect is probably minimal - a slightly smaller module with execution difference smaller than the ticks with which you are measuring.  One could even argue that the first construct is clearer in what is being done so is more maintainable.  Something in me just likes the transforming solution.

                    D’OH moment on getDocument performance

                    John McCann  February 2 2012 12:23:35 PM
                    We had just had a discussion concerning performance - an agent that has to collect information on a few thousand documents to present data to a dashboard.   The agent carefully uses a NotesViewNavigator and ReadViewEntries to maximize performance.

                    I was debugging the agent to fix a few typos in an update I made and had reason to look at the Domino server console for potential error messages.  I saw on the console, warning messages from the anti-virus software about an attachment it couldn't scan.

                    Then, D'OH, I made the connection.   It isn't just that using the NotesView and ViewEntries to get the information you need is so much more efficient than getting a document collection and opening documents.  I suspect that the real performance gains are the fact that you avoid the antivirus scan of the document and all the attachments.