 John McCann | John M McCann March 4 2014 01:52:44 PMI received this error message attempting to run a LotusScript agent that connects to a database using ODBC and calls Java routines for password encryption and base 64 encoding routines.. It works great when testing on my Designer client when database was Microsoft Access. All is good. But, the real reason for this agent is to handle inbound mail, i.e., run on a Domino server. I merrily switch the agent to be triggered by schedule and run on the server. The agent was smart enough to handle unprocessed documents or look in a view so it could be run manually, scheduled, or on various events. I was ready for first time success, yet got the following error message on the console: AMgr: Agent 'foo' in 'bar.nsf' did not process all documents successfully. Check the Agent Log for more information: Unknown LotusScript Error. What! Not a new error. Google even found an entry from 1996 where someone had encountered this problem. No, I am not using NoteUIWorkspace in a scheduled agent. Grr. Time for my favorite rant about how the poor attitude towards diagnostics by Notes/Domino development has contributed to Domino's current marketplace situation. You know which view wasn't found, why not tell us. After a few hours of having to rebuild my agent a few lines of code at a time, I finally tracked it down. Somebody (I will remain nameless) forgot to include the following two statements: UseLSX "*javacon " Use "javaroutine" '<< name of java script library I hope this saves some reader the hours I had to waste on yet another obscure, worthless error message. John McCann November 30 2013 07:32:07 AMI have been parsing a rather large CSV file using LotusScript. The Instr function was getting used extensively. For example, to count the number of quotes in a string, the following code is used: lngPos = InStr(1, strData, strQuote,k)
lngQuotes = 0
Do While (lngPos > 0)
lngQuotes = lngQuotes + 1
lngPos = InStr(lngPos + 1, strData, strQuote,k)
Loop | The 4th parameter of the Instr function is compMethod, a number designating the comparison method. I wondered what effect the compMethod would have. For your reference, the options are: 0 case-sensitive, pitch-sensitive 1 case-insensitive, pitch-sensitive 4 case-sensitive, pitch-insensitive 5 case-insensitive, pitch-insensitive If you omit compMethod, the default comparison mode is the mode set by the Option Compare statement for the module. If there is none, the default is 0 - case-sensitive and pitch-sensitive. I took a line of data that I was handling. It is 1321 characters long. It contains 658 quotes (") to delineate text strings for the 329 fields the record contains. Most of the fields are zero length strings. The record contains only 355 characters of 'real' data. I ran the above code segment 1000 times on my core i7 system using Domino 9.0.1. I received these results in seconds.: | compMethod | Time | | 0 | 3.89 | | 1 | 29.25 | | 4 | 5.41 | | 5 | 30.17 | I was only searching for pairs of quotes and delimiters. If you are searching for lots text strings and are trying to ignore case, it may be more efficient to lowercase your strings before using instr if you have repeated searches for the same patter or against the same string. I suspect it will be highly data dependent. What if I just counted the number of quotes. For i = 1 To Len(strData)
If Mid$(strData,i,1) = strQuote Then
lngQuotes = lngQuotes + 1
End If
Next i | It turns out, this was even faster. For my sample data, it took only 2.76 to iterate 1000 times over this code segment. But, would that be true for all data? I constructed another test case. It was the same length, 1321 characters, but it only contained 3 fields; 6 quotes. I had to increase the iterations 10-fold to get usable results. 10,000 iterations yields 27.18 seconds for the raw count, very inefficient. I got the following for Instr: | compMethod | Time | | 0 | 0.36 | | 1 | 3.18 | | 4 | 0.49 | | 5 | 3.26 | Without a doubt, this confirms my wife's argument for me to be more sensitive. John McCann September 20 2012 05:30:18 PMThis is a LotusScript example of how NOT to code Select Case. I found this in an application I had to change. The code was in a function that tries to find the proper document in the Domino Directory based on having a full name, an email address, and/or the name from the x.509 certificate. The routine takes a parameter on how to search: ' Parms: rintType ......... flag that indicates what type of search to do
' ......... 1 = Attempt to match on NameFull
' ......... 2 = Attempt to match on EMail
' ......... 3 = Attempt to match on EMail Address if NameFull fails
' ......... 4 = Attempt to match on PKI Name
' ......... 5 = Attempt to match on PKI Name if NameFull and EMail fail | Note that types 3 and 5 instruct the routine to use multiple methods until success is achieved. The code was basically this: ' view we need Domino Directory
Set vwNAB = dbNAB.GetView("($Users)")
' Assume NOT found
rintFound = 0
' Processing depends on Request Type
Select Case rintType
Case Is = 1, 3, 5
'.. see if we can match on FullName
Set docPerson = vwNAB.GetDocumentByKey(rstrFullName, True)
If Not (docPerson Is Nothing) Then
rintFound = 1
End If
Case Is = 2, 3, 5
'.. if no match already or Email only .. see if we can match on EMail Address
If rintFound = 0 Then
If rstrEMail <> "" Then
If vwNAB.FTSearch(rstrEMail) > 0 Then
Set docPerson = vwNAB.GetFirstDocument()
rintFound = 2
End If
End If
End If
Case Is = 4, 5
'.. see if we can match on PKI Name
If rintFound = 0 Then
If rstrNamePKI <> "" Then
Set docPerson = vwNAB.GetDocumentByKey(rstrPKIName, True)
If Not (docPerson Is Nothing) Then
rintFound = 3
End If
End If
End If
End Select | Select case only takes a single "Case". If rintType is 3 or 5, the 2nd and 3rd Cases never get chosen, only the first. The routine fails to find the person document, even when there is a match on email or PKI name. I can only guess that the original hacker thought Select Case evaluated each case in turn. By the way, there was another gotcha here. The variable for the view, vwNAB, is established outside the routine that actually contains this code. The routine fails if multiple techniques were used to call the function in a single execution of an agent, say Type=2 followed by Type=1. The FTSearch is never cleared and any non-FTSearch based lookup into the view probably fails. Yes, I did rewrite this routine. Lesson for the day: Select Case executes at most ONE case. John M McCann August 24 2012 09:45:54 AM This is a re-post of an earlier item to fix a bug in the code, add more diagnostic information, and recreate the item after an update exceed the 32K limit on a field.
After spending considerable wasted time on XPages Server Side JavaScript @DbLookup results, I thought I would share my findings. My results are specific to 8.5.3, though I suspect they are applicable to all 8.5.x versions.
The official IBM documentation indicates that @DbLookup (JavaScript) "Returns view column or field values that correspond to matched keys in a sorted view column". The "Return value" is "any" and the description is "An array containing the column or item values". Most of us now know this is incorrect. If one or zero records matched the key, the value returned is a string, not an array.
What many don't know are the circumstances where the result is "undefined". I found two. If you have specified the database incorrectly, e.g. made a typo, the result will be "undefined". The second situation is where you don't have access to the database. Mine was a keyword lookup type of database that had an ACL with maximum Internet name and password set to "No Access". The parameter [FAILSILENT] seems to have no affect on this.
In the days attempting to debug my code, I also confirmed the multiple undocumented database specification techniques available, not just the server name array. All the following worked (once I increased maximum Internet access):
// the defined way, an array of two elements
var db = @DbName();
// arrays pointing to another database on the same server
var db = [@DbName()[0], "folder\\filename.nsf"]; // note double slashes
var db = new Array(session.getServerName(), "folder/filename.nsf"); // slash the other way not doubled
// C API at lowest level still uses old Notes 2 conventions of double bang
var db = @DbName()[0], + "!!" + "folder\\filename.nsf";
// And, you can specify replicaId either as a string or single element array
var db = ["85256FF7:12345678"];
var db = "85256FF7:12345678";
// all work in this lookup
@DbLookup(db, viewname, "key", column, "[FAILSILENT");
With all this additional information about DbLookup that I uncovered the hard way, I thought it only appropriate to share.
I also decided to update Tom Steenbergen's excellent wrapper routine for DbLookup with this information. While doing so, I identified that I really needed control over where I wanted the caching to occur. This lead to the discovery that @Unique affected the cache, and other issues. Therefore, I came up with the following derivation:
/* *****************************************************************
* Returns @DbLookup results as array and allows for cache
* Author: John McCann - derived from work by Tom Steenbergen
* @param server -name of the server the database is on (only used if dbname not empty, if omitted, the server of the current database is used)
* @param dbname -name of the database (if omitted the current database is used)
* @param cache -empty for nocache, otherwise scope at which to cache (application, request, view, session)
* @param unique -"unique" for returning only unique values, empty or anything for all results
* @param sortit -"sort" for returning the values sorted alphabetically
* @param viewname -name of the view
* @param keyname -key value to use in lookup
* @param field -field name in the document or column number to retrieve
* @param keywords - one or more comma separate strings containing [FAILSILENT], [PARTIALMATCH], or [RETURNDOCUMENTUNIQUEID]
* @return array with requested results
****************************************************************** */
function DbLookupArray(server, dbname, cache, unique, sortit, viewname, keyname, field, keywords) {
var result;
try {
var cachekey = "dblookup_"+dbname+"_"+@ReplaceSubstring(viewname," ","_")+"_"+@ReplaceSubstring(keyname," ","_")+"-"+@ReplaceSubstring(field," ","_");
// if cache is specified, try to retrieve the cache from the appropriate scope
switch (cache.toLowerCase()) {
case "application":
result = applicationScope.get(cachekey);
break;
case "request":
result = requestScope.get(cachekey);
break;
case "view":
result = viewScope.get(cachekey);
break;
case "session":
result = sessionScope.get(cachekey);
}
// if the result is empty, no cache was available or not requested,
// do the dblookup, convert to array if not, cache it when requested
if (!result) {
// determine database to run against
var db = "";
if (!dbname.equals("")) { // if a database name is passed, build server, dbname array
if (server.equals("")){
db = new Array(@DbName()[0],dbname); // no server specified, use server of current database
} else if (dbname.indexOf("!!")!=-1 || dbname.indexOf(":")!=-1){
db = dbname; // string value if double bang or replicaID spec
} else {
db = new Array(server, dbname);
}
}
var result = @DbLookup(db, viewname, keyname, field, keywords);
if (result==undefined){
// if Mark Leusink's debug toolbar installed, put out diagnostics there
if (dBar) {
dBar.error("DbLookupArray returned undefined, cachekey=" + cachekey);
// additional diagnostics
var dbCheck:Notesdatabase;
if (!db.equals("")){
if (server.equals("")){
dbCheck = session.getDatabase(@DbName()[0],dbname,false);
} else if (dbname.indexOf("!!")!=-1 || dbname.indexOf(":")!=-1){
dbCheck = session.getDatabase("",dbname,false)
} else {
db = session.getDatabase(server, dbname, false);
}
if (dbCheck == null || dbCheck==undefined){
dBar.debug ("DbLookupArray unable to access database, server=" + server + ", name=" + dbname);
}
} else {
dbCheck = session.getCurrentDatabase()
}
if (dbCheck.isOpen()){
var vwCheck:NotesView = dbCheck.getView(viewname);
if (vwCheck==null || vwCheck==undefined){
dBar.debug ("DBLookupArray Unable to find view, name=" + viewname);
}
}
dBar.debug("DbLookupArray key value=" + keyname + ", field=" + field + ", keywords=" + keywords);
}
// have result, process it
} else {
if (result) {
// cache before manipulating
switch (cache.toLowerCase()) {
case "application":
applicationScope.put(cachekey,result);
break;
case "request":
requestScope.put(cachekey,result);
break;
case "view":
viewScope.put(cachekey,result);
break;
case "session":
sessionScope.put(cachekey,result);
}
if (typeof result == "string") {
result new Array(result);
} else {
// sort and unique only apply if multiple results
if (unique.toLowerCase()=="unique") result = @Unique(result);
if (sortit.toLowerCase()=="sort") result.sort();
}
}
}
// we cached before operations on result set performed, so redo these if necessary
} else {
if (typeof result == "string") {
result new Array(result);
} else {
// sort and unique
if (unique.toLowerCase()=="unique") result = @Unique(result);
if (sortit.toLowerCase()=="sort") result.sort();
}
}
} catch(e){
// this is our own error capture routine
result=jsError(e);
if (dBar) {
dBar.error("DbLookupArray Error " + result);
}
} finally {
return result;
}
}
| John McCann August 8 2012 01:13:33 PMI did not think an application rendered well using the Arial font in Internet Explorer 9. So, I wanted to change it to Verdana. It took me a while to chase down all the references to font-family in the multitude of disjoint places. I was not impressed by the redundant specifications instead of letting inheritance work. I tried using some recommendations to specify the dojo claro theme instead of the default tundra. I didn't get claro's fonts either. Since it took me a while, I thought I would share what I did. Hopefully, I can save someone else's time. If you know of a better technique, please share. The application has its own theme, specified once on the application properties, XPages tab.  In the theme, which is extending the oneUIV2.1 theme, we specify our own style sheet, MSE.css. Make sure it is the last, even after any dojo theme override. <theme extends="oneuiv2_1_gen1">
<resource dojoTheme="true">
<content-type>text/css</content-type>
<href>/.ibmxspres/dojoroot/dijit/themes/claro/claro.css</href>
</resource>
<resource>
<content-type>text/css</content-type>
<href>MSE.css</href>
</resource>
</theme> | Then our theme. For your reading enjoyment, I also included the clearfix hack for clearing floats. /* clearfix hack from Jeff Staff: http://perishablepress.com/new-clearfix-hack/ */
.clearfix:after {
visibility: hidden;
display: block;
font-size: 0;
content: "";
clear: both;
height: 0;
}
* html .clearfix { zoom: 1; } /* IE6 */
*:first-child+html .clearfix { zoom: 1; } /* IE7 */
/* override all the places the oneui thinks it needs to specify the font-family */
body.lotusui,
.lotusui button, .lotusui input, .lotusui .lotusSymbol, .lotusui select, .lotusui textarea,
.xspDataTableFileDownload table table td,
.xspDataTableViewPanel table table td,
.xspInputFieldTextArea,
.xspText, .xspTextComputedField, .xspTextLabel, .xspTextViewTitle, .xspTextViewColumn, .xspTextViewColumnComputed, .xspTextViewColumnHeader
{font-family: Verdana, Arial, Helvetica, sans-serif;} | John McCann May 17 2012 10:56:57 AMArgh, ran into a problem with my inbox filled with duplicates while I was having problems with an IMAP source. I wrote a duplicate email eliminator that I thought others might be able to use to save themselves some time. ' Agent Duplicate Deleter ' Purpose: Delete duplicates emails from selected list ' Change History: ' May 17, 2012 - John McCann ' - Initial Creation Option Public Option Declare ' Class Msg ' Description: Information to compare and find the email Class Msg Public strUNID As String Public strMsgID As String Public strOther As String Public strSubject As String End Class Sub Initialize Dim session As New NotesSession Dim dbThis As NotesDatabase Dim dcThis As NotesDocumentCollection Dim docThis As NotesDocument Dim itmMessageID As NotesItem Dim itmOther As NotesItem Dim strUNID As String Dim fRemoved As Boolean Dim lstMsgs List As Msg Dim lstIDs List As String Dim vntIDs As Variant Dim msgThis As Msg Dim msgBase As Msg Dim i As Long On Error GoTo This_Error Set dbThis = session.Currentdatabase Set dcThis = dbThis.Unprocesseddocuments Set docThis = dcThis.Getfirstdocument() While Not docThis Is Nothing strUNID = docThis.UniversalID ' going to match on one of the message IDs Set itmMessageID = docThis.GetFirstItem( "$MessageID") If itmMessageID Is Nothing Then Set itmMessageID = docThis.GetFirstItem( "$IMAPUID") End If ' Need at least another field for uniqueness Set itmOther = docThis.GetFirstItem( "$INetOrig") If itmOther Is Nothing Then Set itmOther = docThis.Getfirstitem( "$Orig") If itmOther Is Nothing Then Set itmOther = docThis.Getfirstitem( "$Abstract") If itmOther Is Nothing Then Set itmOther = docThis.GetFirstitem( "DomainKey_Signature") End If End If End If ' create the message for our list Set msgThis = New Msg With msgThis .strMsgID = itmMessageID.Text .strSubject = docThis.Subject( 0) .strOther = itmOther.Text .strUNID = strUNID End With ' save the message Set lstMsgs(strUNID) = msgThis ' create a list by IDs for dup elimination If IsElement(lstIDs(msgThis.strMsgID)) THen lstIDS(msgThis.strMsgID) = lstIDS(msgThis.strMsgID) & ";" & docThis.UniversalID Else lstIDS(msgThis.strMsgID) = docThis.UniversalID End if Set docThis = dcThis.Getnextdocument(docThis) Wend ' now, figure out which ones to remove ForAll msgID In lstIDs vntIDs = Split(msgID, ";") ' only if more than 1 If UBound(vntIDs) > 0 Then Set msgBase = lstMsgs(vntIDs( 0)) ' compare each to the first For i = 1 To UBound(vntIDs) strUNID = vntIDs(i) If strUNID <> "" Then Set msgThis = lstMsgs(strUNID) ' if all three items match, then remove If msgThis.strSubject = msgBase.strSubject Then If msgThis.strOther = msgBase.strOther Then If msgThis.strMsgID = msgBase.strMsgID Then Set docThis = dbThis.Getdocumentbyunid(strUNID) Call docThis.Remove(True) Erase lstMsgs(strUNID) End If End If End If End If Next End If End ForAll This_Exit: Exit Sub This_Error: MsgBox "Error " & Error & ", Subject=" & docThis.Subject( 0) & ", Time=" & CStr(docThis.Created) Resume this_Exit End Sub John McCann February 29 2012 11:20:27 AMI picked up someone else's code today and saw a technique that struck me. The application was set up to need a three character day of week. I look at the code and see the following construct: intDayOfWeek = Weekday(Now) Select Case intDayOfWeek Case Is = 1 strDayOfWeek = "sun" Case Is = 2 strDayOfWeek = "mon" Case Is = 3 strDayOfWeek = "tue" Case Is = 4 strDayOfWeek = "wed" Case Is = 5 strDayOfWeek = "thu" Case Is = 6 strDayOfWeek = "fri" Case Is = 7 strDayOfWeek = "sat" End Select Besides the unnecessary "Is = ", something just nagged at me as there has got to be a better way. Scratched around for a few minutes and came up with what I think is a simpler solution: strDayOfWeek = lcase$(format$(Now(),"ddd")) The found construct was only in one time initialization code, so performance effect is probably minimal - a slightly smaller module with execution difference smaller than the ticks with which you are measuring. One could even argue that the first construct is clearer in what is being done so is more maintainable. Something in me just likes the transforming solution. John McCann February 2 2012 12:23:35 PMWe had just had a discussion concerning performance - an agent that has to collect information on a few thousand documents to present data to a dashboard. The agent carefully uses a NotesViewNavigator and ReadViewEntries to maximize performance. I was debugging the agent to fix a few typos in an update I made and had reason to look at the Domino server console for potential error messages. I saw on the console, warning messages from the anti-virus software about an attachment it couldn't scan. Then, D'OH, I made the connection. It isn't just that using the NotesView and ViewEntries to get the information you need is so much more efficient than getting a document collection and opening documents. I suspect that the real performance gains are the fact that you avoid the antivirus scan of the document and all the attachments. John McCann August 6 2011 07:21:34 AMI was upgrading a major application to a new release. When we deployed on the QA system and started running against the full data load, the overnight processing agent that goes in and touches a good percentage of the 100K records in the application started crashing the server. We got the dreaded "LSXBE: ****** Out of Backend Memory *******"" error. We reported the problem to IBM who asked for all the logs, copies of the database, etc. After having a good laugh that they would not be getting a copy of databases, they eventually pointed us to a posting that effectively said, "The cause of the problem is not recycling Domino Java API objects correctly." Only problem is that the agent was written in LotusScript, not Java. We had a number of false starts and red herrings (session.getDatabase will not return a database object if consistency check going on - duh). The failing call stack, as shown below, had us focusing on DocumentCollection processing. We were expecting a failure on getting or traversing a collection. We never tracked it down exactly, but are pretty sure the actual failure was on a NotesDocumentCollection.getNextDocument statement. The failing stack was: ### FATAL THREAD 1/3 [ nAMgr: 0f34: 0324] Exception code: c0000005 (ACCESS_VIOLATION) @[ 1] 0x60001706 nnotes.OSLockWriteSem@4+22 (19a) @[ 2] 0x61cf4a78 nlsxbe.ANNote::ANNAddToCollList+56 (0) @[ 3] 0x61d26127 nlsxbe.ANDocColl::ANDCNavigate+535 (0) In one of the iterations of trying to isolate exactly where the problem was occurring, IBM support (thank you Charles) pointed out that we just may be running into resource limitations on the machine since it looked like we were running into memory problems. It seems we had all these objects sitting around in memory. We started looking at the code and said, no, we clean up. See, for each record in the primary database, we create an object, process it, then set the object to Nothing at the end of handling the record. The object we create has other objects it creates, but LotusScript should be cleaning up all those when the primary object gets destroyed, right? Seemed logical, only in Java do you have to do your own recycling. Hmm, I mused. Are there any known problems with NotesDateTime variables and memory leaks? The big change in this upgrade was to add using NotesDateTime variables instead of LotusScript date/time variants since we started worrying about time zones and localizing date and times shown. Charles looked it up and indicated nothing known there. So, just for chagrins, when adding the latest tracing to produce a bunch more messages, I went through and added destructors to all my classes and changed the set to nothing to delete statements. (I also found out that unless a class has a Delete method, don't invoke delete on the class). Lo and behold, the problem goes away. I have neither the tools nor the time to examine internal memory usage at points in time. I will leave that to the Lotus developers. However, what I think I have discovered empirically is one of two situations. And, I don't know which one it is, so I left the code in for both thinking (perhaps wrongly) that it does no harm. 1) LotusScript will not clean up a list of objects contained within another object 2) A NotesDateTime contained within an object is not properly cleaned up when the object is destroyed. Simplified code before (real routine is over 8K lines of LotusScript) Option Explicit Public Class cFoo Public ndtEvent as New NotesDateTime("") End Class Public Class cBar Public lstFoo List as cFoo Sub New Dim i as Long Dim oFoo as cFoo For i = 1 to 10 Set oFoo = New cFoo Set lstFoo(i) = oFoo Next I End Sub End Class Sub Initialize Dim session as new NotesSession Dim dbCur as NotesDatabase Dim vwCur as NotesView Dim docCur as NotesDocument Dim oBar as cBar Set dbCur = session.CurrentDatabase() Set vwCur = dbCur.GetView("SomeView") Set docCur = vwCur.GetFirstDocument() While not docCur is Nothing Set oBar = new cBar ' **** do something with the document Set oBar = Nothing Set docCur = vwCur.GetNextDocument(docCur) Loop End Sub I have not compiled the above routine nor tested it, so it may have typos. But, it should provide the basics of what was going on. I assumed (bad choice by me) that the set of oBar to Nothing would clean up the memory of the cBar object as well as the 10 cFoo objects it created. When I changed the code to have my own destructor routines and invoke Delete as appropriate, the memory problems went away. As indicated above, I wasn't able to determine the root cause of the memory problem, so I shotgunned and cleaned up things that had changed (Datetime) and things I knew about. Option Explicit Public Class cFoo Public ndtEvent as New NotesDateTime("") Sub Delete Set ndtEvent = Nothing End Sub End Class Public Class cBar Public lstFoo List as cFoo Sub New Dim i as Long Dim oFoo as cFoo For i = 1 to 10 Set oFoo = New cFoo Set lstFoo(i) = oFoo Next I End Sub Sub Delete Forall Foo in lstFoo Delete Foo End Forall End Sub End Class Sub Initialize Dim session as new NotesSession Dim dbCur as NotesDatabase Dim vwCur as NotesView Dim docCur as NotesDocument Dim oBar as cBar Set dbCur = session.CurrentDatabase() Set vwCur = dbCur.GetView("SomeView") Set docCur = vwCur.GetFirstDocument() While not docCur is Nothing Set oBar = new cBar ' **** do something with the document Delete oBar Set docCur = vwCur.GetNextDocument(docCur) Loop End Sub Let me know if this solves a memory problem for you and I will pass the feedback to Lotus support that there really is a problem in this area. 2011-08-18 Update: It has been suggested that delete method for cBar Erase the lstFoo element instead. Not having the time to test all the options, I've updated my production code to do both - issue the delete then Erase lstFoo. |
|