NOTIS comes to Harvard

(and stays for nineteen years)

Prologue

 

At the request of Jerry Specht, three veterans of the Harvard University Library (HUL) Office for Information Systems (OIS) have searched their memories and existing documentation to assemble an account of Harvard’s selection, implementation and local development of NOTIS.  These are

·       Dale Flecker, leader of the project, Head of the library systems office from 1980 until July 1985 when he received the title of Associate Director for Planning and Systems in the University Library, which he held until his retirement in 2008.

·       Tracey Robinson, who now is Managing Director of OIS’s successor unit, Library Technology Services, of Harvard University Information Technology.  Tracey came to OSPR in 1981, hired specifically to work with extension of automated services for library acquisitions and fund accounting.

·       Charles Husbands, Senior Systems Librarian, a programmer-analyst knowledgeable about Harvard’s pre-NOTIS automated applications who eventually became intimately acquainted with nearly all NOTIS/HOLLIS programs and functions.  He retired in 2007. To Charlie has fallen principal responsibility for putting this report into words. Should a first person singular pronoun appear in the text, it will surely refer to him; i.e., to me.

Missing from this roster is Priscilla Caplan, who also played a major role in NOTIS installation, specification and HOLLIS development, then left Harvard in 1993 for professional opportunities in Chicago and subsequently in Florida where she is less conveniently consulted than the three of us who remain in the Boston area.

 

The collaborators in this effort have often been surprised at the gaps in their memories and in the readily available documentation.  The reader is advised not to rely on this account as definitive. Considerable detail is in the realm of best guess. 

 

We want to make it clear that the success of NOTIS at Harvard is due not just to our efforts but to those of many, many Harvard colleagues, especially our fellow members of the library systems office (OSPR subsequently OIS) who, as a team, worked to install, maintain, enhance, and operate NOTIS-based HOLLIS over its Harvard lifetime of nearly two decades.    

Mise en scène

 

This section may seem an awfully long introduction to the central theme of this story, but conversion of pre-existent data from several sources and production systems to fit into a NOTIS framework is a signal feature of Harvard’s experience; so here we go. 

 

The Harvard University Library began using automated techniques in the early mid-1960’s, quaint as those seem fifty years later. There were small independent projects run by the libraries of the schools of Business and Medicine, but the major focus of development lay in the central assemblage of units known as the College Library, among which Widener was the giant in size and influence. (All in all there were about 90 identifiable units in HUL at that time, many of them tiny. There has been considerable consolidation since.)  The College Library Data Processing Department in Widener, founded in 1964, was the home of local automation development. It was all local then.

 

The first production system in use at Widener was a unit record circulation system developed in-house by Foster Palmer, at that time Head of Widener’s Public Services Department. Loans were recorded on IBM punched cards and manipulated by sorting and collating machines. A nice sidelight on this pioneering effort is that it replaced an edge-notched card system implemented by Frederick Kilgour, when he worked in Widener circulation right after graduating from Harvard. Kilgour’s innovation was designed to cope with increased loan volume. Palmer’s system was developed for exactly the same reason. It, too, became cumbersome as circulation activity continued to increase, although crisis was deferred by progressive shifts of operations from card sorter and collator to an IBM 1401 computer and then, before 1980, to IBM 360 series machines. Because the Palmer system represented dates sequentially in four decimal digits, starting from 1 January 1963, a date later than 17 May 1990 would break it.

 

The second major 1960’s system was conversion of the Widener Library Shelflist to an electronic file originally maintained on magnetic tape, working again from punched card input.  This project has been well described in an article by Richard DeGennaro in College and Research Libraries, 1970 v.31(5) p.318-331.  Its relevance to the NOTIS story is that the never-finished project produced by the end of its 1970’s heyday a file of more than a million brief bibliographic records corresponding to Widener holdings as represented in handwritten shelflist entries..

 

Finally, the locally developed Computer Assisted Order System (cheekily known by its acronym, CAOS) went into production. The impetus for acquisition automation was the need for current information on the status of a large number of restricted funds.  Again, the machinable input was punched cards.  Original outputs were paper purchase orders and claims, vouchers for payment of invoices, and a variety of fiscal reports.  First implemented for the 1401 in 1967, it was made considerably more powerful when it was reprogrammed for PL/1 in 1970.  Sometime thereafter, several voluminous paper reports were superseded by computer output microfiche (COM).  The specifics of original CAOS functionality are now obscured by its evolution around 1976 into a still more capable version known as CAPS, the Computer Assisted Processing System. A problem: both CAOS and CAPS used typed search slips as source documents, thereby requiring much of the data to be keyed twice.

Cataloging in OCLC and participation in CONSER soon provoked development of other local systems. These activities generated MARC records that were collected and maintained, in a Bibliographic Master File (BMF) and a CONSER serials file respectively. CONSER, especially, aimed to capture data representing holdings of all HUL units for a Central Serials File. A few library units had ways to produce MARC records on site without recourse to OCLC or, eventually, to RLIN. OSPR accumulated these under the rubric “Local MARC”.    

The Spirit of ‘76

 

In that Bicentennial year and the years immediately following, a number of important things happened to affect Harvard’s automation efforts.

·     Several library units outside Widener, and even outside the College Library began to be seriously interested in participating in OSPR services. A few actually began using CAPS. The Boston Theological Institute, of which the Harvard Divinity School was a member, added its serial holdings to the CONSER File.

·     The College Library Data Processing Department was reconstituted as a University Library unit, with the object of focusing major automation efforts in a single place. It operated under the name Office for Systems Planning and Research (OSPR) until 1993 when it was renamed the Office for Information Systems (OIS) to encompass its responsibility for implementing productive information systems and to acknowledge that research was not a primary activity.

·     Changes in cataloging practices—the advent of AACR2 and Widener’s decision to abandon its unique classification scheme in favor of that of the Library of Congress—encouraged Harvard to close its five and a half million card Union Catalogue (a main entry catalog) and to begin a supplementary catalog, also on cards. The data in the BMF enabled OSPR periodically to generate a COM version of the supplementary catalog, that also included works on order from CAPS. It was known as the Distributable Union Catalog (DUC, known as “the duck”) because copies for public access were made available to all library units requesting them. Author & title and subject versions were generated.

 

There followed a period of transition of leadership in the University Library and in OSPR that precluded major initiatives but allowed opportunity to consider what might and what should be undertaken. It was clear that systems built to handle the needs of library units throughout the university would be needed to fulfill the mission of OSPR. The university’s technical environment was advancing to a point where batch processing systems that relied on keypunch machine preparation of input data seemed antiquated, Some operations, notably the duplicate keying of CAPS input, became burdensome. The provision of CAPS information via batch reports lacked the desired timeliness.

 

It was also a period of technical transition outside the university. Greatly expanded storage capacities and new telecommunication possibilities opened opportunities for new ways of approaching library problems. The first Online Public Access Catalogs debuted. The earliest generation of integrated library system products appeared on the market—turnkey systems, most of them.

 

One for the money

 

At the dawn of the ’80’s with Professor Oscar Handlin newly appointed as Director of the University Library and Dale assuming the leadership of OSPR, Harvard was well positioned to consider improving its aging library automation efforts.

·       The batch systems were cumbersome for efficient processing and would become more so if expanded to serve more libraries. Consider for instance:

o   Double keying of input data.

o   Transport of materials—cards, printed products—around campus.

·       There was a need for more timely data than batch CAPS could deliver.

·       Efficient linkage of acquisitions and cataloging operations would require more sophisticated bibliographic data (i.e. MARC) than CAPS had provided.

·       A system sharing data across the University Library would be highly desirable.  

That last point begs a digression: Harvard’s traditional philosophy of financial organization has long been characterized by the phrase “Every Tub on its Own Bottom,” which implies a great deal of independence in the way the College and each graduate school manages its own business. A similar style of particularity pertains within the hierarchy of each school. There are agencies that provide university-wide service but they are relatively few. Operationally, ETOIOB means that if one unit chooses—it is usually a choice—to receive services from another, it typically has to pay for them (One consequence of this is an enormous chart of accounts. The various branches of the College Library alone have access to a few hundred restricted funds; adding those of other faculties brings the total to around a thousand. But the chart of accounts is only one example, practically any sort of functional entity you can name, any table you might need to build, exhibits a similar degree of granularity.)     

 

A small study conducted jointly by Harvard’s Office of Information Technology and OSPR indicated the economic feasibility of replacing CAPS with an online system, a prospect unimaginable only a few years earlier. But enlarging the scope of the online acquisitions system to serve the University Library campus-wide would be very costly, not least because of the need to purchase a large number of terminals and build a network for them. The university approached the Pew Memorial Trust for support. The appeal was successful, resulting in the award in 1983 of a grant of $1.2 million for the purpose.    

Two for the show

 

As early as the end of 1980, an Online Acquisitions Planning Committee had been appointed to examine online acquisitions processing systems in use or development elsewhere and to specify the functions a single such system must have to accommodate the needs of the University Library’s many components units.

A concomitant part of the committee’s charge was to determine whether Harvard should buy an existing system or design its own.  While home-grown design was attractive, rough preliminary estimates suggested that the cost of acquiring an existing system would be about two thirds of that for building one in-house.  But which one would best meet Harvard’s needs, or seemed susceptible of modification to do so? Three types of external CAPS replacement options were examined:

·       Remote systems (OCLC, RLIN, BroDart, Baker & Taylor) – problems with these included the cost of telecom, limited functionality, impossibility of modeling Harvard’s complex organization           

·       Turnkey minicomputer systems, of which GEAC looked the most promising. While there was good functionality, there were problems with fund accounting, lack of MARC in acquisitions modules, weak search, and concern with GEAC’s custom hardware.

·       Systems from other institutions, notably

o   The Washington Library Network’s system. Its chosen DBMS was a big negative,

o   UCLA (concern about the programming and lack of documentation),

o   Northwestern University’s NOTIS.

 The committee’s investigation identified NOTIS as the best candidate. A contract to purchase the system was signed in 1982. The strengths of NOTIS included the following points.

·       It had been designed and written by a university (Northwestern) not for sale, but for its own use.

·       It had been operating successfully both at Northwestern and at a remote site, the National Library of Venezuela.

·       It appeared capable of scaling up successfully to the sizes Harvard would require.

·       It ran on an IBM mainframe, thereby fitting comfortably into Harvard’s computing culture and environment.

·       Purchasing the system would include access to its source code. This was an invaluable feature because we knew that we would want to add a fund accounting module, which NOTIS-3 lacked. It also seemed likely that the Harvard library’s complicated structure would require further modifications to the system.

·       Its programs were carefully written to minimize the amount of program storage used. (This is an issue which was fading rapidly in significance, but seemed important at the time.)

·       It was an integrated library system encompassing functionality far beyond acquisitions. Thus it could serve as a home for other applications when we chose to automate them.

Three to get ready

 

In the spring of 1983, with the NOTIS software installed on a mainframe at the Harvard Computing Center and the Pew grant in hand, we needed a name for the prospective system. Cilla and Charlie independently hit upon HOLLIS, the Harvard On-Line Library Information System, the acronym appropriately recalling the family name of the library’s most generous eighteenth century donors. Cilla’s vigorous promulgation of the name sold the idea. The Online Acquisitions Planning Committee was renamed the HOLLIS Implementation Planning Committee. The project assumed a new level of reality as more people beyond the IPC and OSPR became involved in getting ready for implementation. Work that needed to be done can be grouped under a few large headings, but these are not silos. There are many interactions required between areas.

·       Specification and documentation

o   IPC set up working groups to develop and document detailed specs for needed or desired changes to NOTIS code, tables, etc. Cilla chaired a group on serials, Tracey one on monographs. Surely there were other groups at this stage. (Delegates from the monograph and serials groups made a site visit to Northwestern 20 April 1983 to see the system in action.)

o   Specify and populate tables defining ordering units, receiving units, accounting units, etc.

o   Later: Prepare user documentation

·       Education and training

o   First task: Train the working groups

o   Later: Train tech services staff

·       Physical infrastructure

o   Select and acquire hardware, primarily terminals

o   Plan and install terminal network spanning several buildings

·       Administrative matters

o   Reconfigure OSPR into a Development group (Cilla, with Jon Rothman and me) and a Production Services group (Tracey, with most other OSPR staff.)

o   Determine which library units intend to participate from production day one.

o   Negotiate charges/assessments to participating library units.

·       Computing activities

o   Set up test partitions for use by working groups and by OSPR.

o   Plan storage requirements etc. for production system.

o   Program new functions; e.g. fund accounting.

o   Program modifications requested by working groups, etc.

o   Plan and program conversion and migration of CAPS, CONSER and Local MARC bibliographic data into a single MARC-tagged HOLLIS bibliographic data file

o   Reformat CAPS order data into a NOTIS-compatible format for the HOLLIS order file

o   Set up routines for batch product generation; e.g. order and claim forms, as well as other batch data transfers, and regular reports such as one reporting fund balances and outstanding encumbrances.

o   Work out interfaces with affected non-library units; e.g., Comptroller’s Accounts Payable and Vendor file

o   Set up production partition

o   Test, test, test

·       Data preparation

o   In part, OSPR work as mentioned variously above under Computing

o   In part, user work; e.g. organizing serial check-in data for optimal handling when HOLLIS comes up, initializing fund file.

 

Feedback from the working groups led shortly to modifying the NOTIS that we had received from Northwestern.  A memorable early decision was to eliminate the separate copy holdings record in favor of a new repeatable field in bibliographic records. Its numeric tag would be 995. It would commonly be known and displayed as Location (LOC). That field, with its numerous subfields, served us well throughout the life of the system. Also, probably at this time, the NOTIS-3 volume holdings record was replaced by a record in the MARC Holdings format. Further divergence would come later.

 

It is good to remember that at this point Harvard was preparing to implement only NOTIS acquisitions, cataloging and serials functions; i.e., only technical services staff functions. Circulation and a public access catalog awaited NOTIS-4.

And four to go!

 

HOLLIS began production on 1 July 1985. (The choice of a fiscal year boundary simplified several operations, notably fund file initialization.) The Harvard Librarian issue of that September reported that “On Day One, there were 32 administrative units in the HUL system actively participating in HOLLIS, using 132 terminals; other library units may join HOLLIS later…Like the DUC, HOLLIS is intended to bring to the University Library some of the advantages of centralization, while preserving the relative autonomy of individual units that characterizes Harvard’s library system. Participation in HOLLIS is voluntary, and decisions on whether to participate and the degree of participation are made by individual units. Generalizing broadly, most of Harvard’s larger libraries except the Business School’s Baker Library, which already had a computerized acquisition system in place, are now ordering materials through HOLLIS…”  

 

The same source recorded initial file sizes. The Bibliographic file contained 412,756 records, of which 73.9% were monographic. “About 30% of these were…in HOLLIS terminology, standard cataloguing records” the balance being “provisional.” The Order File contained 101,736 records, of which 40.5% were for monographs. The holdings file contained 81,614 records. A modest beginning.

 

At the opening ceremony, Professor Handlin, who had recently been succeeded as HUL Director by Professor Sidney Verba, characterized HOLLIS as the product of “massive consultation.” He was right, but we believe that extensive involvement of library staff in design and implementation decisions was an important part of the ready acceptance of HOLLIS. Change is hard, after all, but especially if it is into the unknown.

But wait, there’s more!

 

After the 1985 HOLLIS launch, we shipped our specs and code for the fund accounting and invoice generation modules off to Northwestern where they became known as “the Harvard enhancements” until their incorporation, in revised form, in NOTIS-4.  It would have made little sense to retrofit the canonical NOTIS version into our working system; so fund accounting became one more NOTIS/HOLLIS divergence.

 

However, NOTIS-4 did contain an updated circulation module and revised OPAC code both of which we wanted to incorporate into HOLLIS.

 

The primary focus after the initial HOLLIS launch was on the complex task of bringing HOLLIS out of the library’s back rooms to serve library users directly with a public access catalog. Achieving that goal would require several sorts of development:

·       Evaluation the NOTIS OPAC (LUIS) module for suitability:

o   Would its guide, index, record display hierarchy work for Harvard?  The answer was affirmative, but not without some modification to guide screen functionality and customization of screen display features.

o   Test its scalability.

·       Construction of the bibliographic file. The HOLLIS of 1985 was essentially an acquisitions system. A substantial majority of its bibliographic records were imported from CAPS, not rich enough for effective public use. The BMF had good records from OCLC and RLIN-based cataloging for all Harvard libraries in the Day One file, with others being added regularly. A project to merge the BMF into HOLLIS, which we called Database Unification (DBU). would have to

o   Specify how to identify duplicates between the two files and program the specification. 

o   Specify how to resolve duplicates and program the specification, which must choose one or the other record to be retained, or to create algorithmically a hybrid from the two. That was pretty interesting, especially establishing rules for hybrid formation which involves examining several fields, fixed and variable, and deciding algorithmically what says and what goes.

o   Actually load the records.  All the off-line duplicate identification and resolution work just mentioned equips each incoming record with a header describing whether the record should simply be added, or should replace its mate in the HOLLIS file.

·       Maintenance of the bibliographic file. Once built, the file would need to be updated with new cataloging data from OCLC, RLIN and other possible sources.  Tactics for duplicate detection and resolution are very similar to those of DBU.  The operational difference is that DBU can be thought of as a onetime event, but updating goes on more or less continuously. The NOTIS-4 solution to the actual loading of new records into the system was to run them in as a batch job with the system down and then regenerate the indexes.  This approach did not promise to scale well for Harvard; so instead, we placed the prepared incoming records in a transient data queue so they could be merged into the bib file while the system is running. In this way indexing gets done just as it would if the record had been created/modified by a terminal operator, thus obviating the need to regenerate the indexes. Note that our invention of the LOC field to replace the copy holdings record simplified this operation considerably.

·       Expansion of the terminal network to locate terminals in library public areas where card catalogs used to reside, etc.

·       Preparation of user documentation.  Training of library staff, too            

 

The HOLLIS OPAC became available, cautiously billed as “a test version,” on 6 September 1988. Staff members at a few units reported watching students come in and begin to use the new catalog without apparent difficulty. The official OPAC launch took place on 29 September amid pride, joy and relief. As in the 1985 debut, those emotions were shared by a large number of staff members who had played roles in specifying particular parts of the functionality. As least half a dozen small working groups had been formed—and given catchy acronyms by Cilla—to deal with specification particulars ranging from screen layout to duplicate detection and resolution rules. In the Director’s annual report for 1988-89, Professor Verba referred to the creation of the catalog as a “gargantuan effort.” The microfiche DUC could safely be retired.

 

The HOLLIS OPAC had several interesting features, some of which were definitely Harvard innovations. Others may have been present in a NOTIS-4 release. At this point we are not clear about their provenance, so we’ll simply talk about them without implying a claim of responsibility.

·       The first screen presented to the catalog user we called the “catalog selection screen.” (“Catalog” in this case = NOTIS “Institution.”) The user could choose in which catalog to search.  At first there were two options.  The primary one was HU, the Harvard Union Catalog, containing about 1.7 million records when the HOLLIS catalog went live.   

·       The alternative was OW (Older Widener) which we built by converting the 1.2 million brief records created by the Widener shelflist conversion project,

·       Among our specified indexes was a Widener classification call number index which was able to retrieve records from both HU and OW with a single search, marking them to show clearly from which file each record had been retrieved.

·       Later on, a number of other “catalogs” were added to the selection screen.  These included commercially available periodical indexes (e.g., Academic Index, Legal resources, Psychological Abstracts, PAIS ), local resources (e.g., a file of Unitarian pamphlets, materials on course reserve in Lamont library), and a Library Guide providing information on individual Harvard libraries in locally generated records created in a custom MARC format with a set of tags defined to hold pertinent data, e.g., location, hours, loan and access policies.

·       We modified the ABEND handling routine so that after generating a transaction dump for analysis, it did not display the ABEND message but returned the user abruptly to the catalog selection screen.  Fortunately, public catalog ABENDS proved to be few.

·       We developed methods to access the catalog from remote terminals by dial-up connection.

·       We did not immediately activate the NOTIS/BRS keyword search capabilities, because of concern that the volume of such searches would overtax the system. That suspicion led to extensive capacity testing which indicated that searches would become intolerably slow when more than about two dozen queries were in process at once.  A single search request with many terms could also take a very long time. With this knowledge, we activated keyword search with limitations. Only author, title, and subject words would be indexed. Only HU would be indexed, and that only in catalog mode, not in tech services mode. A limit of twenty-five simultaneous searches was imposed.

 

Aside:  One the documents examined as we prepared this report is a paper I wrote giving a fairly detailed description of LI790BAL, the central program for processing catalog mode requests, as it existed in HOLLIS just before the public catalog debut Unfortunately, differences from the distributed NOTIS version are not clearly identified. The smallest certain difference was its name, for we had developed a variant naming convention. Reading it now I am impressed by a number of things, first and least by my ability to understand it moderately well twenty-six years later. But more important, it is possible from this distance to admire the program’s complexity and observe that, not only did it work, but it was constructed so well that modifying it successfully for our needs was fairly easy. In the heat of battle you don’t have time to appreciate things aesthetically.

 

Meanwhile, while catalog development specification and development were underway, planning for circulation implementation began, too, for the Palmer system in Widener would blow up in late autumn 1989 when the due date for extended loans would fall beyond day 9999. Hence circulation presented an urgent deadline. We added another person, Caren Smith, to the OSPR Development group for the circulation project.

 

There was a great deal of interest from libraries other than Widener to participate in the new system; thus interest in participating in the planning. As in other applications, local unit autonomy enhanced complexity of the job. Loan and fine policies varied widely across campus. Some variation could be preserved, some required compromise. As the application that has the most direct effect on individual users, circulation raises more sensitive practical issues than others. So, social matters were more complicated than technical ones and building the necessary tables was challenging. Technical issues centered on equipment:

·       Extending the terminal network, both terminals and wiring,

·       Acquiring barcode readers,

·       Acquiring barcode labels.

 

Widener began HOLLIS circulation in July of 1989. Several other libraries followed soon thereafter. Implementation went fairly smoothly. Memorable little software glitches arose from the difference between NOTIS-3 and NOTIS-4 specifications of the fullword Julian forms for date and time variables. The date bug arose in December when it assigned due dates earlier than the charge dates. The time bug was similar but did not surface until a few years later when we began to use the system for hourly loans of reserve materials near midnight. Hard to find, easy to fix.

 

In the fall of 1987, the HOLLIS network included 175 terminals, serving the acquisitions and cataloging functions. Two years later, after addition of the OPAC and circulation, the network had grown to a total of about 450 terminals, including 145 for the catalog, over 60 for circulation, the remainder attributable to growing participation in the tech services applications.

But wait, there’s a whole lot more!

 

While the HOLLIS catalog, covering post-1976 cataloging, was able to satisfy a significant fraction of user requirements, the continuing need also to consult the pre-1977 Union Catalog on cards for a complete search was an annoyance both practical and strategic. Planning commenced for the conversion of the big catalog and its integration in HOLLIS. A cogent presentation by the libraries convinced university administration to fund the project, which we called, unimaginatively, RECON.

 

To effect such a massive conversion in a reasonable length of time, required mobilization on many fronts. The most obvious need would be getting the actual data conversion accomplished. The method chosen was to contract with OCLC to search its database, claim matching records for the appropriate holding codes, and key unfound records. OCLC worked directly from the catalog cards, which were shipped in large batches managed at the Harvard end by a team also responsible for quality assurance checking of OCLC’s keying/claiming operation.

 

But interesting modifications were also needed to OIS operations and NOTIS/HOLLIS software and database management:

·       Tuning of front end processing (several steps culminating in actual data loading) to minimize human intervention:

o   Take advantage of job scheduling software,

o   Receive data via FTP instead of on tape as had been the previous practice,

o   Use VSAM key sequenced data sets instead of transient data queues for better error recovery capabilities,

·       Automatically start processing input queues twice daily to keep them at a reasonable size—queues, plural, because we established one for the OCLC and RLIN cataloging (etc.) destined for the HU catalog and a second for records going to all other “catalogs.”

·       Segmenting the HU bibliographic file. RECON would exceed limits on the size of a single VSAM cluster. So we divided the bibliographic file into two physical files treated logically as one. Batch and online I/O programs were modified to test the key (HOLLIS record number) of a record being processed to see which file it resides in.

·       Improving index structure and maintenance. NOTIS index maintenance had been seriously resource intensive even of files of pre-RECON size. The method of splitting index blocks limited the number of records with similar index entries that could be added between index regenerations. To minimize the number of costly and time-consuming index regenerations, we

o   Wrote new programs to deblock and reblock an index without having to regenerate it from the bibliographic records. 

o   Doubled the length of index blocks and of index block keys.

o   Doubled the number of NOTIS-type (equal-key) block splits to 16.

o   Invented a new type of split in which the new lower block gets a new key representing the true highest key in that block if it is not the same as the old key. This type of split could be done as many times as necessary.

o   Created for online index update a new facility to limit index changes to fields known to be affected. This was used in connection with the newly designed field update transactions described below.

·       Heading maintenance. New heading maintenance techniques were introduced that required modified record duplication and resolution routines focused on the heading in need of work. We designed a field update transaction to be applied during normal online update procedures. A field update transaction record contained the HOLLIS record number and images of a field to be deleted or inserted. A replacement was represented in the transaction record by a delete/add pair. Variations could accomplish heading splits, etc.  Transaction records were created in two ways:

o   We altered Gary Strawn’s NOTIS-5 Global change program to output field update transactions instead of complete replacement records.

o   We received from OCLC, as part of the RECON contract, periodic files of field update transactions for changed headings.

 

RECON also provided an opportunity to address with new automated procedures two other situations resulting from our strategy for conversion to HOLLIS.

·       Numerous unlinked items had been created, particularly in Widener, to circulate books not yet represented in the HU database. We wanted to have those items linked whenever possible. Monthly we unblocked the call number index and a program searched it to find call number duplication between an unlinked item and a bibliographic record. When an unambiguous match was found, the program generated a transaction to emulate in background the online item linking operation. Transactions for charged books and other complicated situations were then discarded.

·       As RECON progressed, brief records in the OW database became redundant. In the front end of processing of a RECON batch, incoming records with Widener classification call numbers were searched in the call number index.  If an OW record was found, a transaction to delete it was inserted in the non-HU queue.

  

The OCLC/Harvard RECON contract ran from September 1992 through December 1996. In that interval OCLC claimed or keyed over 4.54 million records for us and added new holdings to an additional 133 thousand. Looked at another way, the contract achieved a throughput of just under three thousand records per day, every day, throughout the contract period. Local RECON operations and the transactions generated by processes described above added non-negligible amounts of data to the input queues.

 

The reader of this section has suffered through a lot of detail, but RECON more than any other project shows the challenging kinds of extensions to NOTIS capabilities that Harvard could successfully implement. They testify again to the basic soundness of NOTIS and to our satisfaction in having chosen it.

 

Entr’acte: A few notes before going on the final chapter of Harvard’s NOTIS-era story.  HOLLIS generally ran well even at the furious computing pace set by RECON activity. Though the processing was automated as much as we could manage, things surely would not have gone so well had we not had an excellent manager of production operations on staff. Linda Marean managed to remain calm, or at least to retain the appearance of calmness, even during a few events that were pretty tricky and threatened to become emergencies. After Cilla left for the University of Chicago in 1993, Linda and I worked quite closely together. We consulted frequently on operational matters. When Linda announced her intention to retire, I so feared the prospect of being asked to take over her responsibilities, even temporarily, that I signed up for a stress reduction course. Fortunately for all concerned, Linda’s successor was hired quickly, avoiding a gap.

 

 One task I assumed after Cilla’s departure was that of broadcasting messages to staff users in the event of service disruptions, planned or otherwise. I attempted to explain, as much as possible, the cause and expected duration of the disruption. Frequently it was possible to tell the story with a little humor. I cannot remember any specifics, but even into my retirement I continue to encounter people who remembered those message or who knew me only through them—an odd kind of fame. One message I do recall grew from my frustration with the number of staff users who would call in saying, “I see the word ABEND on my screen. What does that mean?” I posted a long message that began “When the word ABEND appears on your screen it does not imply that the sun is setting over the Rhine.  What’s going down is your job!”  I went on to explain that the numbers or letters after the ABEND word were important and named common ones that were inconvenient but not dire. I did collect transaction dumps until the end of NOTIS /HOLLIS—quite a lot of shelf space was involved—and found time to investigate and eliminate a few frequently occurring ones.

The times they were a-changin’

 

Questions began to arise as early as 1989 at Harvard about the inclusion of electronic resources in our collections and how they could be accommodated in HOLLIS.  Birth of the World-Wide Web, which happened that same year, would in a very short time recast those questions into a general consideration of the future of information transfer.  Something big was happening. In retrospect the ’90s appear as a revolutionary decade.

 

In November 1993, Dale organized a meeting at Harvard for heads of automation at eight large academic libraries to talk about the status of their “local systems” and what each institution saw as likely evolutionary prospects for its own system.  The discussion could be characterized by

·       Recognition of the general computing trend of the time to move away from using a single powerful mainframe computer in favor of a cluster of linked computers, UNIX servers for example, in various client /server configurations.

·       Assessment of the library system marketplace as affected by the hardware trend. 

·       General perception that only NOTIS Inc. (Horizon) and Innovative Interfaces were currently the only offerings of interest to very large libraries.

·       Skepticism about the ability of UNIX systems to handle inevitably computer-intensive operations (e.g., indexing) for very large files with acceptable speed.

·       Exploration of the conceptual possibility to dis-integrate systems by acquiring and linking specialized modules, perhaps from different vendors, and building other necessary elements.

·       Inability to identify any areas in which collaborative development would likely be fruitful.

 

At the meeting, Dale stated that “Harvard would prefer to buy its next system but a look at the marketplace is not encouraging.” In fact it took nearly four more years before the new generation of vendor products would mature enough to permit a productive comparative evaluation of them to begin. Even then our requirement that a candidate system must have a large installation in operation had to be relaxed.

 

Meanwhile local needs and expectations that the NOTIS framework could not easily satisfy were multiplying, largely because of excitement about the World Wide Web, the potential of which was being exploited globally with remarkable speed. The proliferation of personal computers, especially when they served as internet nodes, was also affecting the ways research and information seeking would be conducted.   

·       Users began to discover that many information resources, including the sorts mounted as alternative “catalogs” in HOLLIS, could be more effectively be addressed directly via the internet than through HOLLIS. Complementarily, maintenance of the alternative catalogs was burdensome for OIS staff.

·       More generally, new digital library resources were appearing that did not fit well into the transaction processing oriented CICS environment. Web interface and Z39.50 connectivity were necessary but were lacking from HOLLIS.  

·       Users of the catalog wanted to be able to copy bibliographic data directly from screen displays into their individual work environment for use in citations and bibliographies. The MVS/CICS environment of HOLLIS presented obstacles to doing this well. 

·       Spurred by developments in RLIN, interest was rising among users and librarians in being able to enter, display, and process data in non-Latin scripts.

 

At the same time Harvard, like many other universities, was interested in replacing mainframes with newer technology. As on other campuses, it would be very costly to be the last customer on the big machine. This prospect exerted a gentle push to migrate, but the pull of increased functionality was the stronger motivation for us to advance HOLLIS beyond its NOTIS framework. In summary, major reasons for replacing NOTIS were:

·       It was simply cumbersome in the digital world; i.e., integrating the ILS with the web and digital content was becoming essential, and mainframe-based systems like NOTIS were not prepared to do that well, if at all.

·       We needed to reallocate technical resources from maintenance of aging ILS software to digital library applications.

·       Non-Latin script support was infeasible in NOTIS.

·       Client-server architectures could make use of burgeoning desktop processing power and functionality in ways a mainframe-based ILS could not.

·       We wanted a commercial relational DBMS to support data manipulation and reporting facilities. A “distributed reporting” function had been available to staff in several Harvard library units since 1994. It was based on periodic extractions of selected HOLLIS data reformatted as a Sybase database and accessed through a desktop query language (GQL, subsequently called BI Query.) This was a popular feature, but suboptimal in several ways.

Dénouement

 

By early in 1997 the Automation Planning Committee had identified nine companies whose products seemed worth at least initial consideration as NOTIS successors. Investigation narrowed the field to the point that a year later Ex Libris (Aleph) and DRA (Taos) were the only remaining serious contenders. Taos was by a small margin the favorite and Harvard signed a letter of intent in 1998 with DRA to purchase Taos on the condition that its installation at UCLA proved successful. Its failure to do so eventually left Aleph, after one more state-of -the-market review, the clear choice. In November 2000 Harvard contracted with Ex Libris to purchase Aleph. 

 

The deliberate pace of the selection process had made it apparent that we would still be operating NOTIS-based HOLLIS when two significant deadlines passed, so the final technical work on that system addressed those issues.

·       Year 2000: All date-dependent operations were reviewed and insured against turn of the century errors.

·       End of support for macro-level CICS: All CICS calls were recoded to use the command level interface.

 

Aleph-based HOLLIS went into production 2 July 2002, with a database of 9.25 million bibliographic records, about 22.5 times as many as were involved in the July 1985 HOLLIS launch. We have not been able to track down reliable numbers for other types of records: orders, holdings, items, funds, invoices, patrons, loans, etc. at the time of migration, but they surely would reflect similarly impressive growth.

Envoi

 

“Massive consultation,” “gargantuan effort,” —those are phrases chosen by two successive Directors of the Harvard University Library to characterize the building of HOLLIS. The present narrative has focused on the activities of OSPR.  On reading a draft of this paper, Cilla commented, “The first thing I remember is that we coordinated battalions of committees who wrote so many specs and did so much reviewing and testing. The iceberg to me is all that committee work under the surface and the coding and development sticking up.”  This is a just observation, even if an iceberg metaphor may feel a little injudicious at Harvard. Professors Handlin and Verba were certainly thinking of the depth and breadth of staff participation when they made their remarks, and of the unifying force the creation of HOLLIS exerted upon the University Library as a whole. In 1976 when the decision to build a Central Serials File was made, Karl Nyren waggishly announced the plan in Library Journal under the headline, “Harvard Plans Cooperation with Itself.”  By 2002 that idea was no longer novel because of the coordinated participation by staff throughout HUL, far too many to enumerate, in the successful creation of HOLLIS from NOTIS.

 

Similarly the employees at NOTIS, far too many to enumerate, can claim credit for the system’s success at Harvard and elsewhere, for they created and supported a good product. One of the items in the questionnaire Jerry Specht sent to former NOTIS clients in connection with the building of a NOTIS historical website solicited recollections of particular NOTIS staff members. Here are a few of ours.

 

Dale says, "Jane Burke and I were colleagues for well over 20 years, starting when she was at CLSI.  In addition to working together on NOTIS, we had long discussions of libraries and technology over dinner a couple times a year for decades.  She was aggressive, and a rigorous thinker, and she could be a tough vendor, which one had to admire although at times it certainly got your back up.  While she showed good judgment of technical developments, she was not a technologist, and so at times would say/promise things she shouldn’t have (but what manager hasn't?).  She was smart, perceptive, full of energy and ideas (she never stopped probing for new ideas).  After Jim and Velma, it was Jane that made NOTIS a success, and she was certainly one of the great contributors to the development of IT in libraries."

Jim Aagaard and Velma Veneziano of course we admired in the design of the system and in the consistency of the code.  Tracey remembers Dale, Cilla, and me being especially impressed by the bibliographic indexing techniques when we first examined them in detail. I was acquainted with Velma before we were NOTIS customers. We both had interests in MARBI at ALA where we would get involved in MARC discussions.

Dale and Cilla mention John Kolman. Dale says, “I believe he did work on the system in the early days (he was one of Jim's students, I believe). Then later he returned as something like VP for development where he was a major player in the evolution of the system. John was not just smart and technically competent, but also wonderfully honest and straightforward. He really was one of the strengths of the company.”

In user service matters we remember Jerry Specht and Randy Menakes as helpful and pleasant to deal with. It must be admitted that our memories are fresher of them than of other helpers, because they, like so many of their customers, eventually migrated to Ex Libris and continued serving in similar capacities.

Finally, we think of Stacy Kowalczyk not so much from her NOTIS days as from her subsequent time as a colleague in OIS, a crucial member of the team evaluating systems as replacements for NOTIS, an exponent of an “everything will be just fine” attitude, and a baker of the most wonderful assortment of Scandinavian Christmas cookies.