MEMORANDUM
TO: James M. Eagen III
Chief Administrative
Officer
FROM: Steven A. McNamara
Inspector General
DATE: January 18, 2001
SUBJECT: Systems Integrator Maintenance Services Can
Be Improved
(Report No. 01-CAO-01)
This is our final report on systems integrator maintenance services within the House. The objectives of this review were to evaluate whether systems integrators were providing office equipment maintenance services to the House, which were: (1) timely, reliable, and of high quality; and (2) in compliance with the terms of delivery and prices set forth in their contracts and agreements with the House. In this report, we found that the majority of the Member and Committee offices are more than satisfied with maintenance services provided by their systems integrator. However, their systems administrators identified several areas where maintenance service could be improved. Specific recommendations were made for corrective actions.
In response to
our October 24, 2000, draft report, your office concurred with our finding and
recommendations. The December 4, 2000,
management response is incorporated in this final report and included in its
entirety as an appendix. The corrective
actions taken and planned by your office are appropriate and, when fully
implemented, should adequately respond to the recommendations. Further, the milestone dates provided for
implementing corrective actions appear reasonable.
We appreciate
the courtesy and cooperation extended to us by your staff. If you have any questions or require
additional information regarding this report, please call Christian Hendricks
or me at (202) 226-1250.
cc: Speaker of the House
Majority Leader
of the House
Minority Leader
of the House
Chairman,
Committee on House Administration
Ranking
Minority Member, Committee on House Administration
Members, Committee on House Administration
Systems Integrator Maintenance Services Can Be Improved
I. INTRODUCTION
Summary Of Finding
The majority of the Member and Committee offices rated the services provided by their systems integrator as more than satisfied. However, their systems administrators provided comments that indicated areas where maintenance service could be improved. More effective communication, implementation, and oversight of the Systems Integrator Services Agreement could have improved systems integrator performance and prevented or reduced the problems identified. Due to the lack of enforcement of the agreement terms and conditions, the House of Representatives (House) may have paid for a higher level of maintenance service than it actually received and, with better oversight, problems could have been avoided or corrected earlier.
Background
As of August 2000, there were 12 integrators under agreement to provide systems integrator services to the House. The agreement, at a minimum, requires the integrators to provide computer hardware and software support plans based on time and materials pricing, a la carte pricing, or a minimum number of hours of monthly support. In addition to those required support plans, the integrator may offer other types of support plans such as a comprehensive plan, which includes unlimited hardware, software, and integration services. The agreement, issued and administered by the Office of Procurement (OP), assigns the Contracting Officer (CO) with the overall responsibility for administering the agreement; however, the CO delegated certain other responsibilities to authorized representatives – the Contracting Officers Representative (COR), Contract Administrator (CA), and Technical Service Representatives (TSRs).
The agreement provides the master framework to enable Members, Committees, Leadership, Officers, and other offices of the House to purchase systems integrator services. The agreement calls for the integrator to provide equipment technical support and software support services to each House office that has purchased services under this agreement. Technical support for equipment includes, but is not limited to, systems integration, training, maintenance, and repair. Software support services include software not on the House Supported Software list, which is maintained by House Information Resources (HIR), including the Correspondence Management System (CMS) or other proprietary software system packages. The CMS software is the backbone of the Member office’s daily operation as it provides the capability to keep a record of incoming and outgoing mail, correspond with constituents, maintain the Member’s schedule, and generally meet daily operational requirements.
Under the agreement, integrators are required to provide on-site maintenance service during normal business hours within specified response times. In Washington, DC, the specified response times require a telephone callback within an hour from the initial House office call, an integrator determination within four hours if an on-site diagnosis is required, then four additional hours to arrive on-site, and a total of 18 business hours to resolve the problem. The Congressional district office response times are similar with the exception that the integrator has 12 business hours to provide an on-site response. If the integrator, within 18 hours, fails to (1) deliver the parts or perform the service to restore the equipment or software to full operational status, (2) provide replacement/loaner equipment of equal capability or functionality to the House office until the equipment is repaired, or (3) receive an extension, in writing, from the customer, the integrator, in place of actual damages, must pay to the House a fixed and agreed amount for liquidated damages for each business day of delay the equipment or software was not operational or functional.
In addition to meeting response times required under the agreement, the integrator is required to provide performance summary reports, acceptable in content and format to the COR. The performance summary reports are required monthly and should pertain to the services rendered the previous month. The supporting details of the summary information should be provided to the COR, upon request. According to the agreement, the format and details of the performance summary reports should be agreed upon in the Agreement’s Administration Plan between the House and the integrator.
Objectives, Scope, And Methodology
The objectives of this review were to evaluate whether integrators were providing systems integrator maintenance services to the House, which were: (1) timely, reliable, and of high quality; and (2) in compliance with the terms of delivery and prices set forth in their agreements with the House.
To determine the level of satisfaction with the timeliness, reliability, responsiveness, and quality of service, we selected a sample of 62 Member offices and 11 Committee offices. The sample was developed from a listing supplied by Office Systems Management, which identified each Member and Committee’s systems integrator and plan selected. Using this list as the basis for our sample, we compiled a list of the prominent integrators, (integrators with the most clients), and types of plans provided. Prominent integrators were subsequently classified into large and small categories based on the number of Member/Committee offices using their integrator service agreements.
We then selected a representative sample of integrators based on the percentage of the types of maintenance plans currently in place for both the Member and Committee offices. After selecting the sample, we interviewed each office’s systems administrator using a standard questionnaire that solicited responses to time on the job, experience, selection of the current integrator, knowledge of the contract terms and conditions, integrator’s timeliness, reliability, responsiveness, overall quality of service, satisfaction with maintenance service, record keeping, quantity of maintenance requests/visits, problems with the integrator, and what changes, if any, these systems administrators would recommend. We asked the systems administrators to quantify their answers pertaining to their integrator’s timeliness, reliability, responsiveness, and overall quality of service with a numerical score from 1 to 5, with 5 being the highest or extremely satisfied, 4 being very satisfied, 3 being satisfied, 2 being less than satisfied, and 1 being the lowest or dissatisfied.
We also asked for a verbal explanation of their rating to provide additional insight into their survey responses. Some offices lowered their ratings after verbalizing problems with their systems integrator, but most did not. We were unable to independently confirm the systems integrators’ non-responsiveness because the Member offices, Committee offices, and OP did not maintain sufficient records of systems integrator performance. In addition, we did not verify the billings and payments for computer equipment maintenance for two reasons. First, we reviewed the billing and payment process and determined that the internal controls were sufficient to conclude that the risk of overpayments is low. Second, we learned that to retrieve the cost data needed for the analysis from the Financial Management System would take as long as four months and would be very labor intensive.[1] Therefore, the cost to analyze the amounts billed and prices paid would likely far exceed the benefit of finding potential payment errors.
II. RESULTS OF REVIEW
The majority of Member and Committee offices were generally more than satisfied with the maintenance services provided by their systems integrator agreement. However, their systems administrators provided comments that indicated areas where maintenance service could be improved. More effective communication, implementation, and oversight of the Systems Integrator Services Agreement could have improved systems integrator performance and prevented or reduced the problems identified. Due to the lack of consistent systems integrator performance and enforcement of the agreement terms and conditions, the House of Representatives (House) may have paid for a higher level of maintenance service than it actually received and, with better oversight, problems could have been avoided or corrected earlier.
Survey Responses
The following figures and paragraphs describe the maintenance concerns the systems administrators identified as preventing their giving the integrators higher ratings as well as the internal control weaknesses we identified.
Figure 1 represents the systems administrators’ overall numerical rating of the systems integrators’ timeliness, reliability, responsiveness, and overall quality of service. A numerical rating of 5 means extremely satisfied, 3 means satisfied, and 1 means dissatisfied. The individual ratings of each integrator are displayed horizontally and listed vertically from the largest to the smallest integrator. The columns provide comparison of each integrator’s scores. The overall averages represent the total numerical score for each category divided by the number of Member and Committee office systems administrators interviewed.
Offices
|
|
|
|
Overall |
|
Integrator
|
Visited |
Timeliness |
Reliability |
Responsiveness |
Quality |
|
|
|
|
|
|
|
|
1 |
45 |
3.62 |
3.84 |
4.00 |
3.82 |
|
2 |
18 |
3.58 |
3.42 |
3.64 |
3.31 |
|
3 |
5 |
4.20 |
4.20 |
4.20 |
4.10 |
|
4 |
4 |
4.56 |
4.00 |
4.50 |
4.38 |
|
5 |
1 |
4.00 |
4.00 |
3.00 |
4.00 |
|
6 |
1 |
5.00 |
4.50 |
5.00 |
5.00 |
Overall Average |
74[2] |
3.73 |
3.78 |
3.95 |
3.76 |
Figure 1 Numerical Rating of Integrators’
Overall Quality of Service
Overall, Member and Committee offices on average rated their integrator’s timeliness, reliability, responsiveness, and overall quality of service at 3.73 or higher. However, 7 of 74, or nearly 10 percent, did not rate the systems integrators as average, or satisfied. Our analysis revealed there was no correlation between integrator satisfaction, systems administrator experience, and type of maintenance plan selected. During the systems administrator interviews, we identified concerns that, if corrected, would enhance the level of overall service beyond the current ratings.
We found that the numerical scores provided by systems administrators appeared to be higher than expected considering the verbal comments provided. This could occur because systems administrators’ expectations of the systems integrators were not based on agreement requirements or that the individual incidents related were not significant to the office. However, the individual incidents taken as a whole demonstrate problems with the systems integrators’ performance under the terms of the contract. The subsequent figures depict concerns of the systems administrators and areas where integrator performance pertaining to timeliness, reliability, responsiveness, and overall quality of service could improve.
Timeliness of Service
Overall, timeliness of service received a numerical rating of 3.73, indicating offices were more than satisfied with their service. According to the agreement, timeliness means responding to the initial call within one hour, making the determination whether to go on-site within four hours, arriving at the office to fix the problem within eight hours, and fixing the problem within 18 business hours. Member and Committee offices’ concerns primarily focused on their integrator’s not meeting the initial one-hour callback requirement, but also expressed concerns about their integrator’s not meeting the 18-hour requirement to correct the problem. We found the most critical period is the first hour after the problem arises, when systems administrators either attempt to fix the problem or call their integrator’s help desk.
Figure 2 Concerns with Integrators’ Timeliness of Service
Systems administrators expressed concerns about their
integrator’s delay in answering the initial phone call and getting back to them
within the hour. Specifically, offices
stated they were placed on hold for long periods, often causing them to miss other
calls, or be tied to their desks waiting for a technician’s response. Other systems administrators stated they
used their office speakerphone capabilities so other work could be accomplished
while waiting for a response.
Additionally, they expressed concerns about being passed from phone menu
to phone menu, and not being able to speak to an actual person, and either
having to leave a voicemail message or call back again to speak to a
technician. A few offices stated the
calls were never returned, and they had to start the process all over
again. Furthermore, one knowledgeable
systems administrator stated the integrator told him the agreement response
times were only guidelines and were not strict response times. To avoid the frustrations experienced in
getting calls through to the help desk, some offices bypass the help desk and
call directly to a specific technician.
Again, the first hour is critical; however, some offices stated that the problems are not fixed or repaired within 18 business hours. Over 40 percent of the systems administrators interviewed disclosed system problems that went beyond the 18 hour response time stipulated in the agreement. For example, one office stated a computer continues to freeze up and the integrator has not been able to resolve the issue. Other offices have had printer problems including error messages and strange characters in documents that have lasted upwards of 3 months. Timeliness of service is crucial to both Member and Committee offices; therefore, the integrators should be required to meet the agreed upon response times.
Reliability of Service
Overall, Member and Committee offices, on average, rated the integrators’ reliability at 3.78, or more than satisfied. We defined reliability as dependable and consistent service by correcting the problem the first time without having to make repeated visits to address the same problem. Twenty-six respondents (35 percent) of the systems administrators questioned the competency of the technicians working for the integrators. The technicians’ inability to fix problems with the CMS and the offices’ hardware resulted in some problems that were never fixed and some questions that were never answered. One Member’s office had problems with their CMS program crashing when they tried to print documents. The problem was called in on a regular basis to the integrator and took a year and nine months to resolve.
To address problems with hardware and software, some offices were encouraged by the integrators’ sales representatives to purchase the latest hardware and software. However, after several offices purchased high-speed desktops, laptops, fax modems, scanners, digital copiers, or the latest version of CMS, the integrators’ technical staff could not get the equipment to work properly. As a result, the offices either returned the hardware or software after several months of waiting for the integrators to get it working, found ways to work around the problem, or did not use the problem-causing hardware or feature in the software package.
Figure 3 Concerns with Integrators’ Reliability
Responsiveness of Service
Interviews with the systems administrators revealed that overall, they rated responsiveness at 3.95, or more than satisfied. Responsiveness for the purpose of our survey was defined as the willingness and ability of the integrator to address the repair, maintenance, and technical and procurement assistance needs of the office.
Figure 4 Concerns with Integrators’ Responsiveness
Systems administrators gave several reasons for not rating their integrator higher than they did. They noted that once the integrators were made aware of an office’s need, it often took the integrator too long to respond and address that identified need, or the integrator fixed the problem but did not provide an explanation of the fix. Several offices noted problems with their CMS packages. Problems identified with printing, using correct salutations, inserting signature blocks, and receiving error messages when attempting to utilize a feature of the software were noted as taking several weeks to over one year to correct. Some systems administrators remarked that they have given up on expecting a resolution or have learned to live with the limitations of the software. This may be due, in part, to some offices noting that integrator call desk personnel are not always qualified to answer the technical questions and do not listen to them as they describe the problem. These offices stated they phone the call desk, explain the problem and their attempts to fix it, then get passed to another technician, and have to run through the problem and attempted fixes over again. This, they remarked, takes considerable time away from their more important duties and responsibilities.
Quality of Service
Figure 5 Concerns with Integrators’ Overall Quality of Service
Quality of service encompasses timeliness, reliability, and responsiveness. Systems administrators rated the quality of service at 3.76, or more than satisfied with the overall service, and expressed concerns with issues they thought could improve the overall quality of service. Systems administrators stated they wanted better answers for longstanding questions pertaining to software (CMS) and hardware. Some of the issues, as explained in earlier paragraphs, demonstrate the lengthy time it takes to resolve some problems.
Additionally, local offices (Washington, DC) were concerned with the maintenance services provided to Members’ district offices. According to the agreement, integrators or their designated subcontractors are to provide the same maintenance service, with a slightly longer on-site response time of 12 hours to the Members’ district offices. Offices stated the integrators take longer to respond and initially use remote maintenance capabilities to solve problems in the district offices. However, if that fails, the primary integrator uses subcontractors that do not have the same CMS knowledge to fix the district office problems. Therefore, it takes longer to fix the problems in the district offices.
Furthermore, offices stated that district office staff has had to box up hardware (desktops and laptops) and return them to Washington to be fixed, without loaners from the integrators. According to the agreement, systems integrators are required to provide a loaner, ask and obtain an extension of time to resolve the issue, or pay liquidated damages. When hardware fails in Washington, other equipment exists to potentially alleviate the problem. However, district offices do not normally have this capability, thereby disrupting the district offices’ operations. Integrators’ and subcontractors’ not fixing problems timely, not providing loaners, and disrupting district office operations is non-responsive to the Members and the systems integrator services agreement terms and conditions.
Improving Integrator Performance
While the majority of Member and Committee offices were more than satisfied with the maintenance service provided by the integrators, the systems administrators’ comments indicated that performance needs to improve in order to comply with the requirements, terms, and conditions of the agreement. In addition, the House needs to enforce the requirements, terms, and conditions of its agreement to assure it receives the required level of maintenance service.
Quality of Service
Our analysis of the systems administrator interviews revealed that overall there are three keys issues affecting the quality of service provided by the integrators. First, the integrators were not providing timely responses to Member and Committee office problems and over 40 percent of our sample had system maintenance problems that went beyond the 18 hour response time stipulated in the agreement. Secondly, systems administrators stated their integrator’s technicians did not always possess the skills necessary to adequately respond to their problems or concerns. As such, once systems administrators became confident with certain technicians, they called the technicians directly, avoiding their integrator’s help desk and the time associated with placing the call, waiting for a response, and explaining the problem multiple times to multiple technicians. Finally, systems administrators stated the district offices were not receiving the degree of maintenance service required under the agreement. Problems were not fixed timely, integrator subcontractors did not possess appropriate knowledge of the CMS packages to correct the problems, and district offices had to package and ship equipment back to Washington for repairs.
Enforcing Agreement Terms
The current Systems Integrator Service Agreement includes meaningful performance measures related to response times, and user remedies if the integrator fails to perform. The agreement, signed and agreed upon, by both the CO and Integrators, contains terms and conditions which clearly state the performance measures for which the integrators’ service in both DC and district offices should be judged. Additionally, the agreement sets forth the duties and responsibilities of key agreement personnel including the CO, CA, COR, and TSRs, and requires the development of an Agreement Administration Plan. The following sections identify weaknesses in the internal control process associated with implementing the current systems integrator agreement.
Develop Administration Plan According to the Systems Integrator Services Agreement, “15 calendar days after award, the Contractor and House will sign a mutually agreed upon Administration Plan. This plan is a detailed description of responsibilities of the House and integrator, an agreed upon meeting schedule (usually monthly), performance measurements and progress review meetings, and standardized reporting format. The goal is to keep track of performance, head-off any potential problems, resolve issues, and maintain a smooth operating Agreement.” During our discussion with the CA, we learned that an Administration Plan was never developed for the Systems Integrator Services Agreement. Therefore, the duties and responsibilities of the agreement were never explained, published, or agreed to by the parties, including the systems administrators.
Communicate Roles and Responsibilities Although the systems administrators are on the “front line” to initiate communication with those who are responsible for contract administration when problems arise or the integrator does not meet the requirements of their agreement, they were not aware of the specific agreement requirements or their responsibilities. Through our discussions, we learned that 64 of 73 (88 percent) of the systems administrators did not have a copy of their agreement. These requirements were in the User’s Guide to Equipment, Software, and Related Services (User Guide) and Vendor Product and Service Remedies for Non-Performance located on the CAO’s web page; however, they were hard to find. In addition, the CAO informed us that they had provided offices with a hard copy of the Vendor Product and Service Remedies for Non-Performance. These documents identify the systems administrators’ contract administration responsibilities to record and track systems integrator performance and remedies for problem resolution. However, 62 of 73 (85 percent) were not aware of the established response times, the basic performance measurements stated in the agreement or their contract administration responsibilities. Furthermore, only 31 of 73 (42 percent) of the systems administrators kept any kind of records, and those records were mainly copies of the integrators’ job tickets.
Monitor Integrator Performance Although the agreement requires the COR and TSRs to monitor the systems integrators’ performance, little empirical data was available that would indicate that monitoring occurred. According to the agreement, the COR is responsible for “…determining the adequacy of performance by the Contractor in accordance with the terms and conditions of this Agreement, acting as the CO’s representative in charge of work at the site, ensuring compliance with Agreement requirements insofar as the work is concerned, advising the CO of any factors which may cause delays in performance of the work, [and] conducting such reviews as required…” In addition, the TSRs “are responsible for the day-to-day inspection and monitoring of the Integrator’s work. They will confer with representatives of the Integrator regarding any problems encountered in the performance of the work and generally assist the COR in carrying out his responsibilities.”
Improve Problem Resolution Procedures According to the CA, when systems administrators elevated maintenance problems to the TSRs, written records or emails related to those problems were kept. However, they were not forwarded to the COR to identify recurring maintenance problems or trends related to non-performance of agreement requirements so that proper action could be taken. Only isolated problems that became severe in nature were elevated to the COR, and they were dealt with on an office-by-office basis. Although our review revealed that many offices had similar recurring problems, only one office pursued liquidated damages for a problem that persisted 21 months. The systems administrator stated that the pursuit of damages wasted more staff time than the amount of the damages recovered. Instead of only resolving problems on an individual office basis, the contracting officer should also enforce the contract requirements based on trends or recurring problems within the House as a whole.
Obtain Integrator Performance Reports Our discussion with the COR indicated that the integrators were submitting reports, but the reports were not useful since they did not give sufficiently detailed information to identify overall problems or trends. The reports stated the number of problem calls received and the number of problems closed each month. However, they did not indicate the nature of the problem or how much time it took to close the problem. Therefore, the CA or COR did not have an effective tool to determine if problems were occurring.
Conclusion
Due to lack of consistent systems integrators’ performance and enforcement of the agreement terms and conditions, the House may have paid for a higher level of maintenance service than it actually received. In addition, better oversight could have prevented the problems or corrected them earlier. To ensure that the terms of the current agreement are enforced, a contract administration plan should be written to identify integrator and House responsibilities and effectively communicate these duties. To assist the House, the CAO needs to improve integrator accountability by enforcing and enhancing performance measures, and strengthening liquidated damages clauses in the new contract. Currently, the CAO is in the process of developing a new systems integrator contract, planned for issuance during the 107th Congress, which should address these concerns.
Although offices were more than satisfied, they identified improvements that were still needed to address the ongoing problems that were either not resolved timely or not resolved at all. In order to identify and fix problems quickly, an effective problem resolution process should be established. This process should give the systems administrators a one-call approach to address performance problems that exist beyond the 18-hour resolution time frame. As a result, this should alleviate the current administrative burdens on systems administrators for monitoring integrator performance. A prominent and proactive role by the CAO could better assist in identifying, reporting, and analyzing problems, which should provide for timelier, effective resolution. A simple solution to quickly address problems would be to attach a 3” by 5” label to the systems administrators’ computers identifying the contract performance measures and the number to call when a problem exists.
Recommendations:
We recommend the Chief Administrative Officer:
1. Develop a comprehensive Agreement Administration Plan that:
a. Clearly communicates the duties, responsibilities, and oversight roles of all parties to the agreement,
b. Measures integrators’ non-performance, and
c. Provides meaningful performance reports from the systems integrators.
2. Develop formal problem resolution procedures that identify trends and quickly and effectively resolve the issues among the parties.
3. Ensure Member and Committee offices are informed of the systems integrators’ performance measures and their own duties and responsibilities under the agreement.
4. For the new contract, proposed for the 107th Congress, improve integrator accountability by enhancing performance measures, improving the enforcement provisions, and strengthening the liquidated damages clauses.
Management Response
On December 4, 2000, the CAO concurred with the recommendations. In addition, the CAO pointed out that computer maintenance is only one facet of the computer vendor support model used at the House. To gain a complete understanding of the computer vendor support model used by the House, this audit along with the results of other studies should be considered to form a composite picture of the House computing environment.
A formal contract administration plan will be implemented to clearly communicate the duties, responsibilities, and oversight roles of all parties to the agreement by January 3, 2001. In addition, HIR has taken steps to establish a formal problem resolution process through (1) appointing a Customer Relations Manager; (2) making improvements in problem reporting through the HIR Call Center and TSRs; (3) scheduling monthly meetings with System Integrators to identify outstanding issues, develop courses of action, and track issue resolution; and (4) developing guidelines for when COR coordination with the CO is warranted. The Formal Problem Resolution Process and TSR Standard Operating Procedures will be updated to reflect these changes by January 31, 2001.
In addition, steps have
been taken to provide information to the House community on Office
rights and remedies in dealing with vendors and contractors as well as the responsibilities of House
personnel and vendors. OP will make
available to each House Office a copy of its Systems Integration agreement, and
a quick reference card will also be prepared and distributed by March 31,
2001. Additionally, OP, in conjunction
with
the COR, will provide a series of briefings on the System Integrator
Agreements. Finally, the CAO will
shortly send to the Committee on House Administration a Request for Proposal
which will be a new computer services contract format to replace the present
system integrator agreement that will enhance performance measures, improve the
enforcement provisions, and strengthen the liquidated damages clauses.
Office of Inspector General Comments
The actions taken or planned for the recommendations are responsive to the issues identified and, when fully implemented, should satisfy the intent of these recommendations. Further, the milestones dates provided for completing these actions appear reasonable.
