AUDIT OF NEW SAR INITIATIVES PROGRAM
ENVIRONMENT CANADA
NIF PROJECT – EC 8/97
AWOS INSTALLATION – SOUTH COAST OF NEWFOUNDLAND
November 2000
Table of Contents
Executive Summary
NIF funding of $600,000 was approved in August 1997 for the installation
of two AWOS (Automatic Weather Observation Station) stations in the South
Coast of Newfoundland.
The NIF criteria were met in the following areas:
- A Program MOU was in place.
- The project was consistent with NIF funding priorities.
- The project application generally met the requirements of the Guide
with the exceptions noted below.
- The selection process was appropriate.
- The overall project objectives were achieved.
- Progress reports and the Final Summary Report were submitted on a
timely basis although their content was brief and not very informative.
- The approach to communication was appropriate for this type of project.
Areas where the criterion were not fully met include:
- The proposal/annex did not have a section addressing the reporting
requirements and pertinent information to be released. The proposal
also did not have a specific workplan which was linked to milestones.
It was difficult to assess the different steps required to accomplish
the project and monitor the results as the assignment progressed. In
addition, the original annex was not signed by NSS officials even though
the amended annex had all of the required signatures.
- The quarterly reports were incomplete and very brief. Detailed information
on the project was not included in the quarterly reports and the information
provided was not sufficient for NSS officials to monitor the progress
adequately. Furthermore, the quarterly reports submitted after April
1998 did not conform to the forms contained in the 1998 NIF Guide.
- There was no formal post-project evaluation done.
Overall the objectives of the project were achieved in that the installation
of the AWOS stations was successfully completed. However, there are some
project management issues which were not respected or performed according
to the guidelines.
1.0 Introduction
The New Initiatives Fund (NIF), administered by the National Search &
Rescue Secretariat (NSS) is designed "to provide funding for projects
that will enhance search and rescue (SAR) in Canada." Approved projects
are subject to audit as set out in the NIF Guide and the Memorandum of
Understanding (MOU) agreed between the NSS and the Sponsoring Department,
Province or Territory.
The AWOS Installation – South Coast Newfoundland proposed by Environment
Canada was selected for audit in 2000/01. The project involved the installation
of two Automatic Weather Observation Stations (AWOS) on the South Coast
of Newfoundland to improve the Marine Weather Forecast and Weather Warnings.
The audit, conducted in October /November 2000, included the examination
of the seven criteria set-out in the 1995 and 1998 New SAR Initiatives
Fund (NIF) Guides and covered the following:
- compliance with the MOU;
- proposal application, selection and funding;
- project management;
- progress reporting;
- project claims and payments;
- post-project evaluation; and
- communication of results.
This was accomplished by performing the following:
- Reviewing NSS project records and discussion with NSS staff.
- Consultation and discussion with the project manager at Environment
Canada.
- Review of claims processed.
- Detailed testing of claims.
- Assessing whether or not the NIF project was managed in compliance
with the NIF Guide and the MOU.
There were two different NIF Guides in effect during the timeframe in
which the project was undertaken. Both guides were considered during the
course of the audit as the organization must follow the most recent guidelines
set-out by the NSS.
The results of our review are summarized in this report.
2.0 Project Description
With the closure of the Canadian Coast Guard station in the St. Lawrence,
along with the de-staffing of human observation platforms along the South
Coast of Newfoundland, Environment Canada believed that the installation
of Automatic Weather Observing Stations (AWOS) would strengthen their
forecasting and monitoring abilities thereby providing critical information
on hazards to SAR personnel on rescue missions.
The project proposal was developed by Environment Canada (EC) and approved
by the NSS. The total estimated cost was $632K with 95% of the funding
provided by NIF ($ 600K). Included in the budget was $350K for equipment,
$6K for prototype engineering, $70K for installation, $60K for spare parts
and $4K for transportation. The costs were to be incurred equally over
two years; $300K in F97/98 and $300K in F98/99.
In August 1997, the NSS approved an amendment to the annex with respect
to the funding allocation. Instead of advancing funds equally over two
years, they agreed to provide funding of $370K in F97/98 and $230K in
F98/99. This amendment was made in order to respond to the closure of
the Port aux Basques manned station which meant there would be no manned
observations from Port Aux Basques to St. John's on the south coast of
Newfoundland. In order to have a weather monitoring system available for
that region, it was decided that the preferred location would now be Port
aux Basques (instead of Burgeo) and the Burin Peninsula. It was also deemed
more cost effective to buy most of the equipment in F97/98.
The project was completed on schedule and under budget by $100K due to
cost savings realized on land. Environment Canada was able to use land
already owned at Port Aux Basques and in Marystown, and entered into a
partnership with operators of the municipal airport where the AWOS's were
respectively installed. The level of NIF funding was consequently reduced
to $500K as the $100K surplus was returned to NSS.
All participants view the project as a success. The data from both sites
is used extensively by the marine community and other users of the weather
data, including Search and Rescue.
3.0 Audit Findings
The findings are summarized by the criteria contained in the NIF Guide.
3.1 Program MOUs
Program MOUs are agreed and signed by the participating federal SAR
departments, provincial/ territorial EMO offices and the NSS. The MOUs
are based on the principles set out in the Guide. They are reviewed periodically
and updated, as required.
Findings
- An MOU exists between the NSS and EMO, which was signed on July 30,
1997 and August 6, 1997 respectively.
- The MOU is consistent with the NIF Guide and covered the project
period.
Conclusion
The criterion to maintain an agreed, signed and current MOU was met.
3.2 Funding Priorities
Projects must address the priorities of the NSP.
Findings
Prior to the listing of the six NIF priorities in the 1998 Guide, ICSAR
was stated as saying they were "… not able to agree which areas
of NIF sponsorship are more or less important than others." The result
was that priorities, in the proper sense of the term, did not exist when
the project was selected in 1997. The 1995 NIF Guide simply states that
organizations with responsibilities in the SAR field are eligible for
NIF funding for projects which could enhance the effectiveness of the
NSP and noted a requirement for proposals to meet the priorities of the
National SAR Program.
While it is difficult to conclude categorically that the project was
in line with NSP funding priorities, the projects were reviewed by EC,
the NSS, a merit board and ICSAR, before LMSAR approval. . Given the type
of activity and the objective of the project, the proposal is deemed to
be consistent with the overall intent of the NSP and the NIF as they were
defined in 1997.
We also consider the project met the following 1998 NIF Priorities:
- identification of better information, processes and practices for
the planning, delivery and evaluation of SAR prevention.
Conclusion
The criterion for the project to address NSP priorities was met.
3.3 Project Application
Based on the 1995 NIF Guide:
All project applications must comply with the form and content requirements
set out in the Guide and include, at a minimum, a:
- Project description
- Project budget linked to milestones
- Project work plan linked to milestones
- Project management plan
- Project results/ benefits statement in measurable terms
Findings
- The project application clearly states the project rationale, description
and priority.
- The project budget contained an appropriate level of detail and was
consistent with the NIF format in use at the time.
- The proposal did not contain a project management plan and work plan
detailing the activities that were to be undertaken in each fiscal year.
The only information regarding milestones included in the proposal related
to the amount of funding that would be required per fiscal year. As
such, it is difficult for NSS to properly evaluate the proposal and
determine its relevance and whether or not the timeline is realistic.
- The proposal did not contain information regarding reporting frequency
which makes it difficult to obtain sufficient information to assess
the progress of the report.
Conclusion
The criterion was not fully met. The project application was generally
consistent with the NIF Project format in use at the time. However, we
found that insufficient details were provided on management plans, project
work plans and reporting frequency. Due to the lack of information, it
was not possible for management to properly monitor the successful completion
of the project.
Recommendation
We recommend that when submitting a proposal to the NSS for NIF funding,
Environment Canada should ensure that the application is complete and
contains all of the information required by the NIF Guide. In particular,
Environment Canada should put an emphasis on the project management and
planning phases and ensure sufficient
relelvant information is provided.
3.4 Project Assessment
All project proposals are reviewed by the sponsoring organizations,
the NSS and representatives of the sponsors who make up the NIF Merit
Board, against the stated funding priorities and the program terms and
conditions set out in the Guide.
Findings
Project proposals are normally submitted from EC to the NSS. EC's proposal
submitting process is as follows: after the reception of a NSS call letter,
the EC representative will forward the call letter to the different EC
divisions. Interested parties are asked to discuss their project idea
with the EC representative. After discussion with the representative,
and if required with NSS, EC determines if the proposal meets the NIF
criteria. Interested parties then draft a proposal, and submit them for
review to the EC representative.
The NSS process includes a review and approval of each EC proposal, first
by the NIF program managers, then by the Director Federal Coordination.
Proposals not considered acceptable are either rejected as inappropriate
or returned to the sponsor for amendment. After review by the NSS, proposals
are submitted to the NIF Merit Board.
- The initial proposal did not have the NSS approval signature, however,
the amended proposal was signed by both parties (August 1, 1997 and
August 5, 1997 respectively).
- On September 25, 1997, the project was ranked by the NIF Merit Board
as 11 th out of sixty-six proposals for funding in FY 98/ 99. The Merit
Board rankings were accepted by ICSAR on November 26, 1997 and by LMSAR
under cover of an NSS letter dated November 27, 1997.
- The project proposal/ approval document was subsequently formalized
by the NSS and the EMO, four months after LMSAR approval.
Conclusion
Although the criterion was met, the proposal had very little information
to make an appropriate assessment of the project.
Recommendation
We recommend a more rigorous approach for the review of NIF project applications.
Increased diligence in the assessment of the viability of the project,
the accuracy and detail of the budgets and cash flows and the relevance
of the project will ensure NIF funds are spent appropriately.
3.5 Project Funding, Claims & Payments
Project approvals must be confirmed by signature prior to the start
of the project. The sponsoring organization prepares and submits claims
in accordance with the requirements of the NIF Guide. The NSS reviews
all claims for accuracy and compliance with the project approval. All
payments will comply with the project approval and the requirements set
out in the NIF Guide.
Findings
- The initial Project Proposal Form was signed by Environment Canada
in July 1996 but was not signed by any NSS official.
- The amended annex was signed by both Environment Canada and NSS officials
in August 1997 before the start of the project.
- An invoice for NIF funding of $370,000 representing funds for FY97/
98 was submitted to the NSS by EC on September 5, 1997 and was approved
for payment on September 15, 1997. The payment was made September 25,
1997. Per the last quarterly report of the fiscal year, all of the funds
had been spent.
- A second invoice for NIF funding for FY 98/ 99 of $230,000 was submitted
to the NSS by EC on July 16, 1998. The invoice was approved for payment
on July 29, 1998 and was paid September 28, 1998.
- Per the Final Summary Report, the total amount of funds spent equalled
$500,000. The remaining $100,000 in unspent funds was returned to the
NSS by way of cheques in January and April 1999.
- Funds were expended within the approved project timeframe.
- Based on our review of support for expenses, expenditures were consistent
with the project's goals and objectives. Therefore expenses were incurred
according to the proposal.
Conclusions
The criterion was met.
3.6 Project Reporting
Specific reporting requirements for each project are incorporated into
the approval document. They are tailored to the realities of the project
but, in general, will call for quarterly reports that outline the activity
results to date and the expenditures for the past quarter against approved
plan and indicate the activity proposed for the subsequent period. The
sponsoring organization is responsible for submitting the reports in accordance
with the approved timetable.
Findings
- In all, only five of the six required quarterly reports covering
both the 97/ 98 and 98/ 99 fiscal years were submitted to the NSS by
EC. All of the reports were submitted using the form contained in the
1995 NIF Guide. Some of the information such as NIFID#, summary description
of initiative, milestones, total approved and submitted funding was
missing. The information provided was very brief and not very descriptive.
It was difficult to assess whether the project was on track or delayed
and if it was successful or not. EC should have used the form contained
in the 1998 NIF Guide for the reports submitted after April 1998.
- The Final Summary Report was submitted using the format of the 1998
NIF Guide.
Conclusions
In our opinion, the reporting criterion was only partially met. Although
quarterly reports by the project manager were submitted on a timely basis,
they were incomplete and very brief and vague.
Recommendation
We recommend that quarterly reports be submitted when all of the required
information has been obtained. Reports submitted to the NSS should be
in the format requested in the most updated NIF Guide and should be detailed
enough to make an assessment on the progress of the project.
3.7 Project Monitoring
The sponsoring organization is responsible for monitoring the projects
being carried out by a third party.
Findings/ Conclusions
This was an EC initiative only and it did not involve any third party.
3.8 Post Project Evaluation
Based on the 1998 NIF Guide:
The sponsoring organization undertakes an evaluation of the project's
effectiveness against the objectives established and agreed to at the
approval stage. An initial evaluation should be carried out withing one
year of the project completion and the findings submitted to the NSS for
review and tabling at a subsequent ICSAR meeting.
Findings
- Criteria for evaluating project success were included in the initial
and amended proposals and included gathering statistics on the new stations,
interviewing SAR personnel for remarks on the usefulness of the information
obtained from the AWOS as well as undertaking informal surveys of client
groups to evaluate the effectiveness of the information for the air,
land and marine communities.
- There is no documentation which indicates the evaluation was conducted
by EC.
- As of this date, no findings have been submitted to the NSS for their
review. The Final Summary Report indicates that the evaluation is on-going.
Conclusions
The requirement for an evaluation of the project's effectiveness against
the agreed objectives has not been met. Although EC says the post project
evaluation is on-going no supporting evidence could be obtained to corroborate
that fact.
Recommendations
We recommend that all projects, which are subject to NIF funding, have
a post project evaluation of the project's usefulness and success.
We also recommend that the NSS perform a follow-up on projects for
which no evaluation have been submitted.
3.9 Communication of Project Results
Communicating the results of projects will be done in consultation with
the project sponsor.
Findings
- A formal communication plan was not included in the project proposal.
The Final Summary Report did nonetheless indicate that there were press
releases from local newspapers.
- Correspondance with the NSS was made regarding information to be
included in the press releases.
Conclusion
The communication criterion has been met.
3.10 Overall Project Management
The project must be managed in accordance with the policies and practices
of the sponsoring organization as well as the requirements of the NIF
Guide and the approved management plan set out in the project application.
Findings
- A formal project management plan was not included in the original
proposal.
- EC was able to install the two proposed AWOS stations on time and
was able to do so at a saving of $100,000 over what was budgeted. This
was achieved as a result of using existing land owned by EC instead
of purchasing additional land to install the two stations, thereby saving
installation fees.
- The project manager was involved and informed throughout the project.
He actively participated in the implementation of both AWOS stations.
- The planned activities were carried out.
- The findings noted throughout the audit report generally indiate a
reasonable level of compliance with the NIF Guide.
Conclusions
The overall management of the project appears reasonable for an initiative
of this type. The criterion has been met.
|
