3.0 FINDINGS AND RECOMMENDATIONS

The evaluation report findings with corresponding recommendations are grouped according to the subject headings for the evaluation objectives and core issues to be addressed, as mentioned previously in Section 2.1. The Recommendations have been made only for areas deemed requiring improvement. They are set out in italics for ease of reference and are numbered 1 through 6, starting in Section 3.2 below.

3.1 RELEVANCE

3.1.1 CONTINUED NEED FOR PROGRAM

An assessment was made of the extent to which the program continues to address a demonstrable need and is responsive to the needs of Canadians. There is overwhelming agreement amongst proponents that there is a strong continued need for the SAR NIF program. This is borne out through the responses to the survey questionnaire and follow-up interviews with contribution recipients, Merit Board members and provincial partners that the objectives of the SAR NIF program are being mostly met. More specifically, there is also general agreement that the program is responsive to the needs of Canadians and that the first two objectives of the SAR NIF program as stated below are being met:

• enhance the effectiveness of the SAR response in federal, provincial and territorial jurisdictions;
• promote and support projects designed to develop and improve SAR prevention; and
• share SAR response and prevention best practices throughout the SAR community.

Comments received indicate that more could be done to promote the third objective through an enhanced communications strategy. This is discussed in further detail in a subsequent section 3.2.1 (b) as is the related recommendation.

3.1.2 ALIGNMENT WITH GOVERNMENT PRIORITIES

An assessment was made of the linkages between program objectives and (i) federal government priorities and (ii) departmental strategic outcomes.

The SAR NIF program remains relevant and important to the DND/CF as it contributes directly to one of the three programs identified in the Program Activity Architecture (PAA): "Contribute to Canadian government, society and international community in accordance with Canadian interests and values." In particular, the SAR NIF program enables the DND/CF to "support Government of Canada (GOC) Programs."

3.1.3 ALIGNMENT WITH FEDERAL ROLES AND RESPONSIBILITIES

An assessment was made of the role and responsibilities for the federal government in delivering the program.

It has been well established over the years that Search and Rescue (SAR) is a critical service to Canada and Canadians given the country’s size, climate, topography and population. As noted by the Office of the Auditor General ⁵ , Canada fulfils its air and marine search and rescue obligations under international conventions.

Federal SAR incidents constitute marine, air and humanitarian. For ground SAR incidents the provinces and territories have the main responsibility.

The federal role was re-affirmed by Cabinet. The SAR NIF was set up by direction of Cabinet decision 6-0078-86RD (01), 10 April 1986, under the authority of the Minister of National Defence and by Treasury Board Decision 805407-1 of 11 June 1987.

Pursuant to a Treasury Board decision of 06 June 1996, the LMSAR/MDN had the authority to approve contributions, and associated terms and conditions, to non-federal entities. In 1998, the LMSAR/MND approved a contribution program to provide broader use of the SAR NIF ⁶.

3.2 PERFORMANCE

3.2.1 ACHIEVEMENT OF EXPECTED OUTCOMES OR EFFECTIVENESS

An assessment was made of progress toward expected outcomes (including immediate, intermediate and ultimate outcomes) with reference to performance targets and program reach, program design, including the linkage and contribution of outputs to outcomes. Reference should be made here to the Logic Model for the SAR NIF Program, reproduced in Annex D to this Report and to the Performance Measurement Strategy (referred to as the RMAF in the SAR NIF Terms and Conditions) ⁷. For each of the program activities listed in the sub-sections below are shown the nature of the outputs and extent of the outcomes. The evaluation assessment in each case is based on an examination of the revised 2005 SAR NIF Terms and Conditions, of MOUs, contribution agreements, ICSAR and Merit Board proceedings and minutes, and project files and as well the weight of opinions obtained through surveys and interviews conducted.

a) Set Plans and Priorities

The purpose of the re-design of the SAR NIF program was to address the deficiencies noted in the 2005 audit and evaluation reports.

Examination of the 2005 SAR NIF Terms and Conditions, of MOUs, contribution agreements, and project files indicates the following:

• The expected output included Program re-design to address the deficiencies. This resulted in the development of revised terms and conditions and program objectives and guidelines. This in turn resulted in the following outcomes:
• There is strong agreement amongst most survey respondents and supported by documents examined that the TBS Policy on Transfer Payments is now respected and that there is appropriate authority for the NIF contribution payments by Federal Partners.
• SAR NIF program requirements & set objectives support the National SAR Program’s Strategic Directions and Program Plan. It should be noted that the National SAR Program itself is beyond the scope of this evaluation which focuses on the contribution component of SAR NIF.

b) Establish Program Design & Operating Mechanisms

• The expected output was the development of a communications plan. The strategic directions and priorities for the National SAR Program included renewing the NSS website to improve the availability and accessibility of search and rescue information ⁸. The 2005 Evaluation Report had also recommended that:



• – “NSS website provide more detailed SAR best practices (collected from the Partners) based on NIF project results, and the contact details of the local experts”; and



• – “NSS endeavour to better publicize NIF successes to Canadians through articles in the mainstream media and in conjunction with other federal promotional material.”



• – “NSS initiate additional forums throughout the year that bring together Partners and NIF project proponents to discuss and share experiences and information on past, current and future NIF projects in an open manner.”



• The actual outcome was that much of the SAR NIF content of the NSS website had not been updated during the period of mid 2006 to March 2009. Many respondents to the survey indicated that many of the on-line tools previously available were removed from the NSS website. Survey respondents indicated a need for the NSS to provide more information and tools on the NSS website, including examples of successful SAR NIF applications. Current examples of successful projects were unavailable to potential SAR NIF contribution applicants. We are also not aware of whether there has been publication of NIF success stories to any great extent in the mainstream media.



• A SAR NIF strategic communication plan has been developed. Some of the main objectives of the plan are to increase awareness of the SAR NIF; provide success stories to internal audiences and to rebrand the SAR NIF. Planned activities and deliverables are:



• To develop a Video (long term planning)



• To design and develop a NIF Information Kit folder;



• To develop a new distribution process and compile a distribution list for dissemination of NIF news, information etc.;



• To design and develop various NIF promotional items (magnets, pen, pamphlet, floating key chain, mini-flashlight, poster);



• To design a display booth and pop-up banners;



• A SAR NIF one pager has also been developed and was distributed at Sarscene 2008 in Newfoundland



• A second output was the development of Memoranda of Understanding (MOUs) with managing departments and agencies. The 2005 Evaluation Report had also recommended that:



• – “NSS review all existing NIF administrative and management guidelines and MOUs and ensure that they are clearly worded and the roles and responsibilities of the NSS and the partner are clearly defined.”



• The actual outcome was that 12 MOUs have been signed in this period, once when the program was renewed in 2005. Six MOUs relate to VOTE 5 and six relate to VOTE 10. Aside from meeting the above-noted concerns, the agreements were vetted by the DND Legal Services to ensure among other concerns that all of the agreements were in compliance with the Treasury Board requirements.



• As mentioned in Section 1.2, eligible recipients of the SAR NIF contributions are provincial and territorial governments, profit and not-for-profit Canadian organizations from volunteer, academic or private sectors and individuals with responsibilities in the National SAR Program. The related output was the development of Contribution agreements between the NSS/DND and the provincial/territorial and other recipients involved and have resulted in the following outcomes:



• All contribution agreements have been vetted by the DND Legal Services to ensure among other concerns that all of the agreements are in compliance with the Treasury Board requirements.



• Merit Board members and funding recipients have noted that there is a lengthy approval process for contribution agreements. Timing is of key importance to the delivery of funds. This is discussed further in Subsection 3.2.1 (c) below.



• Contribution agreements with the provinces/territories require a single “omnibus” proposal from each province/territory. This means that the omnibus proposal may be considered ineligible if certain components of the application do not meet the eligibility requirements of the SAR NIF. Some Merit Board members have noted that this is an “all or nothing approach.”



• Inclusion of an “indemnity clause” in the contribution agreement has resulted in non-participation from several provincial/territorial governments. The new Policy on Transfer Payments provides more flexibility for Program Managers, including the exclusion of the indemnity clause from contribution agreements.



• The number of provincially sponsored contribution agreements approved under SAR NIF and the amounts approved have tended to be relatively small compared to that of projects sponsored by the federal partners. Some provinces/territories question whether the SAR NIF is providing a balanced response to SAR needs across Canada.

Recommendation 1:

The NSS Executive Director should ensure that the NSS website includes appropriate and up-to-date information regarding the SAR NIF program, including examples of successful initiatives recently undertaken.

Recommendation 2:

The NSS Executive Director should as part of the SAR NIF contribution program renewal process explore the possibility of creating a more efficient and equitable way to provide funding for provincially sponsored projects designed to support volunteer groups. The NSS should also review the omnibus process and determine the effectiveness of this strategic method of providing funds.

c) Review of Proposals & Approval of SAR NIF Projects

• The expected outputs were enhanced processes for proposal review and project approval. The 2005 Evaluation Report had also recommended that:



• – “NSS initiate discussions among all NIF Partners (including direct dialogue with small recipient organizations) to determine how best to remedy the issues of administrative support and guidance, and ensure that small groups in particular are not prevented from applying for NIF funding due to perceived and actual administrative requirements.”



• – “NSS ensure that a mechanism is in place to always provide, either prior to or during the Merit Board meeting, third-party reviews and expertise for any highly technical or scientific proposals that are received by the Merit Board. For example, separate Merit Board meetings could be held to review R&D proposals and non-R&D proposals so that third party experts are invited to the Merit Board session during which their expertise is required.”



• – “NSS immediately review the timeframes and approval processes that are in place to determine how best to reduce the NIF project approval period, from start to finish, and to be capable of announcing new approved proposals before the end of March.”



• However, the following outcomes were noted:



• The evaluation survey results are quite consistent in that most recipients had nothing but praise for the assistance and explanations provided by the SAR NIF program staff during the application process.



• Merit Board members confirmed that the situation has much improved whereby for the review of R&D proposals, an initial review and advice is provided by third party experts to the Merit Board. It should be noted that a majority of the R&D projects supported by the SAR NIF are outside the scope of this report as they are funded via the transfer payment process (Vote 5).



• There is an expectation that the selection and approval process would be undertaken with due diligence and in a timely fashion. However, the selection and approval process still tends to drag out to some 15 months, which is much too long, resulting in project management problems down the road.



• Lengthy approval process also results in project objectives being one year out of synchronization with National SAR Program priorities according to some Merit Board members.



• The proposal selection and project approval process needs to consider the prioritization of objectives – there are too many federal and provincial objectives at play to consider.



• Some Merit Board members indicated the need to clarify the “the rules of engagement” for the Merit Board process to formalize the proceedings. This would clarify their roles and responsibilities as Merit Board members. There is no evidence in meeting minutes that they are approved by the Merit Board members in attendance.

Recommendation 3:

The NSS Executive Director should as part of the SAR NIF contribution program renewal process explore ways to considerably shorten the contribution approval process to Government of Canada / DND norms.

Recommendation 4:

The NSS Executive Director should direct that the rules of engagement for the Merit Board should be re-examined with a view to take into consideration the need for improvement expressed by the members.

d) Project Management and Monitoring

• Among the expected outputs was the presence of information systems to provide financial management and reporting capability so that projects can be appropriately monitored. This is consistent with the National SAR Program Strategic Directions and Priorities that include improving SAR Program information management and data exploitation to guide planning decisions and develop prevention and awareness campaigns ⁹.



• The actual outcome is that although there have been several attempts made to provide this capability (NIF Information System, as well as an internally developed database) respondents indicated that the system still lacks functionality and there are recognized gaps. Merit Board members also indicated that there is no up-to-date searchable data base of successful projects to guide potential SAR NIF applicants.



• Another expected output was that there would be tight scrutiny of projects.



• In terms of outcomes, this is borne out in the survey responses from recipients who indicate due diligence is being exercised by SAR NIF staff of project financial and operational milestones and as evidenced also in the examination of selected SAR NIF project files.



• Examination of files and survey responses indicates due diligence is also being exercised by SAR NIF staff in the processing of project claims and payments.

Recommendation 5:

The NSS Executive Director should direct that priority steps be taken to ensure that the SAR NIF information systems are robust enough to satisfy government financial requirements as well as program management and user needs. It should be able to track the authorized carry forward to the next year of unspent SAR NIF project dollars at fiscal year-end.

e) Program Accountability

• There were expected outputs designed to ensure program accountability. Most of these activities were budgeted for, but not undertaken for whatever reason, resulting in no output, and no way to bring about the favourable and desired program outcomes. The missing outputs include:



• – SAR NIF recipient audits (Over the past four years, the NSS has completed one audit of a SAR NIF project and is in the process of completing two additional audits. An additional report has been produced by Transport Canada in regards to their role in the delivery of the SAR NIF) ;



• – To the NSS/DND knowledge one audit has been initiated by a provincial/territorial recipient;



• – SAR NIF project evaluations (none were undertaken);



• – Collection and interpretation yearly of performance measurement information;



• – Annual reports on SAR NIF operations.



• However, the (5year) Review of the SAR NIF program has been initiated, with this summative program evaluation as a component of the review. The following outcomes are expected as a result:



• SAR NIF program renewed, with a focus on results as an immediate outcome.



• Revamping of plans and expectations as required, as a medium term outcome.



• Review and revision of the Performance Measurement Strategy.

Recommendation 6:

The NSS Executive Director should direct that the renewed SAR NIF’s Program Management Strategy includes an appropriate auditing and evaluation framework along with the requirement for systematic collection of performance information and the regular issuance of an annual report on SAR NIF operations.

3.2.2 DEMONSTRATION OF EFFICIENCY AND ECONOMY

An assessment was made of resource utilization in relation to the production of outputs and progress toward expected outcomes.

• An expected output would be the delivery of the SAR NIF as designed.



• There is an expectation that the SAR NIF program would be delivered with economy and efficiency. Efficiency of delivery implies that one can compare the delivery to known benchmarks for delivery of similar programs. Given that there are no readily available benchmarks, such a comparison would be difficult.



• What is known is that file examination of projects indicates due diligence is being exercised by project administrators. In fact some budget items in projects examined indicate under-spending. We are told that volunteers, who are involved in SAR NIF projects, tend to be frugal by nature.



• Table 3 below shows the forecast spending amounts for SAR NIF. The forecast Operating and Maintenance (O&M) budgets show no increase during the period; in fact the forecast figures show a decrease. This is indicative of a program that is managed through careful spending in terms of both O&M and as well salary dollars.

• 5. 1992 Report of the Auditor General of Canada — Chapter 8 - Search and Rescue.



• 6. NSS Web-site: SAR NIF Program Authorities.



• 7. See the RMAF for SAR NIF. Logic Model is in Exhibit 1; and Performance Measurement Strategy sets out performance indicators in Exhibit 3.



• 8. DND Reports on Plans and Priorities – 2008-2009 Strategic Outcomes.



• 9. Previously cited: DND Reports on Plans and Priorities – 2008-2009 Strategic Outcomes.

 Table of Contents

ANNEX A – MANAGEMENT ACTION PLAN -> Next Page