Background 8.1 The Employment Development Programs offered through the Department of Training and Employment Development meet a variety of different needs and serve a variety of clients. They are an important aspect of the work done by the Department and make up a large part of the expenditures of the Department ($131 million in 2001). There are five main programs (see Exhibit 8.1), most of which have various sub-programs or components. The Work Ability Program provides wage subsidies to eligible employers who are able to provide job experience and training to individuals to help them become “job ready”. The Workforce Expansion Program has two components, the wage subsidy portion which is intended to aid in creating permanent employment or annually recurring seasonal jobs in New Brunswick, and the Self Employment Benefit component which helps clients create jobs for themselves through self employment. The focus of the Training and Skills Development Program is to help clients receive training or educational programs which will allow them to return to work. The goal of the Student Employment and Experience Development (SEED) program is to ensure employment opportunities are available to students who need work experience as part of their eventual integration to the workplace. 8.2 The four components of the Employment Services Program address the needs of individuals, employers and communities in adapting to changing labour market needs. Adjustment Services help employers and employees through many challenges facing business today: growth, new technology, downsizing and human resource issues. Training and Employment Support Services are aimed at persons with permanent or long-term disabilities who need to develop marketable skills in order to enter the labour force. Employment Assistance Services provides access to employment-related services to clients through the use of external service providers. The Research and Innovation service exists to support research and innovative pilot projects that identify better ways of helping persons prepare for, return to, or keep employment and be productive participants in the labour force. Scope 8.3 The main focus of our audit was the management of the programs to see how the Department was doing in attempting to meet the economic and employment strategic goals and objectives of the Province. Effective management comes from the operation of the programs, the systems they rely on and evaluation of results. 8.4 Our work covered the Employment Development Programs funded through both provincial and federal monies in the fiscal years 2000-2001 and 2001-2002. The Department uses two main systems in the operation of the programs: Employment Development Tracking System (EDTS) and Skills Loans and Grants (SLG). Our systems work concentrated on the EDTS system as it covers the largest number of programs and slightly over half of the dollars spent. The actual testing of contracts covered the larger programs: Work Ability, Workforce Expansion, and Training and Skills Development (TSD). We did not look at contracts under the Student Employment and Experience Development (SEED). We conducted minimal testing under three of the four components of the Employment Services Program: Adjustment Services, Training and Employment Support Services (TESS) and Employment Assistance Services. We did not test the Research and Innovation service of the Employment Services Program. Our work included interviewing departmental staff, reviewing documented policies and procedures and testing individual contracts from various programs. We tested 91 contracts in total and the amount chosen from each program was determined using the budget figures from 2000-2001 and 2001-2002 (see Exhibit 8.1). The field work and testing was conducted in January, February and March 2002 and the results reflect conditions in effect at that time. Exhibit 8.1 Employment Development Programs 8.5 The three objectives for our audit were: • to assess the adequacy of the Employment Development Tracking System internal controls designed to provide timely and reliable financial information; • to assess the adequacy of management and control processes over the approval and payment of Employment Development funding; and • to determine if there are adequate procedures in place to measure and report on program effectiveness. 8.6 To assist us in conducting our work we developed fourteen audit criteria. 8.7 Our chapter is presented following these objectives and criteria. It is divided into sections based on the three objectives and in each section we present the criterion, and our findings relating to the criterion. We indicate whether the criterion has been met and provide recommendations if applicable. Results in brief 8.8 Although there are internal controls in place designed to provide timely and reliable financial information, there are improvements that the Department could make. The Department should finalize its information technology security policy, and provide clear guidance to all regions on documentation that should be maintained to support contracts. 8.9 Potential clients are well informed about the programs for which they may be eligible. Eligibility criteria exist, but are not always clearly documented, and draft program guidelines have not been finalized. Nevertheless, applicants under the programs appear to be treated equitably and all employees hired under the contracts we tested met the established client profiles. Applications are approved and the decision communicated on a timely basis, and the disbursement of funds is made on the basis of a properly approved wage claim. However, an audit function should be established to verify client assertions in both the applications and the wage claims. 8.10 Programs are not evaluated on a regular basis. Nor does the Department publicly report on the effectiveness of its programs. The departmental annual report describes overall goals for the Labour and Employment Division, but there are no reported specific objectives or performance indicators tied to these goals. Adequacy of internal controls 8.11 Our first objective was to assess the adequacy of the Employment Development Tracking System internal controls designed to provide timely and reliable financial information. 8.12 This system is used to support the management of many of the Employment Development Programs. With the EDTS system, documents can be viewed on-line, applications can be approved electronically and changes to contracts can be made and processed immediately. 8.13 During the course of our work on this audit we noted that the Office of the Comptroller had done a review of the implementation of the EDTS system. We spoke to the Office of the Comptroller about their work and examined their files in order to determine if we could rely on their work. We were satisfied with the work they had completed and decided that we could rely on their work in the area of internal controls designed to provide timely and reliable financial information. As a result of this, much of our work concerning the three criteria under the first objective consisted of a review of the Comptroller’s work. Is there security over data? 8.14 Our first criterion was: Key controls should be in place to ensure the security over data captured and processed within the Employment Development Tracking System. 8.15 The first criterion questions whether the security of both financial and client data in the EDTS system can be counted on to provide accurate and reliable accounting information. Controls are necessary to ensure that the security procedures are functioning as required. The system contains sensitive information about clients and their contracts as well as the financial information that is transferred to the Province’s financial system. There are many issues that need to be formally documented to help ensure the security of this information. Among them are a security awareness program, a disaster recovery plan, adequate separation of duties, and an acceptable computer usage policy. 8.16 One of the outcomes of the Comptroller’s review was a recommendation for the implementation of a departmental information technology security policy to address the above mentioned issues. The departmental response was that such a policy was to be completed by 31 December 2000. When we asked to see this policy, it was still in draft form. A Department representative is sitting on a committee that includes the RCMP, local law enforcement and the government’s Corporate Information Management Services (CIMS) and they are waiting for this committee to finalize some of the issues before the departmental policy is issued in its final form. The Department does not want to release a security policy that does not coincide with a government-wide security policy to be released by CIMS. We recognize that the process does take some time, but a policy is needed to establish responsibility for security and to emphasize the importance of proper security procedures. Recommendation 8.17 We recommended that the draft departmental information technology security policy be completed and finalized as soon as possible. Departmental response 8.18 The Department recognizes the urgency of introducing a Departmental Information Technology Security Policy to establish responsibility for security and to emphasize the importance of proper security. To this end, we have developed a draft policy which has been awaiting finalization utilizing the Corporate input from Corporate Information Management Services (CIMS) in the Department of Supply & Services. [The Department] will continue to work closely with the Corporate Information Management Office in an effort to finalize the policy. Conclusion 8.19 The Department partially met this criterion. Some controls are in place to ensure the security of data captured and processed within the Employment Development Tracking System, but no information technology security policy yet exists to formalize the controls. Is there a separation of duties? 8.20 Our second criterion was: The system should have controls in place to ensure there is adequate separation of duties among staff assigned to the employment programs. 8.21 An adequate separation of duties is a fundamental concept of internal control. Ensuring that no one person can handle all aspects of a transaction from beginning to end reduces the opportunity for any one person to both perpetrate fraud and conceal errors in the normal course of his or her duties. 8.22 We found no problems in this area during our testing or through our review of the Office of the Comptroller’s work. During our work we noted such important controls as the separation of the approval and the payment functions, and the separation of the receipt of applications and the approval of applications. We were pleased to note that different levels of contract authorization exist within the regional offices, for instance Program Officers cannot approve a contract over $50,000 without approval from the Regional Director. During our testing we noted no instances where these above-mentioned controls were overridden. Conclusion 8.23 The Department met this criterion. Controls are in place to ensure the adequate separation of duties among staff assigned to the employment programs. Does an audit trail exist? 8.24 Our third criterion was: The system should have controls in place to ensure there is an audit trail for the information captured by the system. 8.25 An audit trail is needed as a source of evidence to link transactions to the summary departmental financial information and to ensure proper records of authority for contracts exist. It is especially important in an electronic system where many transactions exist only in electronic format. 8.26 Testing the individual contracts revealed inconsistencies across the regions as to what types of documentation are sent to be imaged or are collected by the various regions. For instance, some officers send the business plans required by the Self Employment Benefit Program to be imaged while others do not. Not all regions send the committee minutes generated by the Adjustment Services program to be imaged. Due to these inconsistencies, it was not always possible to follow all contracts from start to completion in the EDTS system; the audit trail was compromised. Recommendation 8.27 We recommended that the policies and procedures of the various programs give clear guidance on what documentation is to be sent to be imaged under each of the programs and what should be kept by the Program Officers and Employment Counsellors. Departmental response 8.28 The Department agrees that it is important to have policies and procedures in place to give clear guidance to staff on what documentation is required for each program. In support of this recommendation, the Employment Division has recently undertaken and completed a review of its Imaging Reference Manual that outlines what documents are required to be imaged by individual program area. 8.29 In addition, the Employment Division recently introduced a Quality Assurance and Monitoring function within its Employment Programs and Services Branch. An integral activity within this function will be to review individual contracts to ensure staff is adhering to established policies and procedures. Conclusion 8.30 This criterion was partially met. There are controls in place to help ensure there is an audit trail, but improvements can be made. Approval and payment of funding 8.31 Our second objective was to assess the adequacy of management and control processes over the approval and payment of Employment Development funding. Are people aware of the existence of programs? 8.32 Our first criterion was: There should be adequate procedures in place to ensure that people eligible for help under an Employment Development Program are aware of the existence of these programs. 8.33 This criterion addresses the management process of disseminating knowledge of the programs to the people who would benefit from them. Objectives will not be met if the clients the programs are aimed towards are unaware the assistance is available. 8.34 Our work indicates that clients are well informed about the existence of these programs and that there are many avenues from which to learn about them. We were pleased to see that clients learn about these programs from employment counsellors, the internet, and information sessions hosted by the various regional offices. Clients are also referred by various third party agents such as community business organizations, local chambers of commerce and even the Atlantic Canada Opportunities Agency (ACOA). Conclusion 8.35 The criterion was met. There are adequate processes in place to ensure that people eligible for help under an employment development program are aware of the existence of these programs. Are eligibility criteria clear? 8.36 Our second criterion was: Each of the Employment Development Programs should have clearly stated eligibility criteria, terms and conditions for both employer and employee target groups. 8.37 This criterion addresses the existence of eligibility criteria for both employees and employers for each of the Employment Development Programs. Such criteria exist to help the programs achieve their goals and objectives and to ensure consistency and fairness in the delivery of the programs. 8.38 We found that eligibility criteria did exist for the fiscal years 2000-2002 and that they were adequate when considering the objectives of the various programs. In most cases the criteria had not been updated for the new programs, but officers had been informed of changes. The draft policies and procedures we received in March 2002 detail these criteria adequately and appear reasonable. Our recommendation under the next criterion addresses the issue of finalizing these draft guidelines. 8.39 However, we did find that there is one program where an important condition is not covered by the application form or the Letter of Offer. Employers were requested to complete a client evaluation under the Work Ability Program. This evaluation could have been used by the employment counsellors to determine if further intervention was needed with a client. However most of the employers did not complete and return the form as requested. Given that one of the objectives of this program is to provide unemployed New Brunswickers who are being ‘case managed’ with an opportunity for skill development, these evaluations are necessary for the counsellor to know where further skill development is required. This is an important document and its importance should be stressed in writing so that it is clear to the employer. Recommendation 8.40 We recommended that the Letter of Offer or the Application for the Work Ability Program be amended to include a clause specifying that the employer agrees to send in the client performance evaluation form as part of the Attestation of Salary. Departmental response 8.41 The client performance evaluation form and the Attestation of Salary have been incorporated into one document. The Letter of Offer for the Work Ability Program has been amended to include a clause specifying that the employer must send in the Client Performance Evaluation/Attestation of Salary Form before obtaining final contract payment. Conclusion 8.42 This criterion was partially met. For the fiscal years 2000-2002 eligibility criteria existed but not always in a documented and formalized form. The necessity of completing a client evaluation under the Work Ability Program is not covered in either the application or the Letter of Offer of this program. Are policies and procedures documented? 8.43 Our third criterion was: Documented policies and procedures should exist to guide Program Officers through key aspects of the process. 8.44 This criterion addresses the question of whether or not policies and procedures exist to guide officers through such processes as approval, payment and monitoring. Having such policies documented is especially important when programs change or new employees come into the Department. 8.45 We had requested copies of the policies and procedures for the various programs for the years 2000-2001 and 2001-2002. Many of the programs changed in 2000-2001, so documents available for that fiscal year were outdated and did not reflect the program changes. Program changes were communicated through the use of workshops for officers and programs were delivered with guidance and support from central office. It should also be noted that the delivery structure changed in the same year from a central perspective to a regional one. With this change Regional Offices were given the ability to approve a greater variety of contracts, including ones under the Work Ability Program, and officers were given greater flexibility in dealing with clients. At the same time the number of offices grew, as sub-regions were added to the various existing regions. 8.46 We received draft guidelines for most of the programs near the end of the audit, in March 2002. Although these guidelines do address many of the problems we discovered during the course of our audit, they were written nearly two years after the program changes were initiated. This led to inconsistencies in the way programs were administered across the Province. Examples of this include: • Under the Training and Skills Development program, officers were to get client performance reviews from the schools their clients were attending every four months. Some offices followed up monthly, others followed up at major school holidays (for example at Christmas and Easter), and others asked the student to call a couple of times during the training. • Monitoring procedures varied greatly across the regions. Two regions routinely visit clients under the Workforce Expansion program and have developed their own forms on which to record these visits. Others visit if they get the chance or if they happen to be ‘in the neighbourhood’. Some officers rarely do monitoring visits. The type of evidence gathered when monitoring also varied: some verify payroll records to wage claims, others just verify that the employee is there and doing the type of work they were hired to do. Officers found it difficult to know what to do, as there were no policies to which to refer. The lack of an audit function within the Department makes these monitoring visits important as a control function. • In many cases the people working in the regional offices had typed up their own version of the policies that applied to the various programs and would refer to them when asked what documentation they had in relation to the programs. Recommendation 8.47 We recommended that the draft guidelines be finalized as soon as possible and any future program changes be supported by documented guidelines at implementation. Departmental response 8.48 Guidelines for all five employment programs have been finalized and are available to all Employment Division staff through the departmental intranet site. Any future program changes will only be introduced once documentation supporting the changes has been included in the program guidelines and modified on the intranet site. Recommendation 8.49 We recommended that the guidelines include direction on monitoring and include such things as how often and when to monitor as well as what things officers should be looking for. The creation of standardized forms would make monitoring visits more consistent across the Province. Departmental response 8.50 The Employment Division is in the process of developing a Quality Assurance and Monitoring Strategy for its employment programs. This strategy will provide direction on monitoring of contract activities to staff and will address this issue of standardization and consistency across the Province. We expect to introduce the first part of the Strategy by the end of the current fiscal year. Conclusion 8.51 This criterion was not met. For the years in question the policies and procedures were not up to date as new ones were being developed. Policies did not cover the process of monitoring either the budgets or the contracts. Policies did cover the application, approval and payment processes. Are selection criteria documented and consistently applied? 8.52 Our fourth criterion was: Applicants should be assessed against documented selection criteria and only those meeting the criteria should be approved. 8.53 In order to ensure that the programs are treating all applicants equitably, there must be selection criteria for all programs and all officers must follow these criteria when assessing the applicants. Approval and rejection must be based on whether or not the applicants meet the required criteria. In all our discussions with Program Officers we found they were using selection criteria consistently across the regions. 8.54 In our testing, we found that all applicants are assessed using the same criteria and that rejections are based on these criteria. However, as noted earlier, there were not always current documented policies to which to refer. This along with the fact that one program changed selection criteria part way through the fiscal year meant that it could be a challenge to ensure that the criteria were met. The Workforce Expansion criteria changed in October 2001, so that only employers who were going to hire youth or post secondary graduates would qualify for funding. This mid-year change frustrated many officers and confused employers. 8.55 We also found that the quality of the evidence gathered to support the client’s acceptance into the program varies across the regions. Under the new Workforce Expansion guidelines, some officers requested birth certificates to verify if an employee hired was a youth, others looked at driver’s licenses and others took the information from the Notice of Hiring as being correct. Some officers visited clients before approving them under the program, others made do with a phone call. Even the quality of the referral sheets used under the Work Ability program varied and not all of the referrals examined during testing were completed correctly. For example, some referral forms were more informative about a client’s career plan than others and a few of the forms examined did not contain the name of the Employment Counsellor or Case Manager referring the client to the program. Recommendation 8.56 We recommended that the Department provide policy guidance on the evidence needed to verify that applicants meet the eligibility criteria of the various programs. Departmental response 8.57 Guidelines for all five employment programs have been finalized and are available to all Employment Division staff through the departmental intranet site. These guidelines provide policy guidance on the evidence needed to verify that applicants meet program eligibility criteria. 8.58 As part of our testing for this criterion, we reviewed applications to the various programs. At this time it came to our attention that not all of the programs actually have application forms. For instance there is no application form for the Adjustment Services component of the Employment Services Program. Section 4 of Regulation 89-145 of the Employment Development Act states “an application for financial assistance under an employment development program shall be made on a form provided by the Minister for that program.” We believe that this would require that all programs have an application form. Application forms are also an important source of information about both the clients of, and applicants to, the programs. Recommendation 8.59 We recommended that the Department provide application forms for all of the Employment Development Programs. Departmental response 8.60 The Department agrees in general with this recommendation. Currently all programs with the exception of two components under the Employment Services Program, Training and Employment Support Services (TESS) and Adjustment Services, have application forms. Under the TESS component, applicants must provide project proposals as an application to receive funding support. However no formal application process is in place under the Adjustment Services component. The Department will review this component to ensure a standard application process is in place. Conclusion 8.61 This criterion was not met. From our discussions with Program Officers and based on our sample, all clients are assessed using the same criteria, but there were not always documented criteria to which to refer. As a result, all officers are not gathering the same quality of evidence to verify that the clients meet the criteria. Are all applicants treated equitably? 8.62 Our fifth criterion was: The Department should have policies and procedures to ensure all applicants for employment, within specific target groups, are treated equitably under the applicable programs and that all employees hired meet the client profiles established in the guidelines. 8.63 Policies are needed to ensure that all applicants are treated fairly and have an equal chance of participation in the programs. Programs have limited budgets, so in order for the programs to meet their objectives, participants must meet their respective client profiles. 8.64 From individual interviews about the application processes of the various programs with Program Officers and Employment Counsellors, it appears that all applicants are treated equitably. All Work Ability participants are referred by a counsellor and have an approved career plan. Under the Self Employment Benefit program, all applicants must prepare a business plan that is then assessed by a committee consisting of community and department members. Training and Skills Development applicants must demonstrate that they have researched their chosen career and they are likely to be able to support themselves once through their respective programs. Clients will be approved if they meet the profiles established by the guidelines. The information received from the various regional personnel was verified through our testing of individual contracts. Conclusion 8.65 This criterion was partially met. Although applicants under the programs are treated equitably and all employees hired under the contracts tested in our sample met client profiles from the guidelines, there were no current documented policies and procedures in place. Are decisions timely? 8.66 Our sixth criterion was: Project application approval or rejection should occur in a timely manner and the decision should be communicated on a timely basis. 8.67 We looked at the program guidelines we were given to determine if they contained suggested turn-around times. Since the guidelines contained no suggested turn-around times for the application process, we spoke to Program Officers to get an idea of how long the process usually takes for the various programs. We then examined applications and the related contracts or rejection letters to verify the turn-around times we were given by the Program Officers. Times we were given by the officers were supported by our tests. For the Workforce Expansion program the process can take anywhere from one to fourteen days. Due to the need to match client employment needs with the employer’s available positions under the Work Ability program, the process can take an average of two weeks to a month. Under both the Training and Skills Development and Self Employment Benefit programs, the time before approval can be given varies due to the information required from the client. Once the information is received, approval can be given quite quickly. Approvals under the Employment Services programs also occur in a timely manner. Recommendation 8.68 We recommended that the programs’ policies and procedures include suggested turn-around times for the approval process. Departmental response 8.69 Program administrative guidelines will be amended to include suggested turn-around times for each program’s approval process. Conclusion 8.70 This criterion was met. Applications are approved and the decision communicated on a timely basis. We feel the process could be improved by the addition of recommended turn-around times to the programs’ policies and procedures. Are wage claims properly approved? 8.71 Our seventh criterion was: Disbursement of funds should only be made on the basis of a properly approved claim. 8.72 An important control over the funds disbursed is proper approval of the wage claims. We tested a number of claims from various contracts under the Work Ability, Workforce Expansion and Training and Skills Development programs over the years 2000-2002 and found no instances where wage claims were not properly approved and processed. Conclusion 8.73 This criterion was met. Based on the tests performed, disbursements of funds are made on the basis of a properly approved wage claim. Is there an audit function? 8.74 Our eighth criterion was: An appropriate audit regime should exist to verify client assertions in both the applications and claims. 8.75 An audit function is essential to promote accountability and control. Procedures to verify client assertions in applications and claims can help detect false claims and errors. Further it can ensure that all program criteria are met. 8.76 In our opinion, some type of audit function is also implied in the Regulation. Under the Employment Development Act, Regulation 89-145 states: 7 A person who receives financial assistance under an employment development program shall a) keep such records, books, accounts and other documents as are required under the rules of the program, and b) make such records, books, accounts and other documents available to the Minister for inspection. 8.77 The reference in this regulation to “… records, books, accounts and other documents …” being available for inspection implies that the Minister will carry out some form of audit work to verify client assertions. 8.78 We discovered that there used to be an audit division that visited clients in the field. They verified wage claims, ensured payroll remittances were made and verified client assertions. This function was discontinued in late 1998, at the time of a major new system implementation, and has not been reinstated. A new internal audit function is presently being developed but its function will be focused on internal compliance with relevant Acts and Department policies regarding payment. There are no plans to visit employers and clients to verify their assertions in both applications and claims. The head of the audit division feels this is a function better suited for the Program Officers as they are already in the field and know the clients. Some of the Program Officers do this sort of ‘audit function’ when they visit a client for monitoring purposes, but some have been told not to ‘audit’ on these visits. 8.79 As we have noted earlier, there is a lack of consistency in the monitoring currently carried out by the Program Officers. Given the size of these programs, the heavy involvement of third parties, the absence of an audit function and the inconsistency in monitoring, the Department seems to be exposed to an unnecessary risk. For example, as employers come to know that they will not be audited, the possibility of fraud increases. The other side of this is if the potential for audit is known, there will be a tendency to be more diligent in record keeping. Recommendation 8.80 We recommended that an audit function be set up to detect misrepresentations and errors in client assertions, applications and wage claims. There should be appropriate procedures in place to report on the findings of the audit division in an independent manner. Departmental response 8.81 The Department agrees that an audit function needs to be set up to detect misrepresentations and errors in client assertions, applications and wage claims. To this end, we have initiated an internal audit function within the department with the priority focused on employment programs. Preliminary audit programs have been developed for each of the program areas and we are now implementing the audit program for Training & Skills Development, with the intent of implementing the others thereafter. Conclusion 8.82 This criterion was not met. There is not an appropriate audit regime to verify client assertions in either the applications or the wage claims. Reporting on effectiveness of programs Is there regular program evaluation? 8.83 Our third objective was to determine if there are adequate procedures in place to measure and report on program effectiveness. 8.84 Our first criterion under this objective was: The Employment Development Branch should have policies and procedures which ensure that evaluations of its programs are conducted on a regular basis. 8.85 Evaluations are essential for ensuring that programs stay valid and are a necessary stage in the management of the programs. There are many considerations that should be addressed in the evaluation process. Among them are: • evaluations should address the extent to which programs continue to be relevant; • evaluations should determine if a program is actually meeting the goals and objectives that it is supposed to; • evaluations should consider the cost of a program; and • evaluations should consider input from those most familiar with the program, such as clients and departmental staff. 8.86 During our audit, we discussed the evaluation process with the Labour Market Analysis Branch of the Department. We were told the Department is in the process of developing a framework for evaluation of the five employment development programs. This framework will be the starting point towards developing a policy for program evaluation. This is a good beginning, but the programs began to be restructured in late spring 2000, and it was at this point that the discussions began concerning the evaluation of the newly redesigned programs. In order to determine, in a timely fashion, if the redesigned programs are relevant and are meeting the goals and objectives they are supposed to, in our opinion the evaluation framework should have been established at the implementation of the redesigned programs. Recommendation 8.87 We recommended that guidelines for program evaluation be completed and documented as soon as possible. Evaluation criteria should be established when program changes are implemented and should be an important step in the development of the changes. Departmental response 8.88 Board of Management recently approved the Department’s Employment Programs Evaluation Framework. This Framework allows us to review program results on an annual basis and is an integral mechanism in determining the success of the Department’s employment programs. Furthermore, evaluation results will assist in establishing policy decisions that will lead to program amendments or redesign. Conclusion 8.89 This criterion was not met. There are no formal policies and procedures which ensure that evaluations of departmental programs are conducted on a regular basis. Are recommendations from program evaluations responded to? 8.90 Our second criterion was: The Employment Development Branch should have a process in place to respond to recommendations resulting from program evaluations. 8.91 In order to continue to improve and to promote accountability, programs should have a process in place to respond to recommendations resulting from program evaluations. This process allows those involved in the operation to present their views on the success of a program. In addition, if weaknesses are identified in a program, the Program Officers and managers responsible for the program then have the opportunity to indicate their agreement or disagreement. If they agree, then they have the chance to explain how they plan to deal with the weaknesses. 8.92 As previously mentioned, there is no policy at the moment with respect to program evaluations, so there have been no recent program evaluations. Recommendation 8.93 We recommended that the planned guidelines for program evaluation identify a process to respond to recommendations resulting from program evaluations. Departmental response 8.94 The Department will identify a process to respond to recommendations resulting from program evaluations. Conclusion 8.95 This criterion was not met. Does the Department report publicly on program effectiveness? 8.96 Our third criterion was: The external reporting on Employment Development Programs should be in compliance with the annual report policy of the Province. 8.97 Our Office has been looking at departmental annual reports as the major accountability tool for departments since 1989. At that time we noted several major deficiencies in the annual reports that were being prepared. Several recommendations were made for improvement and, in 1994, an Annual Report Policy was issued. The requirements of the annual reporting policy are clear. Section 4 of the policy states: The prime function of an annual report is to be the major accountability document by departments and agencies for the Legislative Assembly and the general public. It serves as the key public link between the objectives and plans of a government entity and the results obtained. 8.98 We reviewed the 2000-2001 annual report of the Department of Training and Employment Development to see if it complied with the Annual Report Policy. The requirements we examined are found in section 5 of the policy: a) [the annual report] should give a clear account of goals, objectives and performance indicators b) the report should show the extent to which a program continues to be relevant c) [the report should show] how well the organization performed in achieving its plans d) [the report should show] how well a program was accepted by its client groups e) [the annual report should present] actual and budget financial information in summary form and a narrative explaining major variances. Requirement #1: A clear account of goals, objectives and performance indicators. 8.99 The 2000-2001 annual report does describe overall goals for the Labour and Employment Division, but there are no specific objectives or performance indicators tied to these goals. The individual employment programs are described, but again there are no goals, objectives or performance measures given. Requirement #2: Continuing relevance of programs 8.100 The continuing relevance of the programs is evident through their redesign in order to better meet the new focus on individual client needs and long-term sustainable employment. 8.101 A quote from the book Accountability, Performance Reporting, Comprehensive Audit - An Integrated Perspective (CCAF, 1996) provides an interesting perspective on relevance. Relevance refers to the extent to which a program … continues to make sense in regard to the problems or conditions it is intended to address. It is an attribute that normally would be considered in preparing a management representation about a major program, activity,… within an organization. Information about this attribute could lead to confirmation, amendment, or elimination of that program, activity, or business. 8.102 In our opinion, the question of whether the program “continues to make sense in regard to the problems or conditions it is intended to address” should be addressed in evaluations of the programs and these evaluations should be reported on appropriately in the annual report. Requirement #3: Were goals achieved? 8.103 The report does say how many people got assistance under each of the programs, but these statistics are not tied back to the goals of the branch or the programs themselves. Without performance indicators there is no clear way to tell if the goals of the programs or the branch were met. Requirement #4: Acceptance by client groups 8.104 There is no mention of how well the programs were accepted by their client groups. Requirement #5: Variance Analysis 8.105 Annual and budget financial information is presented in a summary form but the narratives explaining major variances are superficial at best. We do note that the summary financial information goes into greater detail than is required by the Annual Report Policy. Expenditure information is given for each of the five employment programs and sub-programs. Recommendation 8.106 We recommended the Department review its annual report in relation to the Annual Report Policy and identify areas where improvement is still possible and work to make the identified improvements. Departmental response 8.107 The Department is committed to meeting the annual report policy of the Province of New Brunswick. As such, the Department will review current report structure and content and will make improvements where possible. Conclusion 8.108 This criterion was partially met. External reporting on Employment Development Programs is in partial compliance with the annual report policy of the Province of New Brunswick, but areas for improvement do exist and should be addressed.