Background 4.1 According to the Canadian Mental Health Association (CMHA), studies indicate that approximately 20% of the general population has had some form of mental illness in the previous year and approximately 3% of the population is affected by serious mental illness causing profound suffering and persistent disablement. If we add family members, who carry a major burden of care, the figures of those impacted by mental illness in Canada would be multiplied two to three times. 4.2 The CMHA goes on to say that the extent of mental illness can also be understood by the following data: • one out of every eight Canadians can expect to be hospitalized for a mental illness at least once in their lifetime; • mental illness is the second leading condition requiring hospital use among those aged 20 - 44; and • in a recent study of general medical practice in Canada, psychiatric illness was found in one-quarter of patients. 4.3 The Mental Health Services Division (the Division) is one of five divisions within the Department of Health and Wellness (the Department). According to the Department, the Division’s primary role is to provide central leadership and accountability for the effective, efficient, and equitable delivery of all formal mental health services in the Province. 4.4 The Division oversees the operation of 13 Community Mental Health Centres (CMHCs). It also administers psychiatric services agreements with eight Regional Health Authorities (RHAs) in seven regions covering eight psychiatric units, one child and adolescent unit, and two institutions - the Restigouche Hospital Centre in Campbellton and Centracare in Saint John. 4.5 The Division’s budget for the 2002-03 fiscal year was $55.3 million. With this budget, the Division funds the operations of the CMHCs ($24.2 million) and, through psychiatric services agreements with the RHAs, the costs related to the clinical delivery of programs of the psychiatric hospitals and units ($26.5 million). 4.6 Data for fiscal year 2002-03 regarding the number of psychiatric beds, occupancy rates (%), and average length of stay in days (ALOS) at the psychiatric hospitals and units is as follows: Notes: 1 The number of beds that are approved is 191 but the actual number of beds being used is 169. The reduction in beds is due to RHA diversion strategies aimed at reducing the need for hospitalization, while keeping the funds within the mental health area. Examples include the diversion of funds into the Day Hospital Program and for the placement of additional nurses in the emergency room. Beds are closed only after approval has been obtained from the Division. 2 Occupancy and ALOS are based on information available from six out of eight psychiatric units. 3 Funding provided by the Division to both the psychiatric hospitals and units is based on a 90% occupancy rate. Less than 100% occupancy rates, as shown for the psychiatric hospitals, can be due to an accumulation of very short vacancies because any vacancy is filled as soon as the person at the top of the waiting list can be transported. 4 No ALOS data was provided. The forensic unit performs 30-day court-ordered psychiatric assessments and the majority of other units have a combination of people who have been there for many years and others who have been there for shorter terms. 4.7 Over the past fifteen years, the mental health services sector in New Brunswick has undergone significant change. In 1988, a ten-year plan was proposed for a reformed mental health system. The plan was essentially completed in 1996, two years earlier than planned. New Brunswick is recognized across Canada for employing some best practices in mental health reform. Among the key ingredients identified for success, according to the Clark Institute of Psychiatry, were the creation and management of an integrated funding envelope, regionalization, and adoption of a mental health policy committed to reallocating resources from institutions to the community. 4.8 With growing control at the regional level and with more money being distributed to support regional operations, it becomes increasingly important to have central mechanisms in place to ensure the government is achieving its objectives. At the present time, many health services are under the management of the RHAs. Examples include addiction services, extramural hospital, mental health services provided by the psychiatric hospitals and units, and all health services offered by the hospitals in the RHA (cancer care, cardiac care, rehabilitation, etc.). As a reflection of this, the Department issued contributions, grants, and subsidies to hospitals totalling $826.1 million in 2001-02 representing approximately 59% of the entire Department of Health and Wellness expenditures for that year. The need for a strong accountability structure is extremely important. 4.9 Such a structure would allow the Department to maintain control over quality standards of care and consistency across regions. We decided to focus our audit on the accountability processes the Department has in place over the RHAs with regard to the performance of the psychiatric hospitals and units. We focused our efforts on the mental health sector due to both its importance to New Brunswickers and the significant amount of change that has occurred in this sector over the past fifteen years. 4.10 Although mental health is a relatively small component of the total contributions, grants, and subsidies issued to RHAs by the Department, we believe that the recommendations that result from this audit will be applicable to other decentralized accountability relationships in which the Department is involved. Scope 4.11 The objective for our audit was as follows: To assess whether the Department of Health and Wellness has appropriate accountability processes in place for the operations of the psychiatric hospitals and psychiatric units under the direction of the Regional Health Authorities. 4.12 We developed three criteria to assist us in conducting the audit. These were discussed with departmental staff and staff from the RHAs to ensure there was understanding and agreement. Our comments in the report are organized by the criteria and we conclude on whether the Department has met each of them. 4.13 Our audit consisted of interviews with staff from the Mental Health Services Division and mental health staff at three of the seven regions. These regions are responsible for both of the psychiatric hospitals and three of the eight adult psychiatric units. All Mental Health Services Division Directors, with the exception of the Director of Child and Adolescent Services, were interviewed. We excluded the six-bed child and adolescent psychiatric unit from the scope of our audit as it only represents 3% of the total psychiatric unit beds. Other interviews included staff from the Department’s Financial Services Branch, the Patient Advocate Services Co-ordinator, and the Director of Adults With Disabilities and Senior Services Branch within the Department of Family and Community Services. We conducted research and carried out documentation reviews. We also conducted audit testing (involving all seven regions) at the Mental Health Services Division. We used all information gathered to support our findings, conclusions, and recommendations that are presented in this chapter. Results in brief 4.14 We found that the performance targets currently in place are insufficient to enable the Department to properly assess the performance of the psychiatric hospitals and units. We also found that the Department is not receiving sufficient accountability reporting information from the RHAs. We concluded that, given the current level of reporting, it is not possible to properly evaluate the performance of the psychiatric hospitals and units. 4.15 Our overall audit conclusion is that the Department of Health and Wellness does not have appropriate accountability processes in place for the operations of the psychiatric hospitals and psychiatric units under the direction of the Regional Health Authorities. 4.16 We commend the Department for the recognition New Brunswick has earned for employing some best practices in mental health reform. It has achieved this recognition while focusing more on process feedback and qualitative information than quantitative information. We feel the Division’s data collection and reporting processes require improvement since both are important components of an effective accountability relationship. 4.17 We recognize the mental health system is an integrated system that includes services provided through the psychiatric hospitals and units as well as the community mental health centres. One component of the system can not be measured in isolation from the other components. Although the focus of our audit was on the psychiatric hospitals and units, the Department should consider all services within the mental health system when implementing our recommendations. 4.18 The Regional Health Authorities Act sets the stage for an effective accountability framework between the Department and the RHAs. We recognize the legislation is fairly recent and acknowledge that steps to implement the provisions of the Act are underway. Grants issued to RHAs should be managed wisely and prudently by the Department to achieve value for money in the operations of the psychiatric hospitals and psychiatric units. Accountability elements that should be in place include: defined expectations that focus on measurable results, signed agreements that state RHA reporting requirements, and a commitment by the Department to monitor results and take corrective action in cases of RHA non-compliance with the agreements. Accountability 4.19 Prior to discussing our detailed findings, it is important to highlight the term “accountability”. The three audit criteria we have chosen are all components of an effective accountability relationship. 4.20 CCAF/FCVI Inc., a national non-profit organization with more than twenty years experience in researching public sector governance and accountability, provides the following guidance: Accountability is the obligation to render an account for a responsibility conferred. ... Accountability involves an obligation to explain or justify specific actions. 4.21 The essential components of an effective accountability relationship are as follows: • define and agree on roles and expectations; • choose performance measures; • report on results; and • evaluate results and take corrective action where necessary. 4.22 We will be addressing the first two components under our first criterion and the others under our second and third criteria respectively. Performance targets and standards 4.23 Our first criterion was: The Department of Health and Wellness should have performance targets and standards in place for the psychiatric hospitals and psychiatric units. 4.24 Responsibility for the operations of the psychiatric hospitals and psychiatric units rests with the RHAs. The Mental Health Services Division funds the costs related to the clinical delivery of programs such as the observation, examination, assessment, care, treatment, rehabilitation, and maintenance of persons suffering from mental disorders. During the 2001-02 fiscal year, the Division issued $26.6 million of its total mental health budget to the RHAs for clinical costs. The mental health budget is “protected”, meaning that RHAs can only use the funding for mental health expenditures. The Division does not fund physical plant expenditures such as infrastructure and utilities nor does it fund support costs. These expenditures are funded by the Hospital Services Branch within the Institutional Services Division of the Department. 4.25 Given the amount of funding the Department provides the RHAs, and the direct effect it has on the wellbeing of the people of the Province, it is reasonable to expect the RHAs to be able to demonstrate their accountability in a clear and organized manner. Performance expectations and performance measures Defining and agreeing on roles and expectations 4.26 A first and necessary step to an effective accountability relationship is to define the roles of the parties involved, that is, what each of their respective responsibilities will be and how the relationship is to be managed. It is important to have agreement among the parties regarding expected results. They should be clear, understandable, and realistic. This enhances the commitment of the parties to the relationship and allows for parties to be held properly accountable. 4.27 The importance of realistic expectations to an effective accountability relationship should be highlighted. Without a reasonable balance between expectations and available resources, the effectiveness of the relationship is undermined. Expectations that are perceived as unreasonable or unachievable with available resources and capacity will not be taken seriously. On the other hand, meeting expectations with resources that are more than sufficient would not earn much credit; meeting expectations should require some effort. 4.28 References in the remainder of this criterion are made to McEwan and Goldner’s 2001 study entitled, “Accountability and Performance Indicators for Mental Health Services and Supports - A Resource Kit” that was commissioned by the Federal/Provincial/ Territorial Advisory Network on Mental Health. The purpose of the study was to develop a resource kit of performance indicators for provinces and territories to facilitate ongoing accountability and evaluation of mental health services and supports. Members of the Federal/Provincial/Territorial Advisory Network on Mental Health played a key role in directing the project to ensure its relevance to governments, regional health authorities, and mental health program managers concerned with performance monitoring. All provinces and territories were involved in the study. 4.29 Performance expectations can take the form of targets and standards. McEwan and Goldner define targets as commitments made in advance to achieve a stated level of performance. They went on to say that target setting should be based on past performance information, consider comparative performance data from international or national jurisdictions, reflect the input of stakeholders, and challenge the organization to strive for higher quality. 4.30 A standard is a basis for comparison or a reference point against which performance can be evaluated. Benchmarks can be used as standards. McEwan and Goldner pointed out that the concept of benchmarking involves identifying best practice or best performance in a certain area and using this as a standard for comparing local performance. Choosing performance measures 4.31 Once expectations have been set and agreed upon by all parties, the parties should then choose and agree upon performance measures. Actual performance is evaluated against performance expectations to determine if performance achieved is satisfactory. 4.32 Performance indicators are a type of performance measure. McEwan and Goldner state that performance indicators are markers or measures which convey quantifiable information about progress toward goals and objectives. They go on to say that, ideally, indicators should be compared to performance targets or benchmarks. 4.33 McEwan and Goldner found that performance monitoring efforts in most jurisdictions tend to focus on inputs and processes as opposed to outcomes when measuring and reporting on activities. They went on to say that the primary input reported and used at the political level is that of spending or what the spending purchases in terms of beds. They feel that more dollars does not necessarily produce more or better services and that analyzing spending alone does not give an indication of the volume or quality of services delivered or about outcomes achieved. 4.34 McEwan and Goldner define indicators as input, process, or outcome-based: Input Resources put into mental health care and thereby relate to the structural or organizational characteristics of a system or setting. Inputs are often expressed in terms of financial resources or numbers and types of personnel, facilities, etc. Process Key activities of a service or system in the provision of care to persons with mental illness. Commonly reported process measures are service contacts, in terms of numbers of clients, client visits, admissions, etc. Outcome Considered by many to be the most important indicator category yet it is also the most complex and challenging to measure. Outcomes reflect the total contributions of all those who fund, plan, and provide service as well as those of clients and their families. 4.35 McEwan and Goldner feel that, ideally, a clear policy logic should link inputs, processes, and outcomes and that reporting on indicators within only one category is one-sided and can be misleading. Examples of mental health outcome-based indicators (taken from McEwan and Goldner’s resource kit) for those persons served by the Province’s mental health programs include: • consumer/family satisfaction - satisfaction with level of services received; • quality of life - sense of overall fulfilment, purpose, overall satisfaction with life; • functional status - managing money, managing personal hygiene and appearance, utilizing skills such as grocery store and public transportation, maintaining a home environment; • employment status - engaging in meaningful daytime activities such as volunteer activity, maintaining a job; • housing status - living in satisfactory independent or supported housing; • financial status – earning adequate income, receiving disability benefits; and • clinical status - relief of clinical symptoms, associated distress, and degree of interference in daily life. 4.36 Measuring and reporting on outcome-based indicators will give a greater sense as to whether the services delivered met the needs of the mentally ill. Audit findings – Performance targets and standards 4.37 The performance standards the Department has in place for the psychiatric hospitals and units include accreditation standards and provincial standards of care. The only performance target in place is a financial budget comparison. There are no outcome-based performance targets by which the Department measures performance. The new Regional Health Authorities Act provides the authority for the Department to establish an outcome-based performance measurement system but there is still work to be done before it is complete. Accreditation standards and provincial standards 4.38 The RHAs are responsible for implementing the standards of care in accordance with both the Canadian Council of Health Services Accreditation Standards and the provincial standards of care. 4.39 Although it is not mandatory, all RHAs (previously Regional Hospital Corporations) in the Province are accredited every three years. The accreditation process includes a separate mental health component. According to the Canadian Council of Health Services Accreditation Standards, “The Mental Health standards allow an organization to assess and evaluate its activities in the areas of anticipating, planning, providing, and evaluating service to a population accessing mental health services in an institutional or clinic setting.” The standards are divided into nine sections that include: being a learning organization, achieving wellness, being responsive, addressing needs, empowering the clients, setting goals, delivering services, achieving positive outcomes, and maintaining continuity. The accreditation standards include standards that are outcome-based (i.e. client satisfaction, number of complaints, and whether clients achieve their set goals and expected results). 4.40 As part of the accreditation process, a team within each RHA with mental health responsibilities performs a self-assessment against the mental health standards. An independent team comprised of qualified individuals from across Canada also assesses the RHA against the same standards with information they obtain through interviews, meetings, and documentation reviews. Recommendations are made as a result so that improvements can be made. 4.41 The accreditation results are not submitted to the Division unless a request is made. Although the Division places considerable reliance on the accreditation process, it does not review the results of this process to ensure positive outcomes are being achieved in all standards by all regions. 4.42 The provincial standards for psychiatric hospitals and units can be found in Chapter XII of the Standards for Hospitals in New Brunswick, entitled Standards for Psychiatric Services in Hospitals (1998). The document states that the purpose of the provincial standards “is to advise on standards for hospitals that will promote quality care for the people of the Province.” The provincial standards have been developed as guidelines and are felt to be complementary to the accreditation standards. The Department does not have any formal means of verifying if the RHAs are in compliance with the provincial standards and whether desired outcomes (i.e. quality of life and functional status) have been achieved; instead it relies on the accreditation process, qualitative information, and open communication with the RHAs. Annual budget 4.43 The financial target in place for the psychiatric hospitals and units is the annual budget. The Division compares budgets to actual results on a quarterly basis. A budget is an input-based target which, when taken by itself, does not give an indication of the volume or quality of services delivered. Authority for establishing performance targets and standards 4.44 The Regional Health Authorities Act, which became effective on 1 April 2002, includes provisions that establish an accountability structure between the Department and the RHAs. The Minister’s authority for establishing performance targets and standards is also reflected in the Act. 4.45 Section 7(1) of the Act states that the Minister shall establish an accountability framework that describes the roles of the Minister and other government ministers and the Regional Health Authorities and that specifies the responsibilities each has towards the other within the provincial health system. 4.46 Section 9 of the Act gives the Minister the authority to establish performance targets for Regional Health Authorities. It states: The Minister may establish performance targets for a regional health authority with respect to: a) its development as an organization, b) its financial management, c) ensuring access to the health services provided by the regional health authority, d) achieving satisfactory patient outcomes, e) the level of patient satisfaction with the services provided by the regional health authority, and f) any other matter prescribed by regulation. 4.47 We were pleased to see the provisions noted in d) and e) above. These provisions could assist the Department in assessing whether the needs of the mentally ill are being met. According to McEwan and Goldner, health and non-health client outcomes relevant to the care of persons with serious mental illness are encompassed by the concept of quality of life. They state that consumers see quality of life as the ability to achieve what many others take for granted including housing, social support, meaningful activities, and an adequate standard of living. They further state that satisfaction is an indication of the extent to which services and supports meet the needs of consumers and families, and is considered a key dimension of service quality. 4.48 This new legislation gives the Department the authority to establish an outcome-based performance measurement system for the RHAs. The Act has been in effect for just over a year now. We were informed that the provincial health plan, regional health and business plans, and the accountability framework, as required by the Act, are not yet in place. The regional health and business plans and the accountability framework flow from the provincial health plan, which is currently in draft form. Recommendation 4.49 We recommended the Department develop performance targets, with a focus on outcomes, against which it can evaluate the activities of all psychiatric hospitals and all psychiatric units. Examples of the benefits of performance targets 4.50 Two situations were brought to our attention that could have been prominently identified for appropriate action by using performance targets. Shortage of community housing 4.51 During our audit work we found there is an urgent need for appropriate community housing for the mentally ill population of New Brunswick. There are patients in psychiatric unit beds that have been medically discharged and there are patients in psychiatric hospitals who have completed their rehabilitation programs. According to the Chiefs of Psychiatry representing all the health regions in New Brunswick, these patients should all be moved into the community but there are few, if any, community placements available. Housing these individuals in hospital beds limits access by others who need these programs and services. We learned that this has been a problem for approximately two years. 4.52 As of December 2002, there were forty patients in psychiatric hospitals awaiting community placement and ten patients in psychiatric units awaiting community placement. Twenty per cent of psychiatric hospital beds were being occupied by patients who should be placed in the community. We were told by the Department that the average length of stay at the active rehabilitation unit at Centracare should be approximately six to eighteen months but it is often as long as five years as a result of the housing problem. 4.53 While a patient is in the care of a psychiatric hospital or unit, the responsibility for the patient lies with the RHA. While the patient is in the care of a community mental health centre (CMHC), responsibility for the patient lies with the Division. Once the patient is placed in a community residence, the responsibility for the patient lies with the Department of Family and Community Services. 4.54 The Department of Health and Wellness is fully aware of the shortage of community housing as the Division is regularly in contact with the RHAs regarding their mental health issues and problems. 4.55 Having appropriate performance targets in place and having a public reporting of results may have highlighted the need for resolving this issue. An example of a target might be to have patients, on average, released within a specified number of days. If patients are remaining in hospital for a period extending beyond the targeted number of days, it could be an indication of a problem such as limited community placements. Having performance targets and reporting on them could bring issues such as this to light. Such types of reporting could also help to highlight and resolve problems associated with programs that cross departmental lines. Patient advocate services 4.56 A review of utilization statistics in the Patient Advocate Services Annual Report for 2001-02 shows that two regions have a high number of patient advocate cases compared to other regions of comparable size in the Province. The role of the patient advocate is to inform those patients being treated on an involuntary basis of their rights, to represent them at Tribunal or Review Board hearings, and to ensure that the Mental Health Act is appropriately applied. Region 1 (Moncton) had 400 cases and region 6 (Bathurst) had 299 cases while there were only 139 cases in region 2 (Saint John) and 129 cases in region 3 (Fredericton). The annual report noted that these numbers merit more attention and further analysis by the Department. This observation was also documented in the previous year’s annual report but a response to the issue has not been issued. 4.57 A performance system that incorporates the use of targets would bring forward such information in a reliable manner and would highlight items for necessary action. The number of cases per region should be calculated on a per capita basis to enable comparison among regions as well as to a provincial target. Conclusion 4.58 This criterion was partially met. One target was noted, the financial budget comparison, however, it does not provide information on outcome-based results. Standards for performance exist in the form of provincial standards of care, and accreditation standards. Although the provincial standards of care and accreditation standards impact directly on client care, the Department does not have a review process in place to ensure planned outcomes have been achieved in all regions. The new Regional Health Authorities Act provides the authority for the Department to establish an outcome-based performance measurement system but there is still work to be done before it is complete. The one target currently in place is not sufficient to enable the Department to properly assess the performance of the psychiatric hospitals and units. Accountability reporting information 4.59 Our second criterion was: The Department of Health and Wellness should receive sufficient accountability reporting information from the Regional Health Authorities to allow it to evaluate the performance of the psychiatric hospitals and psychiatric units. Reporting of results 4.60 Reporting of results is the third step of an effective accountability relationship. All parties in accountability relationships need to understand what information is to be reported by whom, to whom, and when. In this case, the two main relationships are between the Department and the RHAs and between the RHAs and the psychiatric hospitals and units. 4.61 The RHAs should report in a manner that allows actual results to be compared to agreed upon expectations which would enable the Department to determine if performance achieved was satisfactory. Measuring and reporting on outcome indicators, such as those listed previously, would give the Department some insight as to whether the needs of the mentally ill are being met. Explaining shortcomings in performance and the reasons behind them would also be helpful to the Department. Accountability reporting requirements currently in place 4.62 During our audit, we learned that provincial standards exist for the quality improvement, risk management, and utilization management activities of hospitals that perform psychiatric services. There is also a standard that requires a reporting system for these activities that involves the submission of reports to appropriate government divisions, sections, or departments. 4.63 We were very pleased to note the existence of these standards, particularly the standard which deals with the monitoring and evaluation of the quality and outcomes of psychiatric care and services and the standard which deals with the related reporting requirements. We were surprised to note, however, that the Department does not enforce the reporting requirements of the provincial standards of care. 4.64 During our audit, we found an abundance of performance information being generated at the regional level that is not being submitted to the Department. This includes performance indicators, quality improvement reports, quarterly reports to the Board, accreditation results, and annual mental health reports. The Boards of Directors of the RHAs utilize this information in the performance review process. The RHAs have not been asked to submit the performance information to the Division but some are doing so without being requested. 4.65 The Department has psychiatric services agreements with the RHAs that stipulate accountability reporting requirements. Among them is the requirement that RHAs are to implement the standards of care in accordance with the provincial standards. Recommendation 4.66 We recommended the Department ensure the reporting requirements of the psychiatric services agreements are followed so that it receives appropriate reporting on the quality and outcomes of psychiatric care and services as set out in the provincial standards of care. 4.67 In addition to the requirement that RHAs implement the standards of care in accordance with the provincial standards, the agreements contain four other accountability reporting requirements. Compliance with these requirements is not enforced and the requirements are not sufficient to allow performance measurement. Although the four other reporting requirements are not what we expected to see in terms of good accountability, we will address them in detail and provide some recommendations for improvement. The requirements are as follows: • Regional health authorities must submit monthly financial statements and statistical information to the Mental Health Services Division no later than thirty calendar days from the end of the reported month. The reporting of financial and statistical information must identify only costs related to the clinical delivery of programs funded by the Division. • Quarterly utilization reports must be submitted to the Division. • Upon completion, reports on Suicide Internal Review must be forwarded to the Assistant Deputy Minister of the Mental Health Services Division. • Reporting requirements in the budget letter from the Minister of Health and Wellness are to be followed (each year a new budget letter is prepared). The Minister’s budget letter includes the following reporting requirements: a) Financial statements and statistics must be submitted no later than thirty calendar days from the end of the reported month to Hospital Services. b) There must be quarterly electronic submissions of financial and statistical data through HFUMS (Hospital Financial Utilization Management System) thirty days after the close of the quarter being reported. RHA compliance with current reporting requirements Financial statements 4.68 Only four RHAs out of eight have been submitting their psychiatric hospital and psychiatric unit financial statements to the Division during 2002-03. Three of the four submit their statements quarterly while one submits them monthly. Most statements are received more than thirty days after the quarter end. The result is that only one RHA out of eight is complying with the monthly reporting requirement. 4.69 One RHA that does not submit financial statements to the Division surprised the Division in February 2003 with a large deficit. A call that was made to the RHA the previous month did not identify this looming deficit. This shows the danger of RHAs not supplying financial information as required. Recommendation 4.70 We recommended the Department ensure financial statements are submitted by the RHAs to the Division in accordance with the frequency and timing set out in the psychiatric services agreement. 4.71 We also found that the financial statements show costs related to the clinical delivery of programs as required, but they do not adhere to a common format. Of the four RHAs that regularly submit financial statements to the Division, three RHAs break the costs down by type and one RHA reports only total costs. This makes comparability of financial statements difficult. Recommendation 4.72 To enhance comparability, we recommended the Department devise a common format for RHAs to follow for the financial statements submitted to the Division. 4.73 RHAs also electronically submit financial and statistical information to the Department on a quarterly basis to be uploaded to the Hospital Financial Utilization Management System (HFUMS). A database within the HFUMS contains financial and statistical information on the RHAs. Some RHAs feel the Division should access the HFUMS as it contains detailed financial and statistical information for the RHAs. However, no one in the Division is registered to access the HFUMS. 4.74 If the Department produced reports directly from the HFUMS, this could have the potential of eliminating the inefficiencies, inconvenience, and duplication involved with RHAs submitting multiple copies of manual financial statements in different formats to different branches of the Department. Financial statement users in the Department could have access to current and complete financial information and it would minimize the likelihood of unexpected developments. Recommendation 4.75 We recommended the Department investigate the possibility of updating the HFUMS on a monthly basis and using it as the source of the required monthly financial statements from the RHAs. Utilization reports and statistics 4.76 Utilization reports for the psychiatric units are manual forms that the RHAs must submit quarterly to the Division presenting such information as admissions, separations, occupancy rates, average length of stay, number of patients, total inpatient days and re-admissions. A common form has been designed by the Division for this purpose. There are no targets or standards incorporated into these forms for comparison purposes. Senior management told us that these reports do not provide much information by themselves. They provide information on outputs or processes as opposed to outcomes. 4.77 While six out of eight psychiatric units submitted their utilization reports quarterly during 2001-02, only four out of eight have been doing so during 2002-03. Three of these psychiatric units submit their reports on time. Staff at one of the psychiatric units, that does not submit utilization reports, told us they had not been asked to submit them since the summer of 2002. The reason given for not submitting them was that the Division’s format did not coincide with the psychiatric unit’s format. Another psychiatric unit said they have been submitting their reports but the Division has no record of having received them. 4.78 Utilization reports for the psychiatric hospitals are also manual forms. We noted that these forms indicate that utilization reports are to be submitted monthly while the psychiatric services agreement states that they are to be submitted quarterly. The reports present such information as use of beds by unit, admissions, re-admissions, deaths and discharges. As is the case with the reporting for psychiatric units, a common form has been designed for this purpose that does not incorporate targets or standards for comparison purposes. 4.79 Both psychiatric hospitals have been very diligent in submitting their utilization reports to the Division on a monthly basis and on time, although this was not the case for one of the hospitals prior to February 2002. Recommendation 4.80 We recommended the Department ensure utilization and statistical reports are submitted by the RHAs in accordance with the requirements of the psychiatric services agreement. 4.81 Although the HFUMS contains mental health statistics, we were informed that the Division does not access these statistics and it is unaware of the full statistical capabilities of the HFUMS. It would be important to know whether the HFUMS contains all the desired mental health statistics as required by the Division and Hospital Services. 4.82 This could eliminate the inefficiencies and duplication involved in having RHAs prepare and submit manual utilization and statistical reports to more than one branch of the Department. This could also eliminate the inefficiencies involved with Division staff entering the information from the manual reports into their system. Recommendation 4.83 To eliminate the need for regional submission of manual utilization and statistical reports to the Department, we recommended the Department determine if the HFUMS contains the mental health statistics that would meet the statistical reporting needs of both the Mental Health Services Division and Hospital Services. Suicide Internal Review Reports 4.84 The psychiatric services agreement states that Suicide Internal Review reports are to be forwarded to the Assistant Deputy Minister of the Mental Health Services Division upon completion. 4.85 We were informed that the only Suicide Internal Review reports forwarded to the Department are those prepared at the CMHCs. When a suicide occurs within a psychiatric hospital or unit, the unit manager informs the Department by phone and the RHA performs a review. We were told that if the Department wants to receive copies of Suicide Internal Review reports, the RHAs must comply with the request. Due to the personal and confidential nature of these reports, they are not kept by the Department. Copies sent to the Department are destroyed once the Department is finished with them. Of the three regions visited during our audit, only one had any suicides in the past six years. Recommendation 4.86 We recommended the Department ensure that the process for communicating Suicide Internal Review reports from the RHAs to the Division is conducted in accordance with the psychiatric services agreement. Other accountability reporting issues Sufficiency of current accountability reporting requirements 4.87 Funding should be linked to performance. The RHAs should be expected to demonstrate how their actual performance compared to what was expected. They have a duty to report both the financial and non- financial results they have achieved in relation to the authority they have been granted and the public funds entrusted to them. 4.88 As noted previously, the accountability reporting by the RHAs to the Department is not sufficient to allow performance measurement. We feel the Department should be receiving better accountability reporting information from the RHAs. Examples of sufficient information might include reporting actual performance compared to pre-established targets and standards (this would include the measurement of performance indicators) as well as reporting on the quality improvement, risk management, and utilization management activities as noted in the provincial standards of care. The psychiatric services agreement could be a useful tool in establishing mutually agreed and understood expectations as well as setting out and clarifying the accountability reporting requirements. Recommendations 4.89 We recommended the Department improve the accountability reporting requirements of the RHAs to enable it to properly evaluate the performance of the psychiatric hospitals and units. 4.90 We recommended the Department consider using the psychiatric services agreement as a means of identifying and enforcing improved accountability reporting requirements. Agreement on performance expectations 4.91 The General Regulation – Mental Health Act, states that “The administrator of a psychiatric facility shall furnish such returns, reports and information to the Department as the Minister considers necessary.” Legislation requires RHAs to abide by the reporting requirements of the Minister. 4.92 Signing of the psychiatric services agreement is the step that links the legislative requirements to the RHAs. The psychiatric services agreement is only signed by the Division. It is important to have mutual agreement on the expectations of all parties. Having RHAs also sign the agreement is a sound business practice. Recommendation 4.93 We recommended that each psychiatric services agreement be signed by both the Division and the RHA to ensure mutual agreement and understanding of expectations. Conclusion 4.94 This criterion was not met. The accountability reporting requirements the Department has in place for the RHAs are not sufficient to allow the Department to properly evaluate the performance of the psychiatric hospitals and units. The only reporting required pertains to the submission of financial statements and utilization reports. Reports of financial performance and operational performance are not routinely linked and the depth of information required to make such a comparison is not currently available. RHAs are not expected to demonstrate how actual performance compared to expectations with the exception of the budget to actual comparison. Performance evaluation and corrective action 4.95 Our third criterion was: The Department of Health and Wellness should evaluate the performance of the psychiatric hospitals and psychiatric units and take corrective action where necessary. Evaluating results and taking corrective action 4.96 Evaluating results and taking corrective action is the fourth step of an effective accountability relationship. Effective accountability not only involves reporting of performance but also evaluating the performance and taking appropriate corrective action where necessary. 4.97 Evaluating performance involves comparing actual performance with agreed upon expectations. The use of performance indicators allows the comparison of results with targets, standards, or benchmarks. Both achievements and failures should be recognized and feedback on performance should be provided to the individuals responsible for that performance. Performance evaluation is an integral part of the accountability process as it provides an ongoing means of determining whether satisfactory performance levels have been achieved. 4.98 Corrective action is the process of addressing and rectifying unsatisfactory performance. It could involve such initiatives as modifying unrealistic or simplistic performance expectations, making appropriate program adjustments, and setting appropriate consequences for those responsible for performance (whether they are rewards based or penalty based). To properly hold those responsible to account, effective reporting, evaluation, and adjustment must be occurring. The Department’s evaluation process 4.99 According to the Department, the responsibilities of the Division (with respect to the performance of the CMHCs, psychiatric hospitals and units) include: • defining priorities for service development and implementation in accordance with the provincial mental health policy; • defining and ensuring implementation of core programs and service standards; • defining and monitoring expected outcomes for all levels of service; • ensuring the full implementation of the requirements of the Mental Health Act and the Mental Health Services Act; • allocating financial and human resources and monitoring their use; and • directly managing the Community Mental Health Centres and ensuring the effective fulfilment of psychiatric services agreements with RHAs for in-patient services (psychiatric units and hospitals). 4.100 Despite the fact that these responsibilities have been assigned, the Department does not have a formal evaluation process for the performance of the psychiatric hospitals and units. As noted earlier, the Department is not receiving sufficient accountability reporting information on the performance of the psychiatric hospitals and psychiatric units. As a result, it is not possible to properly evaluate performance. In this section, we identify the Department’s current management and performance evaluation processes for the psychiatric hospitals and units and the tools used for performance evaluation. Current management processes 4.101 The mental health system in New Brunswick is well known across Canada for employing some best practices in mental health reform. It earned this recognition while employing the same or a similar management style that is in existence today. As part of the reform process, the Division relied heavily on qualitative information as opposed to quantitative information. Although the Division recognizes that both are important, it has focused more on processes and has been less aggressive with data collection and reporting. 4.102 Generally, the RHAs monitor themselves and inform the Division of issues, challenges, and pressures they are facing with respect to the psychiatric hospitals and units. The Division is in regular contact with the RHAs via telephone, email, and face-to-face meetings. The Division considers itself to be a source of support for the RHAs on a continuous basis. 4.103 The Division holds regular meetings with regional staff regarding the performance of the psychiatric hospitals and units. Problems are often shared and addressed at these meetings. 4.104 All RHAs are required to have a Management Liaison Committee with representation from psychiatric services of the RHA as well as the CMHCs. These committees operate separately from the Department. Committees meet on a periodic basis to ensure effective co- ordination among mental health programs, to jointly identify strategies to intervene with specific target groups, and to monitor overall service utilization and outcomes. These committees, by working together, resolve regional mental health issues and/or pressures. Committees are in regular contact with the Department to keep them abreast of any pressures, challenges, or opportunities they are facing. The Management Liaison Committee is a collaborative management structure that helps to ensure the continuous care of the individual. Performance indicators Provincial performance indicators 4.105 During our audit work, we noted that a draft document has been prepared by the Department in an effort to identify key performance indicators for mental health. The document, entitled “MHS Performance Indicator Working Group Working Document”, was updated as recently as 16 November 2001 but the indicators have yet to be adopted. The document identifies seventeen possible indicators and discusses potential means of measurement. The seventeen indicators cover the entire mental health system which includes the CMHCs, the psychiatric hospitals, and the psychiatric units; however, many of the indicators are directed at the CMHCs. The document includes four input indicators, seven process indicators, and six outcome indicators. 4.106 The Department has not developed targets or standards with which to compare the indicators. The logical flow of steps in the accountability process is to first define and agree on roles and performance expectations (which includes performance targets and standards) and then to choose performance measures (which includes performance indicators). It is premature to choose performance measures before performance expectations are known. Recommendations 4.107 We recommended the Department adopt a common set of mental health indicators that cover the performance of all operational sectors of mental health. 4.108 We recommended that the indicators have a clear linkage with organizational goals and pre-established targets and standards. Data collection for provincial performance indicators 4.109 We were informed the performance indicator working document remains in draft form due to data collection restraints. While information for some indicators is available now, a management information system is required to enable the Department to obtain information on others. For example, information is needed for resources used versus outputs attained for the CMHCs. A feasibility study was recently conducted and feedback will be brought before the Department’s Management Committee in the near future. 4.110 Hospitals have their own information systems that enable them to measure their own indicators. As noted previously, the hospitals electronically submit financial and statistical information to the Department. We learned that the Division is aware of the financial data that is available in the system but not what statistics are available. A properly designed information system that includes data required for appropriate performance measures is essential for the Department. Recommendation 4.111 We recommended the Department implement systems that are capable of generating information to support the measurement of mental health performance indicators. RHA performance indicators 4.112 RHAs are required to have performance indicators as noted under the provincial standards; it is not an option. The Canadian Council of Health Services Accreditation Standards requires that indicators be selected to monitor the goals, objectives, and desired results or outcomes of the mental health program within the RHA. 4.113 Performance indicators are currently used by the RHAs. These indicators have been developed independently by the RHAs and each carries out its own measurement procedures. The indicators being measured are not consistent from region to region. All three regions visited compare their performance indicators to expectations. While the total number of indicators varied from region to region, we noted that several of them are outcome-based indicators. Because the indicators vary from region to region, a composite benefit of the information generated by the individual processes cannot be realized at the provincial level. 4.114 The Department could look to the RHAs for examples of performance indicators currently in use for the psychiatric hospitals and psychiatric units (e.g. consumer/family satisfaction with the program which could be calculated in terms of total individuals satisfied as a percentage of total individuals surveyed). Psychiatric Services Agreement 4.115 The Department’s primary tool for ensuring the flow of information, which facilitates evaluating the performance of the psychiatric hospitals and units, is the psychiatric services agreement. However, as noted earlier, many requirements of the agreement are not enforced. Financial Statements and Utilization Reports 4.116 We noted earlier that the Department’s primary requirement is that RHAs submit their financial statements and utilization reports to the Division quarterly. If attempts to obtain this information are unsuccessful, the Division formulates its projections using the information that is available. 4.117 Financial statements are reviewed by the Division on a quarterly basis to determine how actual financial performance of the psychiatric hospitals and units is faring compared to budget, particularly with regard to the overall surplus or deficit. The Financial Services Branch uses these financial statements to accrue regional surpluses and deficits and to highlight items for the Division to follow up on. 4.118 Utilization reports present statistical information on the psychiatric hospitals and units. They are reviewed quarterly, primarily for diagnostics and occupancy rates. 4.119 It is a fragmented approach to examine financial information without also examining operational information. Because a RHA is under budget or over budget, does not mean its performance is satisfactory. It could be over budget and provide excellent service or it could be under budget and provide poor service. There are similar concerns in only using utilization information. For example, the average length of stay may be short but it could be the result of discharging patients before they are ready, resulting in compromised quality of care and possibly re-admissions. The Division will not be able to properly evaluate the performance of the psychiatric hospitals and units using this information in isolation. Recommendation 4.120 We recommended the Department incorporate both financial and operational performance information into the performance evaluation process of the psychiatric hospitals and units. Accreditation process 4.121 The Department relies on the accreditation process as a means of ensuring quality standards of care. Although considerable reliance is placed on the accreditation process, we were surprised to learn that accreditation results are not submitted to the Division unless requested. The Division does not review the results of the accreditation process to ensure positive outcomes are being achieved in all standards by all regions nor does it compare the accreditation results by region or report on the results of the accreditation process on a province-wide basis. Recommendations 4.122 We recommended the Department require all RHAs to submit their mental health program accreditation results to the Division. 4.123 We recommended the Division utilize the mental health program accreditation results as a tool in evaluating the performance of the psychiatric hospitals and units. Provincial standards 4.124 The Department relies on the RHAs to “monitor themselves” regarding the provincial standards of care. If the requirements of the provincial standards of care were actually enforced, the RHAs would be submitting the necessary performance information to the Department for the performance evaluation process. Recommendation 4.125 In evaluating the performance of the psychiatric hospitals and units, we recommended the Department utilize the performance information identified as a requirement in the provincial standards of care. The Department’s approach to corrective action 4.126 If the Department learns the performance of a psychiatric hospital or unit is unsatisfactory, it will meet with the RHA to address the problem or discuss it over the telephone. Often, the discussions are financial in nature, for example, if the RHA has a large deficit. Since the Division has not established expectations for performance, with the exception of the budget, RHAs are not held accountable for falling short of non-financial expectations (e.g. unacceptable re-admission rates, unusual average length of stay, number of complaints, and consumer satisfaction rates). Recommendation 4.127 Once performance targets and standards have been established for the RHAs, we recommended the Department take corrective action where actual performance falls short of expectations. Reporting of results of performance evaluation process Departmental annual report 4.128 The Department measures and reports publicly on ten performance indicators in its annual report but only one relates to the psychiatric hospitals and units. This indicator shows the number of patient days of hospitalization for all psychiatric hospitals and units combined, with a year-by-year comparison and a target. This same indicator also presents information on the number of referrals to community mental health centres (CMHCs). The Department’s interest in this indicator results from the shift to community services. It is expected that patient days of hospitalization in psychiatric hospitals and units would decrease, and that more demand would be put on CMHCs to provide alternative service. 4.129 This indicator is a process indicator as opposed to an outcome indicator; it gives no indication as to whether the needs of the seriously mentally ill are being met or whether quality service is being provided. Also, by reporting all psychiatric hospitals and units combined, it is impossible to highlight regional problems. Recommendation 4.130 We recommended the Department report comprehensive performance indicators for psychiatric hospitals and psychiatric units in its annual report. Patient Advocate Services annual report 4.131 Patient Advocate Services prepares its own annual report that presents several statistics such as the number of patient advocate cases, number of admissions, and number of tribunal/review board hearings by region. From this, we were able to identify, for example, that two regions are using the services of the patient advocate much more than other regions of similar size. By reporting performance information by region, this annual report is a useful source of information to the Department which could be used in assessing performance and highlighting problems. Recommendation 4.132 We recommended the Department utilize the Patient Advocate Services annual report as a source of performance information in evaluating the performance of the psychiatric hospitals and units. RHA annual reports 4.133 According to the Regional Health Authority Act, RHAs are required, in their regional health authority annual reports, to report on their performance in relation to the performance targets set by the Minister. Performance targets referenced under the Act have not yet been implemented for Hospital Services or Mental Health Services. We examined all current RHA annual reports and noted that they present very little if any information on mental health performance and mental health performance targets are not used. Recommendation 4.134 Once the Minister establishes and implements RHA performance targets, we recommended the Department ensure the RHAs report on their performance in relation to these performance targets in their annual reports. Conclusion 4.135 This criterion was partially met. The Department is not receiving sufficient accountability reporting information on the performance of the psychiatric hospitals and psychiatric units. As a result, it is not possible to properly evaluate performance. 4.136 With the existence of the draft mental health performance indicators document, the potential exists for an improved performance evaluation system. We also noted the existence of several good performance reporting and evaluation processes within the regions. Departmental response 4.137 The Department provided the following response to our report: Thank you for the Audit Report on the Accountability of Psychiatric Hospitals and Psychiatric Units. I found the report to be accurate and we are in general agreement with your recommendations. As a result of the extensive background information provided we will be able to use the report with key stakeholders in the mental health system to meet our goal of reporting expenditures and outcomes in a comprehensive and transparent manner. Your acknowledgement that mental illness is found in one-quarter of all general medical practice in Canada and that 12.5% of Canadians can expect to be hospitalized for mental illness at least once in their lifetime underscores the importance of an effective and efficient mental health system. In New Brunswick we are committed to a balanced network of institutional and community based mental health services that ensure timely delivery of the most appropriate and least restrictive mental health services. The scope of the audit was defined, “To assess whether the Department of Health and Wellness has appropriate accountability processes in place for the operations of psychiatric units under the direction of Regional Health Authorities.” The first criterion of the report assessed the degree to which the mental health services (MHS) division has performance targets and standards in place. It was determined that the MHS division partially met this criteria and later the report commended New Brunswick’s mental health system’s national reputation and management style in the employment of some of the best practices in mental health reform. It was acknowledged that the management style relies on qualitative information and open and integrated communication across all service delivery sectors. The report went further and underscored the need for a balance between qualitative and quantitative information and recommends that an outcome- based performance measurement system, which was lacking, be realized. We are in agreement with the recommendation that the Department develop performance targets, with a focus on outcomes not only for the psychiatric units and hospitals but the entire mental health system. The report acknowledges that work in this area has been on-going and the foundation pieces are in place. The second criterion of the audit assesses reporting of information and recommends improvement in both the data collection and reporting processes. We are in agreement with the recommendations to receive consistent, timely, relevant and non-redundant data from all RHAs, ideally in electronic format, and will begin work on improving these processes. The third criterion of the report covers the interpretation of performance data and corrective action where necessary. In the absence of performance measures providing the required data for interpretation as noted under the first criteria, the mental health system has successfully relied on an integrated system with open communication and sharing of information. There are consistent provincial and regional meetings of MHS directors, head nurses and chiefs of psychiatry. Most importantly on a regional level there are on-going Management Liaison Committee meetings where both the Community Mental Health Centres and the in-patient psychiatric services meet and work together to monitor overall service utilization and outcomes, and resolve regional mental health issues and/or pressures. These committees remain in regular contact with the Department’s central office and keep them abreast of any pressures, challenges or opportunities. As stated, we agree with the report’s findings and recommendations and are striving to implement an effective accountability framework that reports our expenditures and outcomes in a comprehensive and transparent manner. We have already begun work in this area, and are committed to continuously improve both the accountability and service delivery system in a responsible manner to the benefit of New Brunswickers.