Evaluation Proposal for Multilingual Eye Health Public Education Videos

Program Description and Literature Review

Background

According to CNIB statistics, (1) only one-third of working-age Canadians with vision loss are employed, (2) seniors with vision loss are three times more likely than those with good vision to experience clinical depression, and (3) people with vision loss are at high risk of social isolation, reduced community participation, falls and associated morbidity and mortality such as hip fractures and death (CNIB, n.d.).  Vision loss can lead to other major losses in life, including job, relationships, driver’s license, and independence.  Hence, preserving vision and eye health is not just important to each individual, but to society overall. 

Anecdotally, as a practicing ophthalmologist, I have noticed many patients who speak limited English present with end-stage blinding eye disease.  I am of Chinese ethnic background and am fluent in Cantonese and Mandarin.  Through speaking to those Chinese patients who present with end-stage eye disease, it seems that this is either because they do not know how to navigate our healthcare system or do not know what symptoms to look for.  Sadly, some end-stage eye diseases are not salvageable, and for others, treating advanced disease is often associated with poor outcomes as a result of higher treatment risks and complication rates, longer recovery time for the patient, and higher costs to our healthcare system.

Objective evidence is also consistent with these anecdotal findings.  Health Canada reports that language barrier is a major hindrance to healthcare access in Canada, may provide limitation to the quality of care and patient satisfaction, and is associated with higher costs to our healthcare system (Health Canada, 2003). 

The need

Canada is highly multicultural.  According to the 2011 Census, nearly 6.6 million people or 20.0% of Canada's population reported speaking a language other than English or French at home (Statistics Canada, 2012). 9 out of 10 of these individuals reside in a metropolitan area: Toronto, Montreal, Vancouver, Calgary, Edmonton and Ottawa.  1.8 million of these people live in the Greater Toronto Area (GTA) and one-third of them speak one of Cantonese, Punjabi, Mandarin, Urdu and Tamil (Statistics Canada, 2012). 

The number of visible minorities and those who speak limited English or French will continue to grow.  The top 5 source countries of permanent residents admitted in 2013 are: China, India, Philippines, Pakistan and United States (Citizenship and Immigration Canada, 2014).  Statistics Canada projects by 2031, almost half of the population over 15 years old will be foreign-born or have at least one foreign-born parent and the number of visible minorities will likely double, with majority of them living in one of the Canadian metropolitan area (Statistics Canada, 2010).   

Most of the public education materials currently available in Canada are in one of our official languages, English or French.  With this trend of increasing ethnic diversity in Canada, there will be a need to provide education materials in multiple languages.

Economic analysis

According to The Cost of Vision Loss in Canada report published in 2009 by the CNIB and Canadian Ophthalmological Society,

·         The real financial cost of vision loss in Canada is estimated to be $15.8 billion for 2007, or 1.19% of Canada’s gross domestic product.

o    This breaks down to $500 for every Canadian or $19,370 for every Canadian with vision loss in 2007.

o    The real financial cost consists of:

§  Indirect costs (lost productivity, care and rehabilitation, and others) of vision loss of $7.2 billion

§  Direct costs (health-related) of $8.6 billion

·         The net cost of suffering (also known as the burden of disease) due to vision loss, over and above financial costs, is estimated to be a further $11.7 billion in 2007.

·         Vision loss has the highest health care costs (direct costs) of any disease category in Canada, costing Canadians much more than diabetes, all cancers or cardiovascular disease.

So who pays for the costs of vision loss?  The financial costs breakdown is as follows (CNIB & Canadian Ophthalmological Society, 2009):

·         Individuals with vision loss ($3.5 billion)

·         Family and friends ($474 million)

·         Federal government ($2.4 billion)

·         Provincial and territorial governments ($6.3 billion)

·         Employers ($141 million)

·         Society and others ($3.0 billion)

In essence, the largest financial costs come from taxpayers, as federal and provincial governments bear 55.3% of the costs, and all of society bears a further 18.7%.

Based on a rapidly aging population in Canada and the fact that the top three causes of blindness are all age-related, in the next 25 years, the number of Canadians with vision loss is projected to double (CNIB & Canadian Ophthalmological Society, 2009).  With increase in immigration, any intervention will also need to be culturally sensitive and offered in multiple languages.  Hence, it is more important than ever to place eye health as an important public health issue with culturally-appropriate health promotion in multiple languages. 

The proposed program

The multilingual eye health public education videos (MEHPEV) project consists of producing a series of eye health patient education videos in the top non-official languages spoken in Canada, as well as in English and French.  Videos will be hosted on the University of Toronto Department of Ophthalmology and Vision Sciences (DOVS) YouTube channel and will be available on the DOVS website as education tools for those who speak limited English or French.  Awareness of the videos will be promoted through various ethnic media (television (TV), radio, newspaper, websites), public and private immigrant organizations, and through healthcare professionals. 

Target population

The target audience, to which the MEHPEV will be promoted, is a vulnerable population consisting of those who speak one of the top non-official languages in the GTA.  The number of languages to be included will depend on the funding amount raised for the program, starting with the top spoken languages.  The MEHPEV promotion, and hence the target population, may be expanded to the rest of Canada starting with the other metropolitan cities if the project is proven to be successful locally and a transferability evaluation deems it to be useful. 

The top 10 languages spoken in GTA are: Cantonese, Mandarin, Spanish, Urdu, Punjabi, Tamil, Persian (Farsi), Russian, Italian, and Portuguese.

Video content

For each language, the topics for the videos will start with top 3 causes of blindness (cataracts, glaucoma, and age-related macular degeneration) and 2 other common eye diseases (dry eyes and floaters).  Information on the basic workings of the eye, importance and frequency of regular eye examinations will be incorporated as well as education on the various eye health professionals and how to access eye care.  If there is sufficient funding, additional topics will be added in the future, including other eye diseases, strategies to reduce risk of eye disease and injury, and peri-operative information and instructions. 

Format

Online videos were selected as the media of choice for a number of reasons:

1)     Video was chosen over printed material as those with eye diseases may not have good vision and can rely on the voice component of the videos.  This also has the added benefit of reaching those who are illiterate as well. 

2)     Online format was chosen over one time or intermittent TV or radio broadcasts to allow for long term availability and ease of access. 

3)     Online video format is more cost efficient compared to broadcasting educational videos of several minute duration on TV or radio. 

There are limitations to online video format, as some elderly individuals may not be familiar with internet usage, but it is hoped that some of them will have family or friends who can assist them with access and the ethnic community organizations we are partnering for the video dissemination will have human or technical resources to facilitate access to the videos as well.  Hopefully these and other mitigating measures will help to allow the online videos format to strike a balance between cost, program longevity and ease of access. 

Why is this proposed program of interest to me?

The MEHPEV program started as a personal project.  Each time I see a patient who presents with end-stage eye disease, my heart sinks.  As many of these patients are elderly and speak limited English, they remind me of my grandparents.  I wish I could have helped them earlier, prevented them from losing their vision or having to go through difficult treatment course with higher risks, and I am sure their family would have wished that for them too. 

I entered medicine not to just be a clinician.  I have always had a passion for education.  I am currently actively involved in education of medical trainees, ophthalmologists both locally and internationally, and other healthcare professionals.  I would like to expand my education and volunteer interests to help the general population.  I think that international outreach is highly needed, valuable and encouraged and I am involved with international work.  However, I see a huge need locally right at home.  Even though we have one of the best healthcare systems in the world and we are so fortunate to live in a first-world country, I see many cases of third-world end-stage eye diseases in my practice in GTA.  Hence, I feel very strongly that we should help ourselves right at home and started this program.  

I envisioned this program to require 3 components/stages:

(1)   Obtaining funding

(2)   Video production

(3)   Video dissemination

Project goals

The goals of this project are to:

1)     Reduce the number of patients who speak one of the major non-official languages in GTA presenting with end-stage eye disease (secondary prevention)

2)     Promote healthy lifestyle and eye disease risk reduction (primary prevention)

3)     Save healthcare spending costs through prevention, and early detection and early treatment

How will this program be achieved?

•       Funding

•       Application to governments, hospitals, and granting agencies submitted

•       Seeking individual donations ongoing

•       Seeking corporate sponsorships ongoing

•       Dissemination

•       Media: multilingual TV, radio, newspaper, websites

•       Toronto Public Health in current collaboration

•       Community groups

•       Physician groups (ophthalmologists, family physicians)

Below is a sample video series on cataracts produced in English, Cantonese and Mandarin which was filmed unscripted without any rehearsal.  With funding, the final videos will be more professional with standardized script, rehearsal and computer graphics and animation of eye anatomy and surgery. 

https://youtu.be/MapvrOXJ1Eg (English)

https://youtu.be/QFZJGbm-ooc (Mandarin)

https://youtu.be/gzoAG5g8C_A (Cantonese)

Similar health education programs

Health promotion is “the combination of educational and environmental supports for actions and conditions of living conducive to the health” (Green & Kreuter, 1991, p. 4) and health education is “any combination of learning experiences designed to facilitate voluntary behaviour conducive to health” (Green & Kreuter, 1991, p. 17).  Kok, van den Borne, and Mullen (1997) defined health promotion as a matrix of goals and means.  The health promotion matrix shown in Figure 1, with the goals of health promotion across the top and the means to reach these goals down the left hand column, provides a clear visual relationship between health promotion, health education and prevention (Kok, 1997).  The goals of health promotion are further defined as:

(1)   Positive health – promoting healthy lifestyle

(2)   Primary prevention – risk reduction to prevent onset of disease

(3)   Secondary prevention – early detection and early treatment of disease to control progression

(4)   Tertiary prevention – care of patients with acute and chronic disease to minimize the impact of disease on patient’s function, longevity and quality of life

	Positive health	Primary prevention	Secondary prevention	Tertiary prevention
Education	X	X	X	X
Means for action	X	X	X	X
Pricing	X	X	X	X
Regulation	X	X	X	X

Figure 1.  Matrix of health promotion, health education and prevention

Health education is thus the voluntary behaviour-focused component of health promotion.  Its impact is often only successful when the other means are in place as well.  In the case of the MEHPEV, means for action exists as patients are able to access eye care by an ophthalmologist, through referral from their family physician or optometrist.  However, many patients are not aware of this or the differences between the different healthcare professionals.  Ophthalmologists are physicians and surgeons specializing in the diagnosis and treatment of eye diseases.  The training involves an undergraduate degree, medical school (4 years), and ophthalmology residency training (5 years).  As a specialist physician, referrals are needed from a family physician or optometrist before a patient can be seen by an ophthalmologist.  Consultations and visits with an ophthalmologist are covered by Ontario Health Insurance Plan (OHIP).  Optometrists are not physicians and specialize in vision assessment, prescribing and fitting glasses and contact lenses.  No referral is required to see an optometrist, and annual visits to optometrists are covered by OHIP for those 18 years old and under, 65 years old and older, and in individuals with certain health conditions such as diabetes and high blood pressure.  For others, a typical visit with an optometrist in 2015 can range from $80 to $150.  For those with extended health plans, this cost may be partially or fully covered.  So in addition to providing health education on eye diseases, educating patients about how (the means of action, pricing and regulation) they can seek medical attention is valuable as well. 

            “Vision 2020: The Right to Sight” is a global initiative launched by the World Health Organization (WHO) and International Agency for the Prevention of Blindness in 1999 with the aim to eliminate avoidable blindness by the year 2020 (World Health Organization, n.d.).  The action plan for Vision 2020 has been updated and replaced by the “WHO Global Action Plan for the Prevention of Avoidable Blindness and Visual Impairment 2014-2019: Towards Universal Eye Health” (World Health Organization, 2013).  Several countries, including Australia, developed comprehensive national framework implementation plan that includes government funding, educational programs, health programs and healthcare professional training (Australian Health Ministers’ Conference, 2005).  However, there is no publicly available information on the evaluation strategy utilized for this national initiative.  In fact, no specific performance indicators or tracking mechanisms were introduced until the 2014 implementation plan, which included information regarding funding, key priority areas, indicators and other progress measures (Australian Government Department of Health, 2014).  Future progress report to the Australian national framework may provide more information on the program evaluation strategies utilized.  The evaluation strategy used to assess the public education component of the Australian national framework may provide insights that can help with refining the evaluation strategy used for the MEHPEV program. 

To date, no government-sponsored national eye health initiative has been started in Canada and through extensive literature review I was unable to find program evaluation information on similar multilingual public education program or eye health education program.

Purpose of the Evaluation

In general, a program is only useful if it achieves its intended goals.  A program may be well-funded, have a wealth of human resources, and even research-proven interventions, but may still be unsuccessful in achieving its goals if it has the incorrect target population or ecological context.  In the real world, there is limited funding and human resources, so it is even more crucial to ensure programs are effective and efficient.  Hence, program evaluation and program development complement each other and ideally evaluation should be incorporated from the program-planning stage. 

 The Australian national framework for action to promote eye health and prevent avoidable blindness and vision loss (Australian Health Ministers’ Conference, 2005) is an excellent example of this.  The initial 2005 framework and the 2008 progress report show a highly admirable, comprehensive plan with funding and multiple programs at local, regional and national levels (Eye Health Working Group of the Australian Health Ministers Advisory Council, 2008).  “Unfortunately, as the national Framework did not contain specific performance indicators or tracking mechanisms, it fell short of meeting its goals” (Vision 2020 Australia, n.d.).  Evaluation strategies have been introduced in the 2014 implementation plan and future reports will likely be able to provide further details on evaluation type and outcomes. 

The MEHPEV project is currently in the program-planning stage.  The purpose of evaluation of the MEHPEV is two-fold:

(1)   Constructive – Given the MEHPEV project is the first of its kind, it will be useful to know how to improve the program to make it useful and effective as more languages and video topics are included.

(2)   Conclusive – Since the MEHPEV project was started by volunteers without any initial funding, knowing whether it is useful is important to assist volunteers in deciding whether to continue seeking funding and providing volunteer hours or to abandon the project and devote financial and human resources to other tasks.

Chosen Evaluation Type & Outcome Evaluation Approach

The MEHPEV would benefit most from holistic effectuality evaluation, which is a stakeholder-centered theory and methodology designed to achieve real-world outcome evaluation (Chen, 2014, 266).  In holistic effectuality evaluation, “evaluators integrate the dynamic nature of the intervention program in a community stakeholders’ views and practices with existing scientific methods to develop indigenous concepts, theories, and methodologies for program evaluation” (Chen, 2014, 266).  This form of hybrid evaluation allows for initial constructive assessment of the MEHPEV for its coherence.  Subsequently conclusive assessment of the educational videos and its adjuvants can be performed.  Throughout the evaluation process, stakeholder involvement will be key. 

Logic Model

Before an evaluation of program is carried out, a clear description of the program is valuable.  Two tools that evaluators often use in helping stakeholders to describe their program are: logic models and action model/change model (Chen, 2014, p. 58).  Each has its own merits and emphasis and combined give a very comprehensive description of the program.  Logic models are excellent at reducing a complicated program to “a set of meaningful, manageable components” (Chen, 2014, p. 58).  Action model/change model schema is more complex, but “more useful when program planning or evaluation need to address contextual factors and causal mechanisms” (Chen, 2014, p. 58). 

A logic model is a concept map that illustrates the various components of a program, including theories and assumptions (Brown, 2012).  The basic elements of a logic model are as follows (Brown, 2012; Chen, 2014, p. 59):

Inputs – resources (both tangible and intangible), including contribution in kind, building space, technology, professional expertise, course materials, and partnerships

Activities – how resources are used with logical sequence of steps or processes

Outputs – the direct evidence/end-products, including documents, change, job opportunities

Outcomes – elements of success that results, including final reports, reflective writings, job offers

            The logic model for the MEHPEV project is provided in Table 1.

Table 1.  Logic Model for the Multilingual Eye Health Patient Education Video Project

Inputs	Activities	Outputs	Outcomes – short-term	Outcomes – long-term
Funding Volunteer ophthalmologists who speak non-official languages Video filming and edit team YouTube/DOVS website manager North York General Hospital eye clinic and Kensington Eye Institute eye clinic as filming facility	Produce scripts for each video topic Translation of scripts into multiple languages Film videos Edit videos Seek media contacts for video dissemination (TV, radio, newspaper, ethnic websites) Seek video dissemination through Toronto Public Health, community groups, physician groups Seek additional funding to include more languages and more video topics (government grant applications, approaching individual and corporate donors)	Multilingual eye health patient education videos Video awareness promoted to the target population of (those who speak one of the major non-official languages in GTA) Viewing of the online videos by target population directly or through assistance by community newcomers organizations	Increased awareness of eye disease symptoms in the target population Increased awareness of regular eye examinations in the target population	Reduce number of patients from the target population from presenting with end-stage eye disease Cost savings to our healthcare tax dollars through early prevention and early treatment

Action Model/Change Model

Change models provide descriptive assumptions and deal with the causal processes that are expected to happen to attain program goals (Chen, 2014, p. 66).  Action models provide prescriptive assumptions and deal with what actions must be taken to produce desirable changes (Chen, 2014, p. 66)

            Below is a verbal description of the change model.

Intervention

-       Multilingual eye health patient education videos

Determinants

-       Education of those who speak one of the major non-official languages in GTA (the target population) about eye disease symptoms and importance of regular eye examinations 

Goals and outcomes

-       Goals:

1)     To reduce the number of patients who speak one of the major non-official languages in GTA presenting with end-stage eye disease (secondary prevention)

2)     To promote healthy lifestyle and eye disease risk reduction (primary prevention)

3)     To save healthcare spending costs through prevention, and early detection and early treatment

-       Outcomes:

·         Short-term

o    Increase awareness of eye disease symptoms in the target population

o    Increase awareness of healthy lifestyle and eye disease risk reduction in the target population

o    Increase awareness of importance of regular eye examinations in the target population

·         Long-term

o    Reduce number of patients from the target population presenting with end-stage eye disease

o    Improved eye health of the target population through risk reduction and regular eye examinations  

o    Cost savings to our healthcare tax dollars through prevention, and early detection and early treatment

Stakeholder Involvement

            A stakeholder is a person with an interest or concern in something (“Stakeholder,” n.d.).  The MEHPEV project stakeholders include:

-       Individuals who speak limited English or French living in GTA (the target population)

-       MEHPEV project volunteers

-       DOVS

-       Toronto Public Health

-       Public Health Ontario

-       Physician groups (ophthalmologists, family physicians)

-       All taxpayers (through savings in healthcare tax dollars via Ontario Ministry of Health and Long-term Care)

            As the MEHPEV is a small scale project, there are two key stakeholder groups: target population and MEHPEV project volunteers.  Representatives of each key stakeholder group will be involved from the very beginning of program planning stage of the project and the evaluation process.  Intensive interviews with representatives of each key stakeholder group would be carried out initially.   Questions to ask the key stakeholder groups to facilitate in drafting the Action Model/Change Model include:

-       What video topics are of interest?

-       How long should the videos be?

-       Would subtitles be useful?

-       How many and which languages should the videos be produced in?

-       Given the human resources (volunteers) and financial resources (donations), how many topics and languages can we produce the videos in? 

Involving stakeholders in project planning stage is extremely valuable in gaining insights regarding what the target population wants, what the volunteers are able to produce, and what components to include in the project given the funding and cost limitations.  A disadvantage of involving stakeholders is that sometimes it is not possible to satisfy all the stakeholders or there may be conflicting interests between stakeholders.  For example, the target population likely would want to have the videos produced in as many languages as possible.  However, for the MEHPEV volunteer group, it is not possible to produce numerous video topics and in all languages, as there is a limited pool of fund that we can work with and limited number of volunteers.  So judgement must be exercised to balance all the wishes of the various stakeholders and still work within the funding limits of the project.

Close and regular communication between the evaluator and the stakeholders will occur throughout the evaluation process, especially as the evaluation transition from constructive to conclusive assessment. 

References

Australian Government Department of Health. (2014). Implementation plan under the National framework for action to promote eye health and prevent avoidable blindness and vision loss. Australian Government Department of Health. Retrieved from http://www.health.gov.au/internet/main/publishing.nsf/Content/eyehealth-pubs-impl

Australian Health Ministers’ Conference. (2005). National framework for action to promote eye health and prevent avoidable blindness and vision loss. Australian Government Department of Health and Ageing. Retrieved from http://www.health.gov.au/internet/main/publishing.nsf/Content/eyehealth-pubs-frame

Brown, C. A. (2012). Use of Logic Models to Plan and Assess Graduate Internship Experiences. TechTrends, 56(6), 37–43. http://doi.org/10.1007/s11528-012-0612-2

Chen, H. T. (2014). Practical Program Evaluation: Theory-Driven Evaluation and the Integrated Evaluation Perspective. SAGE Publications.

Citizenship and Immigration Canada. (2014, October 31). 2014 Annual Report to Parliament on Immigration. Retrieved December 10, 2015, from http://www.cic.gc.ca/english/resources/publications/annual-report-2014/index.asp

CNIB. (n.d.). Fast Facts about Vision Loss. Retrieved from http://www.cnib.ca/en/about/media/vision-loss/pages/default.aspx

CNIB, & Canadian Ophthalmological Society. (2009). The Cost of Vision Loss in Canada: Summary Report 2009. Retrieved from http://www.vision2020canada.ca/en/resources/Study/COVL%20Summary%20Report%20EN.PDF

Eye Health Working Group of the Australian Health Ministers Advisory Council. (2008). National framework for action to promote eye health and prevent avoidable blindness and vision loss: progress report. Australian Health Ministers’ Conference. Retrieved from http://www.health.gov.au/internet/main/publishing.nsf/Content/eyehealth-pubs-progres1

Green, L. W., & Kreuter, M. W. (1991). Health promotion planning: an educational and environmental approach. Mayfield Pub. Co.

Health Canada. (2003). Language Barriers in Access to Health Care [Health Canada, 2001] (report). Retrieved from http://www.hc-sc.gc.ca/hcs-sss/pubs/acces/2001-lang-acces/index-eng.php

Kok, G. (1997). Effectiveness of health education and health promotion: meta-analyses of effect studies and determinants of effectiveness. Patient Education and Counseling, 30(1), 19–27.

Stakeholder. (n.d.). [In Merriam-Webster’s online dictionary]. Retrieved December 21, 2015, from http://www.merriam-webster.com/dictionary/stakeholder

Statistics Canada. (2010, January 8). Projections of the Diversity of the Canadian Population. Retrieved December 10, 2015, from http://www23.statcan.gc.ca/imdb/p2SV.pl?Function=getSurvey&SDDS=5126&lang=en&db=imdb&adm=8&dis=2

Statistics Canada. (2012, October 24). Census: Language. Retrieved December 10, 2015, from http://www5.statcan.gc.ca/olc-cel/olc.action?ObjId=98-314-X2011001&ObjType=46&lang=en&limit=0

Vision 2020 Australia. (n.d.). National Framework Implementation Plan. Retrieved December 9, 2015, from http://www.vision2020australia.org.au/our-work/advocacy-campaigns/national-framework-implementation-plan

World Health Organization. (2013). Universal Eye Health: A global action plan 2014-2019. Retrieved from http://www.who.int/blindness/AP2014_19_English.pdf?ua=1

World Health Organization. (n.d.). What is VISION 2020? Retrieved December 9, 2015, from http://www.who.int/blindness/partnerships/vision2020/en/