Ciclovia Recreativa

Evaluation tools for the Ciclovía Recreativa

part 4 of 7

Six steps for the evaluation of Ciclovias Recreativas

Step 1 Engage stakeholders

Step 2 Describe the program

Step 3 Focus the evaluation design

Step 4 Gather credible evidence

Step 5 Justify conclusions

Step 6 Ensure use and share lessons learned

What are the six steps?

This study consists of the application of the six steps of the U.S. Centers for Disease Control and Prevention's (CDC) Framework for Program Evaluation in Public Health. This allows for the simple and effective evaluation of the process and/or results of the implementation of a Ciclovía Recreativa and its complementary programs. Its principal purpose is to use information to identify strengths and weaknesses for the improvement of the program.

What is its purpose?

To influence policy makers and those in charge of the programs finances = sustainability.
To build capacity and community engagement.
To share the observed advantages and disadvantages with other communities.
To show results ensuring financing and sustainability.
To reconsider strategies or redirect resources.

What steps should be followed?

1. Engage stakeholders.
2. Describe the program.
3. Focus the evaluation design.
4. Gather credible evidence.
5. Justify conclusions.
6. Ensure use and share lessons learned.

Step 1: Engage Stakeholders

What is engaging stakeholders?

In this step, try to identify all of the people and organization from every sector that may be related directly or indirectly with the Ciclovía Recreativa.

What is its purpose?

To ensure the success of the evaluation process and its objectivity, avoiding the obtainment of only biased opinions or only those of a single point of view.

What steps should be followed?

Develop a list of all the current stakeholders and do not forget to include key allies that may not have been involved. In order not to forget any key actors, answer the following questions:

Who is involved in the program's operation?
Those involved include: the general coordinator of the Ciclovía Recreativa, the operations coordinator, personnel from other parts of the entity responsible for implementing the program, volunteers, community service workers, first aid personnel, route chiefs, security personnel, logistics personnel, user orientation personnel, physical activity instructors and the complementary activity personnel.

Who are the stakeholders?
The stakeholders include the police, transit authorities, emergency medical services, fire department, and the local authorities of: transportation, education, planning, sports and recreation, public works, parks and the environment. Other stakeholders include: NGOs, universities, service providers, sponsors, transportation businesses, the media and marketing sectors, civil society, chamber of commerce, and informal vendors.

Who will benefit and/or be impacted by the program?
Everyone is a potential user. It is very important to connect with all the members of the communities where the route will pass as well as all the sectors that will be affected by the closing of streets. It is also key to connect to the retailers along the route who frequently oppose the opening of the program. The results of the evaluation can show positive effects or generate consensus on strategies for promoting the common good.

Who makes decisions regarding the program?
A list of all the individuals or organizations with direct decision making ability over the development of the Ciclovía Recreativa.

Step 2: Program Description

What is the Program Description?

This step consists of a detailed outline of the program, which illustrates its core components in order to ensure the stakeholders and allies share the same understanding of the program.

What steps should be followed?

Identify in which stage of development the program is found:

Planning: Has not started.
Implementation: Has been recently developed.
Maintenance: Has been sustained over several years.

Identifying the program's stage of development is very important for focusing the process of evaluation. If your program is in the planning stage you can apply the six steps to determine needs, to foresee any problems your program may have to overcome and to develop realistic and effective strategies to be used at the implementation stage. If it is in either the implementation or maintenance stages, the evaluation should focus on measuring the effectiveness of the activities at achieving its goals over the short, medium and long terms. It should also measure whether or not the program is on track to achieve its final objective(s) or if improvements or changes are necessary.

Statement of Problem

The Ciclovías Recreativas may contribute to the solution of problems involving sustainability of the environment, recreation, physical activity levels, use of public space, quality of life, mobility, social cohesion, etc. By answering the following four questions, you will be able to briefly and simply determine the problem.

To what problem will the Ciclovía Recreativa principally be responding?
What is the magnitude of the problem (including sub-populations)?
What are the consequences of the problem?
What causes the problem?

It is important to document these indicators, exploring local data and statistics from sectors such as health, transportation and the environment where the Ciclovía Recreativa program will be developed. The better the program is associated with real problems in the community, the better the political support and sustainability of the program will be.

Developing a Logic Model

Now that you have identified the problem and principal goal, start to outline your program with a logic model.

What is a logic model?

An iterative description tool that shows the program's principal components and provides a point of reference to consult during the planning, implementation and evaluation of the program.

Why develop a Logic Model?

To describe the central components of the program.

To show the connections between its components and the desired results.

To make adjustments and improvements to the program.

To assess the potential effectiveness of the program and to establish priorities for resource allocation.

What steps should be followed?

You can start by making a list of the resources, contributions and activities of the program and continually asking yourself 'Why?' or 'What purpose does this have?' in order to construct your desired outcomes over the short, medium, and long term. This approach is principally used when you want to describe an existing program, one in the implementation or maintenance stage that does not yet have a logic model.

Tabla: Example of an activity described through a logic model for the Ciclovía Recreativa.

fig2

You can also start with the desired outcomes and work backwards to develop the activities necessary to achieve these results, asking yourself 'How?' and then moving on to the required resources for implementing the activities. This approach is used most commonly in programs in the planning phase and ensures that the proposed activities are coherent and work toward the desired results.

Use the same exercise with each one of the resources or activities you are considering until you have covered all of the program's components and can produce a detailed description. You will be able to consult this tool at any moment in order to evaluate processes or carry out necessary changes.

Step 3: Focusing the Evaluation Design

What is focusing the evaluation design?

Identifying the purpose and principal uses of the evaluation, as well as the most appropriate questions that should be developed.

What is the purpose?

To obtain concrete results that are clear and useful for the program.

What steps should be followed?

Identifying the purpose and use of the evaluation:

Answer the following question: What are the principal purposes of the evaluation?

Example: To obtain knowledge, improve a program and evaluate its effects.

Enumerate all the possible uses that could be given to the final results based on each purpose (as specifically as possible).

Example:

Table: Purposes and uses of the Evaluation

Purpose Use

Gain Insight To evaluate the level of interest and expectations that the community has regarding a Ciclovía Recreativa program and using this information to plan or restructure a program.

Improve the Program To supervise the implementation of the Ciclovía Recreativa route and complimentary programs and use the results to identify aspects that need to be modified or improved (personnel, marketing, security, etc.).

Assess Program Effects
To measure and monitor performance indicators until they are realized and to use these results to secure additional financing.

Define the evaluation questions:

Ask the principal stakeholders what they would like or need to know with respect to:

Characteristics of the route(s).
Complementary Activities.
Short, middle, and long-term results from the program.
Program participants.
Primary effect of the program on organizations or the community.
External factors that influence the program.
Physical activity.
Quality of life
Other aspects.

Attention: It is important to remember that the program's stage of development may change the type of questions to be asked and the type of evaluation you will conduct: process or outcome evaluation. You can conduct a process evaluation on programs in the implementation phase and focus in activities and resources, or you can carry out an outcome one to evaluate the impact or results of programs in the maintenance phase. You can also combine process and outcome evaluation at the same time.

Examples of process evaluation questions:

What are we doing? When? Where? How frequently?

Are we offering the program according to what was planned? If not, why has it changed?

Are there external influences that are affecting the program's contributions or activities?

Are we on track in terms of time and resources?

What is working well and why?

Are we reaching our entire target population?

Example of outcome evaluation questions:

What did we accomplish? Did we achieve our goals? Why or why not?

What is different as a result of our actions?

What can we learn from participants that have left the program?

How expensive was the program in comparison with other physical activity interventions?

Is the program as effective as or more effective than similar programs?

What went right? What went wrong?

Were there any unintended effects of the program?

Step 4: Gathering Credible Evidence

What does it consist of?

This stage consists of compiling the data that will answer the evaluation questions and, with the feedback of stakeholders, will be reliable and relevant.

What is the purpose?

To identify specific indicators that will answer each evaluation question.

What steps should be followed?

Organize the evaluation questions in a spreadsheet with three columns that include the following headers:

Evaluation Questions

Indicators

Data Source

Identify the indicators and the data sources where you will obtain the indicators that relate to each evaluation question.

Where will I find the data?

Data sources for program evaluation include people, documents, existing databases and epidemiological studies. To increase the credibility of the evidence, collect data from more than one source when possible and use data sources that the stakeholders consider reliable.

People: Program participants, staff, general public, community leaders, funding officials.

Documents: Grant proposals, newsletters and press releases; publicity or educational materials; quarterly reports; medical records; administrative records; program attendance lists; asset and needs assessments; local, state, and national government reports.

Observations

Direct observations of physical activity behavior.

Direct observations of the environment or physical activity facilities (SOPARC).

Indirect observation through video camera or infrared light counter.

Existing Data

State and National Behavioral Risk Surveillance System (BRFSS)

National Health and Nutrition Examination Survey (NHANES III).

National Health Interview Survey (NHIS).

School fitness testing.

Crime Reports.

University-based surveys.

Phone book.

Community or city socio-demographic data.

Example:

Table: Questions, Indicators, and sources of information.

Evaluation Questions	Indicators	Data Sources
What is the community's participation in the program's implementation?	• Number of volunteers. • Number of volunteer trainings. • Total volunteer time. • Description of volunteer activities. • Number of vendors, bike repair shops per event.	• Administrative records. • Volunteer activity logs. • Key informants interviews.
What impact is the program having on the community?	• Number of participants per event on the route(s). • Number of inhabitants of the city and/or community where the program is being implemented. • Number of participants in each event's parallel activities. • Characteristics o the participants: sex, age, race, socioeconomic status. • Level of physical activity among Ciclovía Recreativa participants in the last seven days. • Level of quality of life among participants. • Level of social capital. • Noise level. • Emission levels.	• Participation Logs (for reliable participant counting methods see Evaluation section). • Structured participant questionnaires with socioeconomic data. • Key informant interviews. • IPAQ international physical activity questionnaire. • Quality of Life scale from the WHO. • Social Capital scale. • Noise level measurements. • Measurements of particulate matter (PM10 CO NOX).
What is the cost-benefit of the program? (see cost-benefit analysis in public health for physical activity)	• Overall program budget (operational and equipment costs). • Program budget per event (operational and equipment costs). • Funds given by sponsors or private businesses. • Number of contracted staff and volunteers. • Number of participants per event.	• Administrative Records. • Participation logs.

IDEAL INDICATORS FOR USE WITH CICLOVÍAS

It is important to maintain records of the indicators listed below, some of them are basic and others more sophisticated. However, they should be taken into account from the start of your Ciclovía Recreativa. This will help you to easily monitor and evaluate the program's progress. Additionally, you should have a comparison group that will not be affected by the Ciclovía Recreativa for the purpose of evaluating the effectiveness of the program.

*Remember when analyze the before and after of the program or compare places with or without the program, you can obtain measurements of the impact that will help justify your intervention.

Table: Evaluation Indicators for Ciclovía Recreativa Programs

Indicators
Exact length	The precise length of the routes.
Schedule	Days and time of the of the week when the program will operate.
Participants per event	Number of participants per event.
Demographic characteristics of the participants and community where the program is being implemented	Gender, race, age, socioeconomic status, transportation data (public transportation, automobiles, permanent bike paths, etc.) and access to bike paths in parks in the communities where the program is being implemented. *It is important to describe the population that will benefit from the program and record this information for later use.
Human resources	Number of personnel Number of volunteers Strategic allies (police, public health sector, etc.)
Budget and resources	It is ideal to have an exact estimate of the budget used for the Ciclovía Recreativa in order to obtain precise indicators for the budget and cost of each event. *Its important to include complementary activities or programs that are funded through the same budget.
Complementary activities	Record of all activities that are carried out, number of activities, frequency, participants and description.
Communication strategies	Records of the strategies, quantity of promotional material or their diffusion among the media and their frequency.
Donated services	Always document community services with permanent records: security, public restrooms, first aid, etc.
Vendors	Number of vendors per event (if it is an organized component). By surveying the vendors you can obtain valuable information on how they benefit economically from the Ciclovía.
Physical activity levels	You can evaluate the level of physical activity using IPAQ.
Quality of life Indicators	Quality of life can be evaluated using the QOL tool from the WHO.
Social capital	You can use scales that evaluate social capital before and after the program.
Security	You can evaluate perceived security and the number of reported accidents.
Air quality	You can measure indicators of the quality of the air before and after the program.
Noise pollution	You can measure noise pollution before and after the program.

Step 5: Justifying Conclusions

What is justifying conclusions?

This step consists of carrying out the analysis and interpretation of the collected data, comparing the results with baseline indicators.

What is the purpose?

To obtain conclusions based on the collected data.

What steps should be followed?

Data Analysis
Enter the data into a computer (for example, with EpiInfo, a free data base that you can get online at http://www.cdc.gov/epiinfo, or with a spreadsheet program such as Microsoft Excel).
Verify that the entered data does not contain errors.

Set up a table with the data (for example, calculate the number of participants, the percentage of participants that meet physical activity recommendations, etc.).
Classify the data by category (for example, by community, race or ethnic group, level of income or fitness level).
Set up comparisons (for example, the differences between pre- and post-tests or between a control community and one with the intervention).
Present the data in a clear and simple format.

Interpretation of the results

What do the numbers, frequencies, averages and statistical results show about the program?
Are the results similar to what you expected? If not, why do you think they are different?
Are there any other explanations for the results?
How can you compare the results with similar programs?
What are the limitations of the evaluation (for example, potential biases, generalizability, feasibility, validity)? What measurement in the evaluation reflects the program as a whole?
If various indicators were used to respond to the same evaluation question, were the results consistently obtained?
Will stakeholders be able to interpret the findings appropriately?

Step 6: Ensure use and share lessons learned

What is ensuring use and sharing lessons learned?

This step consists of preparing tangible products of the investigation (recommendations and reports), share them with stakeholders and other audiences (communication), and follow up to promote maximum use.

What is the purpose?

To ensure the use of the evaluation results and to avoid putting another un-used file on the shelf that could bring about successful changes.

What steps should be followed?

Make a list of individuals and groups that should listen to the results or recommendations:

Example
Officials from the program and the organization responsible for its implementation, legislators, municipal/city council members, public health officials, business owners, transit officials, community organizations and programs, support groups, academia (researchers), neighborhood watch groups, non-profit health and service organizations, private enterprises etc.

Determine how the information will be disseminated or communicated.

Remember that each community is different and, as such, information should be communicated accordingly. Example: a scientific report for researchers, a flier or press article for the community, an oral presentation or a policy brief summary of results and recommendations for politicians and potential sponsors.

Establish channels of communication that you will use:

Example
Email lists or listservs, websites, community forums, the press (TV, radio or magazines), personal contacts, organizational newsletters, etc.

Determine how you will follow-up the dissemination of the results by:

Reminding stakeholders and the audience of the intended users of the evaluation results.
Prevent lessons learned from being lost or ignored when complex program or policy decisions are made.
Prevent misuse of results by ensuring that evidence is applied to the questions that were the evaluation's main focus and that the results are not taken out of context.

For more information, you can contact: Andrea Torres: ftu1@cdc.gov or Olga L. Sarmiento osarmien@uniandes.edu.co

Additional references are available in the complete version of the manual (download)

<< previous

next >>