Program Evaluation at PittMed

PE Team

Faculty Director of Program Evaluation
Allison Serra, MD, MPH
serraae@upmc.edu

Director of Program Evaluation, CQI, and Accreditation
Greg Null, MA
grn18@pitt.edu

Executive Director, Office of Medical Education
Katie Maietta, MPPM
kmaietta@medschool.pitt.edu

Project Analyst
Michelle Sergent, MPH, MSL
mls124@pitt.edu

Program Administration Assistant Director
Michelle Wargo
miw114@pitt.edu

Program Evaluation: The systematic use of methods to collect and analyze information to determine whether a program is fulfilling its mission(s) and achieving its goal(s).

Goals of Program Evaluation

1. Evaluate and continuously improve every component of 3RC and the curriculum as a whole
2. Maintain full LCME accredited status

Approach to Program Evaluation: The Plan for PittMed

The PittMed Program Evaluation uses a tiered system to evaluate 3RC:

• Tier 1: Individual Course Level
• Tier 2: Segment Level (Foundations, Clerkship, Bridges, Threads)
• Tier 3: Whole Curriculum Level

Data sources:

1. Course (including clerkship) evaluations
2. Faculty evaluations
3. “Feedback Friday” summary notes
4. Formative student assessments
5. Summative student assessments
6. School survey
7. Graduate questionnaire
8. USMLE reports
9. NBME reports
10. Match list
11. Curriculum Map

Structure of Program Evaluation:

¹Definitions:

(D) Design/develop = Develop/create the product or process that is the basis of the task (e.g., the educational program objectives, the plan and tools for course evaluation)

(E) Evaluate = Carry out a process to collect data/information on quality/outcome

(R) Review = Receive and consider the results of an evaluation of the product or process and/or of its outcomes

(Rec) Recommend = Propose an action related to the process or product based on a review or evaluation

(A) Approve/Take Action = Have final responsibility for an action related to the product or process

Activities and Responsibilities of the Program Evaluation Team:

Tier 1 (Course level) activities:

1. Design, review, distribute, analyze course and faculty evaluations
2. Facilitate “Feedback Friday” conversations among students, faculty and staff to reflect on 3RC (we use the the After Action Review Framework)
3. Solicit and track “on the fly” curricular changes from content leaders in response to “Feedback Friday” conversations
4. Assemble a “compendium” of outcomes for each course and distribute to students, Curriculum Committee Executive Subcommittee
5. Membership on  Foundations Course Design Subcommittee
6. Membership on Clerkships/Bridges Subcommittees

Tier 2 (Segment level) activities:

1. Design segment level review template for CC reports
2. Collect and transmit data to curriculum committee/subcommittees and deans as needed
3. Membership on Curriculum Continuous Quality Improvement (CCQI) Subcommittee

Tier 3 (whole curriculum level) activities

1. Respond to requests for consultation on issues related to curricular quality/effectiveness
2. Membership on Curriculum Continuous Quality Improvement (CCQI) Subcommittee

CQI Greenhouse

The Program Evaluation office is actively seeking medical students who wish to develop CQI projects and hone CQI skills. Medical students who have an idea for a CQI project or wish to learn more about accreditation, program evaluation or CQI skills should contact Greg at grn18@pitt.edu or stop in the OMED offices (S585).

News from the Greenhouse

Lauren Stumpp (Class of 2025) presented her project "Students Write the “Scrip” for Feedback Using the After Action Review as a “Skill Acquisition Session” at the Generalists in Medical Education National Meeting in November. Earlier this year, Lauren also presented her project "Who ISN’T Sweating Step 1? Student Led QI Using the After Action Review" at the 2024 national Accreditation Preparation and Quality Improvement Conference.Her presentation won "Best Oral Presentation!" She has now made several national presentations related to her CQI work at PittMed and is even mentoring students at other medical schools as they learn to use the “After Action Review” framework for student-led CQI. We are so proud of Lauren’s efforts and impact!