Program Review Process

Please refer to the standards for this process at:

https://irap.appstate.edu/institutional-effectiveness/periodic-comprehensive-reviews


Graduate Program Assessment:

Standards and Guidelines

In accordance with the University’s Institutional Effectiveness (IE) Model, the following guidelines have been developed to facilitate the review of graduate programs. This document presents a set of common standards, along with suggested indicators that can be used to demonstrate the extent to which graduate programs meet those standards. Programs may elect to include additional evidence to document effectiveness, or may select indicators deemed more appropriate for the program (e.g., accreditation metrics) to demonstrate how it meets the standards. This process is designed to integrate with (rather than to overlay or supersede) the University’s IE Model as well as with any accreditation reviews a program may undergo. The standards and recommended indicators were developed to specifically reflect the structure and characteristics of graduate programs, and this document was designed to replace the graduate program review guidelines that were put in place in 2008.

Program review is an ongoing process used to drive program improvement, with the primary goal of helping graduate programs function at the highest level of academic quality. Consistent with the Appalachian State University IE Model, all graduate programs are encouraged to complete formative reviews as well as periodic comprehensive reviews that incorporate a unit self-study and an external review of the program (see Appendix). Data collection and analysis may be completed by members of a graduate program’s full-time graduate faculty and should adhere to whatever program review expectations have been established by the department and college. Annual and/or “mid-cycle” reviews are anticipated to serve a largely reflective function to assist with program monitoring and planning. Formative reviews should also help to enhance a program’s readiness for the periodic comprehensive review (PCR), which is expected to serve a largely summative function. These recommendations are consistent with the Southern Association of Colleges and Schools (SACS) Core Requirement 2.5 and Comprehensive Standard 3.3.1.

This document is divided into three parts. Part 1 outlines a suggested process for program review; however, decisions about the process used by any program will be dependent on the reporting requirements specified by the home college and/or the University. Part 2 identifies program standards and suggested indicators. Sources for information to address each indicator are shown parenthetically after each indicator. This section also includes reflection questions to help focus the analysis of indicators. Part 3 provides a suggested framework for analysis and reporting for annual (or mid-cycle) reports and for the periodic comprehensive reviews.

PART 1: RECOMMENDED GRADUATE PROGRAM REVIEW PROCESS

  1. Annual (Formative) Review: Designed to facilitate ongoing and formative reflection, and to provide programs with longitudinal data that can be included in the PCR. Members of the department or program’s full-time graduate faculty, led by the graduate program director, collect and analyze information related to the annual review indicators and prepare a summary report that meets college-level and/or University requirements for annual reporting
  2. Mid-Cycle (Formative) Review: Although not specified in the current IE model, a mid-cycle formative review is recommended to assess progress toward the PCR. The mid-cycle review can focus on the annual data collected to date and on results from the Graduate School’s periodic survey of graduates, and may be guided by the reflection questions provided here. Programs that elect to carry out a  mid-cycle review can request that their report be reviewed by an ad hoc committee of the Graduate AP&P, which will provide confidential, formative recommendations to help program faculty prepare for the PCR.
  3. Periodic Comprehensive Review (PCR): In accordance with the University’s comprehensive unit review schedule[1], members of the department or program’s graduate faculty prepare a report that: (a) summarizes information from the formative reviews, (b) includes data showing how the program addresses the graduate program review standards, (c) analyzes major findings, and (d) identifies future directions for the program. This report will be reviewed according to the IE process determined by IRAP and the University Deans’ Council.

PART 2: GRADUATE PROGRAM REVIEW STANDARDS & QUALITY INDICATORS

Standard 1: The program has a clear, current, and comprehensive mission that supports the mission of the College or School and of the University.  Suggested documentation may include (with possible data sources):

PERIODIC COMPREHENSIVE REVIEW

  1. Brief historical and contextual description of the program, with a focus on recent changes. (Program)
  2. Organizational chart or flow chart showing program/department structure. (Program/Department)

REFLECTION QUESTIONS

  1. Have we reviewed our program goals and aspirations, and are these reflected in our program mission statement?
  2. Does our program mission statement align with the missions of the department, college, and university?

Standard 2: The program recruits, retains, and graduates high quality students.  Suggested documentation may include:

ANNUAL OR MID-CYCLE REVIEW

  1. Average UGPAs and standardized test scores for admitted students, as well as for students who were admitted but did not enroll (Graduate School)
  2. Number of complete applications received by the program across all active admission periods for the year, and the acceptance and yield rate for these complete applications. (Graduate School)
  3. The ratio of accelerated admission students admitted/enrolled to students who are admitted/enrolled from outside the program or university. (Graduate School)
  4. The program’s target, and average, time-to-degree, in years. (Graduate School)
  5. DISTANCE EDUCATION PROGRAMS ONLY:  Number of cohorts/sites that are active, and number of students enrolled in each cohort/site. (IRAP/Office of Distance Education)

PERIODIC COMPREHENSIVE REVIEW

  1. Number of complete applications received by the program, on average, over the past five years, and the program’s capture and yield rate over those five years. (Graduate School)
  2. Program enrollment trends over the past five years, broken out by majors, certificates, concentrations, and minors (as applicable). (IRAP)
  3. Student credit hour production trends in the program over the past five years. (IRAP)
  4. Enrollments in similar programs (same CIP code) at other UNC campuses. If the program is markedly smaller than others in the UNC system, provide a rationale for maintaining the program’s enrollment, or a description of what steps are being taken to increase the program’s enrollment. (UNCGA /Program)

REFLECTION QUESTIONS

  1. Have we met our application targets? If not, why not? What can we do differently to enhance the number of applicants to our program?
  2. What factors explain our program enrollment trends? Are we satisfied with these trends? What recruitment strategies have we implemented, including strategies to attract a diverse population of students?
  3. What has been our experience with accelerated admission students? Are changes needed in our approach to accelerated admission?
  4. Does student credit hour (SCH) production within our graduate program meet established targets for the college? What changes in how courses are offered might be needed to reach our SCH targets?
  5. Have we met our time-to-degree target? If not, why not? What can we do differently to allow for timely degree completion by our students?

Standard 3. The program has established a high-quality curriculum that focuses on student learning and scholarly engagement, is responsive to information from stakeholders, and contributes to student success following graduation. Suggested documentation may include:

ANNUAL OR MID-CYCLE REVIEW

  1. The program’s goals and student learning outcomes, as entered into Xitracs, and a description of what action plans have been adopted and implemented as a result of the assessment of student learning outcomes. (Program)
  2. Notable indicators of student scholarly and/or creative endeavors, as demonstrated by student presentations, publications, performances, awards, performance in practicums/internships, etc. (Program)
  3. Percentage of graduates employed in the field of study within six months of program completion, and/or percentage of graduates admitted to doctoral or other terminal degree programs.  (Graduate School/Program)
  4. Where applicable, student performance on standardized tests required for licensure or accreditation. (Program)

PERIODIC COMPREHENSIVE REVIEW

  1. Data from the Graduate School survey of program completers regarding program quality. [Note: if fewer than five graduates respond these data are not reported so will be excluded from the survey report.] (Graduate School)
  2. Information from periodic data collection efforts (surveys, focus groups, etc.) of alumni regarding program quality, structure, and outcomes. (Program)
  3. Evidence of use of program assessment results to revise the program’s curriculum and/or structure. (Program)
  4. Where applicable, results of the most recent accreditation effort. (Program)

REFLECTION QUESTIONS

  1. What have we learned about student attainment of identified learning outcomes, as determined through program assessment efforts?  What programmatic changes have we made in response to program assessment?
  2. Are graduates of our program successful in achieving their professional goals? 

Standard 4. The program is supported by engaged and effective graduate faculty members. Suggested documentation may include:

ANNUAL REVIEW

  1. Percentage of graduate courses taught by tenure-track or tenured faculty, by full-time lecturers or clinical faculty (NTT), and by part-time faculty. If the program has a distance education program, report these data separately for that program. (Program)
  2. Notable indicators of graduate faculty scholarly and creative endeavors, as demonstrated by presentations, publications, external funding, performances, awards, and so on, including collaborative faculty/student scholarly endeavors. (Program)

 PERIODIC COMPREHENSIVE REVIEW

  1. Details about the structural elements of the program to insure effective mentoring of graduate student work, as demonstrated by the ratio of students to graduate faculty who actively advise graduate students, the average thesis mentoring load of full-time program faculty, the percentage of research-active graduate faculty, time toward student completion of capstone research/creative endeavors,  and/or other factors as determined by the program. (Program)

 REFLECTION QUESTIONS

  1. Do we have sufficient faculty resources to offer program courses in a way that allows for timely degree completion by students? What gaps exist in our faculty expertise and/or ability to deliver courses when needed?
  2. Do our graduate faculty serve as effective models of active and engaged scholars?    
  3. Is the process in place for reviewing the qualifications of tenured/tenure-track, NTT faculty, and part-time faculty adequate for assessing the effectiveness of program faculty?

Standard 5. The program has adequate resources to effectively meet its mission and goals. Suggested documentation may

ANNUAL REVIEW

  1. The annual allocation of resources from the Graduate School in terms of assistantship funding, scholarships, and NC Tuition Scholarships, and a list of sources other than the Graduate School that have provided assistantships and/or scholarship funding for students in the program. (Graduate School/Program)
  2. The amount of extramural funding the program faculty have acquired. (A-Grants)
  3. The average number of student credit hours (SCH) generated per graduate course offered by the program for the calendar year. (IRAP)

PERIODIC COMPREHENSIVE REVIEW

  1. Description of program resources, including, as applicable: access to suitable classrooms; availability of library resources; support for faculty scholarly work (e.g., start-up funds, equipment, lab/studio space); adequacy of IT resources and productivity work flow; availability of student research and graduate assistant work space; availability of faculty office space; etc. (Program)

REFLECTION QUESTIONS

  1. What program needs or functions are not being met due to insufficient numbers of graduate assistants within the program?
  2. What efforts have we made, and could we make, to supplement the amount of funding that is available for graduate assistantships and/or scholarships to support students in our program?
  3. Are available spaces and equipment adequate and appropriate for meeting the needs of the program? What additional physical resources are necessary to fulfill the core mission of the program?

Part 3: ANALYZING Major Findings and IDENTIFYING Future Directions

  1. Annual and Mid-Cycle Review

                             i.     What program strengths have been identified?

                           ii.     What weaknesses or areas for improvement have been identified?

                          iii.     What opportunities for future growth or quality enhancement have been identified?

  1. Periodic Review

                             i.     What program strengths have been identified?

                           ii.     What weaknesses or areas for improvement have been identified?

                          iii.     What opportunities for quality enhancement have been identified?

                          iv.     What future goals/directions have been identified for the program for the next five years? (4-5 goals are recommended.)  

                            v.     What specific action steps will the program take to achieve these goals?


[1] Note that the PCR schedule is subject to change; programs should verify the schedule directly with IRAP or their home college.

ASU IE Cycle