Program Review Process_Archived

The major purpose of Graduate Program Review is to ensure that programs are functioning at the highest possible levels of academic quality and are operating in ways that are consistent with the missions of the University and the Graduate School. The process of Graduate Program Review serves as a means to inform the program faculty and university leadership of areas of excellence and areas of needed improvement in particular programs. In this sense, Graduate Program Review is a tool for critical reflection and change.

Fundamental to Graduate Program Review at Appalachian is the notion of Institutional Effectiveness – "ongoing, integrated, and research‐based planning and evaluation processes that incorporate a systematic review," resulting in continuing improvement. In particular, a program must identify expected outcomes; assess whether it achieves these outcomes; and provide evidence of action and improvement based on analysis of those results. (Core Requirement 2.5 and Comprehensive Standard 3.3.1 in The Southern Association of Colleges and Schools (SACS) Principles of Accreditation)

To this end, each graduate degree and certificate program has established goals, objectives, and measures that are in concert with the program's mission and vision; these goals and objectives include learning outcomes. Departments will engage in ongoing evaluation to measure the extent to which the program is meeting these goals and objectives.

The Process

As recommended by the Graduate Education Task Force (2008), graduate programs at Appalachian will be monitored for quality through the following process:

(1) Each year the graduate program faculty will choose a subset of its program's objectives to examine. If the evidence from the data collected suggests that improvement is needed, action plans will be devised and implemented. The results, action plans, and any follow-up will be recorded in the TracDat system annually. Best Practices:

  • Programs should plan to cycle through all of their objectives every five years.
  • When selecting what to study each year, objectives with concerns from prior years for which action plans have been implemented should be revisited.
  • Actual data do not need to be stored in TracDat if a summary outlining the data in aggregate (tables, graphs, charts) can be developed to capture the information.

(2) Each year, the Graduate School will extract the assessment reports for all graduate programs to review progress toward the goals and objectives and prepare a summary. Deans and departments will provide a brief summary of any outstanding achievements and any concerns they have about the program. These reports will be reviewed by the Graduate School, and each program will be more closely reviewed through the Assessment and Program Review Committee of the Graduate Council every other year.

3) In even-numbered years, programs that fall into one of the categories below will be required to submit a comprehensive status report for review rather than just a summary.

  • programs that do not meet minimum enrollment requirements (The minimums used are those set by the UNC General Administration):
    • Master's and specialist programs: the number of degrees awarded in the last two years is 15 or fewer—unless enrollment in the most recent Fall semester exceeds 22, or degrees awarded in the most recent calendar year exceed 9.
    • Doctoral programs: the number of degrees awarded in the last two years is 5 or fewer—unless enrollment in the most recent Fall semester exceeds 18, or the number of degrees awarded in the most recent calendar year exceeds 2.
  • programs deemed of concern by the Provost, a dean or a department chair. In particular, the Dean of the Graduate School is charged with particular attention to the level and rigor of the curriculum.

Timeline

  • May 1: The Graduate School will issue a reminder about the report and the need to place information in TracDat.
  • July 1: The Graduate School will pull the program reports from TracDat (two years worth of assessment information).
  • August (Friday before first class): Summary reports will be submitted to the Graduate School.
  • September and October (Graduate Council meetings): Each year half of the graduate programs will be reviewed by Graduate Council. Feedback will be provided in the year a program is reviewed.

Guiding questions that will be used by the Graduate Council during the review process:

  • Is there a pattern of regular activity in TracDat showing that programs are monitoring the extent to which they achieve their mission and goals?
  • Are the measures being used clear, measurable, and related to the goals and objectives?
  • Are relevant action plans in place and being followed?
  • If the program has been flagged because of low review or another issue, is a plan in place to address this?

Status Report Guidelines

The status report must address the following items (based upon the report required by the UNC General Administration for low enrollment programs):

  • Centrality to University's Mission: How important to the mission of the institution is this program? Can this program be combined with a similar or related program in the present department or in another department?
  • Quality of the Program: What is the quality of the program and what indicators are used to assess the quality? Is the program accredited or has accreditation been sought? Are the students in demand, either by employers or by other graduate programs?
  • Faculty Involvement: How many faculty members are teaching and/or mentoring graduate student research in this program? How are their scholarship interests related to the program? What is the average teaching load of the faculty in the department?
  • Facilities/Equipment: Are available space and equipment adequate and appropriate for the program?
  • Demand: Is the program serving the predicted number of students? What are the job prospects for these graduates? Are there courses in the program that are essential supporting courses for other programs?
  • Costs: Could some program options or concentrations be consolidated or eliminated? What is program productivity as it is reflected in course enrollments? Does the program have under-enrolled courses?
  • Duplication: Can this program's objectives be accomplished equally well through another program? Are courses in the program duplicated in other programs/departments? Could enrollment be increased by sharing some courses through distance education? Is this program distinctive in the UNC system?
  • Critical Mass: What would be the impact on departments or programs if the program under review were eliminated?

The status report will be reviewed by the Assessment and Program Review Committee of the Graduate Council. The Graduate Review Committee will submit a response to the department chair, academic dean, graduate school dean, and the Graduate Council.

Key Indicators of Need and Quality

The Graduate Council has proposed the following as key indicators of fit, quality, and need to further assist programs in understanding how they will be assessed and how to address the eight required points listed above.

(1) Fit: UNC Tomorrow; Mission and Strategic Plan of the University; Goals for Graduate Education

(2) Size of and need for the program.

  • enrollment: academic year headcount; enrollment at each location (including online as a location); comparison with UNC minimum graduation/enrollment requirements; compared to similar programs in NC.
  • critical mass: average enrollment in graduate classes; ratio of students to faculty; number of students per faculty research mentor; availability of course offerings to enable on-time completion without substitution.
  • demand: average # of applications over 5 years; Bureau of Labor statistics; NC or other projections for workforce needs
  • duplication: # of similar programs in UNC system (enrollment comparisons); in NC; in the region

(3) Program Quality Indicators

Required: Curriculum; Student Outcomes; Faculty

Optional Additional: Support of Department Mission; Admissions and Entering Students; Progression Through the Program; Infrastructure Supporting the Program; Student Financial Support

The lists below provides suggested measures to aid programs in determining the best measures given their specific goals and objectives. It is anticipated that programs will choose a few from these lists, but there may be measures not listed that the faculty will employ to better reflect the quality of the program.

REQUIRED INDICATORS

Curriculum.

Possible measures to choose from:

  • progress toward / achievement of student learning outcomes
  • clear differentiation from undergraduate level program
  • clear indication of currency of the curriculum with respect to ethical practices, research and/or professional practice
  • strong disciplinary accreditation review or regional/national ranking
  • honors/awards/competitions received/won by students
  • number of student publications, presentations, exhibitions, performances
  • quality of capstone work (thesis, dissertation, Product of Learning, Recital, etc.)
  • opportunities for community engagement
  • number of dual-listed courses; average % of dual-listed courses in student POS

Student Outcomes.
Possible measures to choose from:

  • general job outlook for graduates in the field
  • % employed within 6 months of degree in positions related to degree
  • % passing licensure/certification/professional exams, w/ comparison to national metrics
  • placement in further study or professional development programs
  • employer satisfaction
  • alumni satisfaction


Graduate Faculty Engaged in the Program.
Possible measures to choose from:

  • # or % of research-active faculty
  • faculty record of scholarship/creative activity
  • # or % of faculty with terminal degree
  • # or % of non-TT faculty teaching in the program
  • # or % of faculty receiving awards (internal, regional, national)
  • ratio of students to graduate faculty
  • amount of external funding
  • distribution of graduate activities among faculty members
  • faculty involvement in graduate education at campus, state, or national level
  • quality of mentoring, including assistance with career placement, doctoral and professional program application, training in research conduct and/or effective teaching,

OPTIONAL ADDITIONAL INDICATORS

Support of Department Mission.
Possible measures to choose from:

  • relationship of GA duties to graduate program goals
  • centrality of GA work to the department mission
  • contribution to undergraduate teaching mission
  • contribution to faculty research mission
  • undergraduate SCH generated via TAs
  • graduate SCH generated

Admissions and Entering Students.
Possible measures to choose from:

  • admitted and/or entering student credentials and abilities
  • size of qualified pool of applicants
  • admissions selectivity (% of pool admitted)
  • admissions yield (% of admits who attend)
  • % of admits on probation
  • success of recruitment activity
  • diversity of applicant pool/admitted/enrolled students

Progression Through the Program.
Possible measures to choose from:

  • time to degree
  • attrition rate
  • number of suspensions/probations
  • number of requests for time extension
  • cumulative GPA of graduates
  • accomplishment of student learning goals/objectives
  • student satisfaction

Infrastructure Supporting the Program.
Possible measures include:

  • library holdings
  • research/studio facilities
  • rehearsal/practice facilities
  • department resources
  • faculty and graduate student space
  • staff support
  • external advisory/advancement board


Student Financial Support.
Possible measures to choose from:

  • % of students supported by University funds; grants; foundation funds
  • funding for student research / conference travel