Program Monitoring and Evaluation - Selected Case Studies

Department of Health and Human Services (HHS)

Client Objective: The Department of Health and Human Services (HHS) awards billions of dollars each year through discretionary and mandatory grants.  The Grants Management Community within HHS Operating Divisions (OPDIVs) is responsible for overseeing and administering these grants.  In 1997, the Executive Committee on Grants Administration Policy (ECGAP) Oversight Group was formed and tasked with developing a new system for providing effective oversight for HHS’ many and varied discretionary grant programs. HHS desired to create a new system that could be self-administered by individual OPDIVs, be cost-effective to administer, and produce reliable and useful grants management performance data.

FMP Approach: ECGAP chose to undertake the Balanced Scorecard (BSC) methodology to oversee grant operations and it tasked the Office of Grants and Acquisition Management (OGAM) to implement the BSC in HHS, department-wide.  OGAM chartered a BSC Users Group, comprised of a cross section of HHS grants management professionals, to implement a BSC approach for grants management.  OGAM hired FMP to lead the Users Group and to design, develop, and implement the BSC for HHS grants management.

FMP used the following approach to design the HHS Grants Management BSC:

• Facilitated BSC Users Group.
• Developed BSC strawman model.
• Developed BSC employee and customer surveys and questionnaires.
• Developed BSC survey database and report templates.
• Pilot tested BSC process in the Heath Resources and Services Administration (HRSA).
• Designed and developed Implementation Guides.
• Designed, developed, and delivered BSC training.
• Implemented the BSC model and processes in several HHS Operating Divisions.

Results: Through this effort, HHS was able to successfully implement a Department-wide balanced scorecard process, including Office of Management and Budget (OMB) approval for a public survey, aimed at grant recipients.  FMP also received contracts from numerous OPDIVs to implement the BSC process in individual OPDIVs.  Data collected through this process has enhanced the Department’s understanding of its grants management function and led to numerous action plans at the OPDIV level designed to improve the grants management function in response to findings identified through the balanced scorecard data. For several years after the BSC implementation, FMP maintained an active role in the on-going upkeep of the BSC process within HHS grants management.  Today the BSC is independently implemented within HHS.

Office of Personnel Management (OPM)

Client Objective: The Office of Personnel Management (OPM) wanted to answer the growing need for a single framework for evaluating human capital transformation, and to conduct a Congressionally-mandated assessment of current alternative personnel systems.

FMP Approach: FMP worked with OPM to develop a framework for evaluating human capital transformation and to use the framework to assess current alternative personnel systems. FMP utilized the following approach:

• Develop Assessment Framework: FMP developed an assessment framework through a combination of methodologies including conducting literature reviews, interviewing subject matter experts, and studying results of previous alternative personnel system projects.
• Identify Assessment Criteria: FMP developed criteria for assessment of Preparedness and Progress dimensions and indicators based on a combination of historical data, best practices/literature reviews, and input from subject matter experts. 
• Collect Data: In order to evaluate current APSs, FMP applied the APS Assessment Framework and collected data from a variety of sources including: OPM archives, Federal Human Capital Survey databases, Agency–specific employee surveys, Agency internal evaluations, Agency HR information systems, and Agency program office.
• Conduct Assessment: FMP assembled an assessment panel to evaluate two current Federal alternative personnel systems. The panel had competency in the design, implementation, and evaluation of demonstration projects and/or alternative personnel systems; Federal human capital leadership; program evaluation; and the design and implementation of major human capital systems. FMP developed and delivered training to the panel, including background information on the assessment framework, the history of the development of the APS, and guidance on how to use provided data and their own expert knowledge in order to arrive at an assessment.

Results: As a result of FMP’s efforts, OPM received an APS assessment framework, which is a structure for determining the extent to which an agency is adequately preparing for and progressing on the strategic human capital transformation goals and objectives of its APS. Along with the assessment framework, OPM received a set of criteria to use in assessing organizations that, based on past experience in both the public and private sectors, and input from key stakeholders in both OPM and other agencies, are essential to the successful implementation of significant human capital system reforms. Finally, OPM received an executive-level dashboard that can be used to communicate summary results to stakeholders. This package can be used to assess future demonstration projects, as well as new alternative personnel systems as they are implemented across the government.