Why Collect Performance Measures Data?
- To show the impact of LIHEAP and other home energy related benefits on low-income households.
- To support funding decision (e.g., to help justify funding increases).
- To help you review your program to be sure you are targeting benefits to the neediest households.
- To identify policies/benefits you may want to revisit/reconsider based upon the data collected. For example, Massachusetts used their data to increase their deliverable fuel heat awards based upon the cost of oil versus utilities such as natural gas or electric heat.
- The last time LIHEAP was reauthorized, language about performance measures was included. See Section 2610 at for information about studies and data collection. Reauthorization of LIHEAP is currently pending.
What Are The Benefits To Collecting Performance Measures Data? How will it be used?
- The Office of the Administration for Children and Families (ACF) will use the information to report to Congress.
- Individual states can use the data to inform their legislators and executive branch departments, i.e., governor's office.
- Data can be shared with the media, research groups, or interested parties.
- You can use the data to review your program effectiveness in reaching your goals.
- Can help with program planning
- Program capacity building
- Will help states identify whether or not they are reaching the goal of providing larger benefits to those with the highest energy burden.
- Demonstrate the positive impact of addressing many health and safety issues.
- NEADA will be able to use the data to advocate for continued/additional LIHEAP funding.
- In the past, much of the focus on measuring success has been on the effort expended and resources dedicated to a program by a government agency (known as "inputs") and on the number of beneficiaries assisted (the "output"), rather than on the effect the program has on the lives of beneficiaries (the "outcome" or "impact"). However, input and output measures do not necessarily indicate success, if the assistance offered is not helpful to the person or entity receiving it. For this reason, there has been an increased emphasis on looking at the effect programs have on their recipients in order to measure program success.
- LIHEAP Statute, As Amended Through August 1, 1999 states: Not later than 18 months after the date of the enactment of the Low-Income Home Energy Assistance Amendments of 1994, the Secretary shall develop model performance goals and measurements in consultation with State, territorial, tribal, and local grantees, that the States may use to assess the success of the States in achieving the purposes of this title. The model performance goals and measurements shall be made available to States to be incorporated, at the option of the States, into the plans for fiscal year 1997. The Secretary may request data relevant to the development of model performance goals and measurements.
- Performance measurement was introduced to LIHEAP through Title III of the Human Services Amendments of 1994 (Public Law 103-252) which reauthorized LIHEAP through FY 1999. HHS was tasked to develop (in close consultation with state, tribal, insular area, and local LIHEAP grantees) model LIHEAP performance goals and measures. Grantees could incorporate the model performance goals, at their option, in their LIHEAP programs for fiscal year 1997, to assess the success of the states in achieving the purposes of LIHEAP. OCS made the model performance goals and measurements available in November 1995 to LIHEAP grantees to be incorporated, at their option, into their LIHEAP plans for FY 1997.
- In 2002 OMB developed the Performance Assessment Rating Tool (PART) as a diagnostic tool to assess and improve program performance of federal programs. The PART assessments were to be used to help inform budget decisions and identify actions to improve program results through follow-up actions and working toward continual improvements in program performance.
- In 2003 LIHEAP went through the PART process. LIHEAP received a "Results Not Demonstrated" assessment rating because it had not been able to develop acceptable performance goals or collect data to determine program performance. Since then, OCS has made an ongoing effort to develop and implement meaningful performance measures and management.
- The LIHEAP Performance Measures Work Group (PMWG) was formed in June 2008 when the Office of Community Services' Division of Energy Assistance (DEA) requested the assistance of State LIHEAP Directors in developing outcome performance measures for LIHEAP. The PMWG includes State LIHEAP Directors who offered to work with DEA on this project. Members of the PMWG met with DEA staff on a monthly basis from June 2008 through June 2009, participated in a performance measures meeting convened by the National Energy Assistance Directors Association (NEADA) in September 2009, and participated in a three-day meeting with DEA staff in November 2009 to develop a draft proposal for a LIHEAP performance measurement system. This proposal identified 16 potential measures, broken down into four tiers that would demonstrate the impact that LIHEAP has on the households we serve. The final proposal can be found at: http://www.acf.hhs.gov/programs/ocs/resource/implementing-liheap-outcome-performance-measures.
- At the 2010 winter NEADA meeting the proposal was released and states were asked if anyone was interested in forming an implementation work group. The goal of this group is to develop a process that will identify the necessary tools and resources (financial and non-financial) to allow State grantees to accurately and comprehensively measure the impact of LIHEAP on meeting the home energy needs of low income households. The Performance Measures Implementation Work Group (PMIWG) has been meeting telephonically and in person since the spring of 2010.
- In January 2012 the PMIWG made their formal recommendations to the Office of Community Services (OCS), suggesting they proceed with three of the identified measures:
- Household energy burden is reduced Average annual primary energy costs Average annual income Average annual LIHEAP benefit
- Home energy crises are prevented Number of HH where LIHEAP prevented a home energy crisis
- Home energy services are restored Number of HH where LIHEAP benefits restored home energy
- The first Federal Register notice was published on June 6, 2013. This notice asked interested parties to comment on: " (a) Whether the proposed collection of information is necessary for the proper performance of the functions of the agency, including whether the information shall have practical utility; (b) the accuracy of the agency's estimate of the burden of the proposed collection of information; (c) the quality, utility, and clarity of the information to be collected; and (d) ways to minimize the burden of the collection of information on respondents, including through the use of automated collection techniques or other forms of information technology. Consideration will be given to comments and suggestions submitted within 60 days of this publication."
- August 5 - Present ACF and Apprise review comments and propose changes to the measures based upon feedback received.
- January 2014 the PMIWG met to finalize the measures, forms and instructions. The new performance measures are being introduced as "developmental measures" which affords ACF and the states the opportunity to work towards the collection of the data while all states are updating their data collection systems to capture the new requirements and the chance to review and revise the measures, as needed, over the next few years.
- Revised measures include:
- Benefit Targeting Index - Demonstrates whether your state is giving higher benefits to higher burden households.
- Burden Reduction Targeting Index - Shows how burden reduction for your high burden households compares to the burden reduction for the average recipient.
- Service Restoration - How many clients had service restored by LIHEAP
- Service Loss Prevention - Shows the number of clients who would have lost service if not for your intervention with LIHEAP funds.
- OCS opted to make some data optional such as the collection of consumption data (units of fuel), the collection of supplemental heating fuel, and information about air conditioning equipment.
- The Grantee Capacity Survey was completed by all state grantees. The results will help ACF/Apprise identify what type of technical assistance is needed to allow grantees' systems to capture and compile the data required for performance measures reporting.
- The second Federal Register notice was published on June 4, 2014. Comments from this notice went directly to OMB (Office of Management & Budget).
- On November 18, 2014 states were notified that OMB had approved the four new performance measures as developmental measures while grantees are building the capacity to collect the necessary data.
- While reporting on FY 2015 data is optional, states that can report the new data for FY 2015 should do so on the Performance Measures Report due in January 2016. All states will be required to collect the data for FY 2016 and report that data in January 2017.
The PMWG finds that LIHEAP faces significant challenges in developing a performance measurement system:
- The program is a block grant and each grantee can use a number of different strategies to address the needs of eligible households, including: energy assistance, energy services, and client services. As such, the performance measurement system implemented by each grantee could be somewhat different depending on the program strategies that it employs.
- The long term outcome of the program needs to be measured outside the context of an individual state program, since the effectiveness of the program must be considered in the context of the eligible, rather than the recipient population. However, at the same time, the PMWG felt that it is important to develop short-term and intermediate-term performance measurement indicators that give state program administrators information on the effectiveness of their programs.
- States will need to develop client waivers for the collection of annual usage information from vendors.
- States are in different stages, technologically, with some states still operating off of paper. For these states, the development of a computer system to capture the information will be a challenge to develop and pay for. LIHEAP Coordinators will have to work closely with their IT folks to make sure they can capture and collect all the necessary data elements.
The PMIWG - Tools/Communications Developed to Assist States
State Engagement The work group has used various methods to educate states on the LIHEAP performance measures and to obtain their feedback and involvement in the implementation planning process. The following activities were organized to assess states' data collection capacity and to increase their awareness around LIHEAP performance measures data:
- State Grantee Needs Assessment Survey - an online survey that was designed and administered to gauge grantees' capacity to collect data related to the proposed LIHEAP performance measures. The survey received a 90 percent response rate and the findings were disseminated to state LIHEAP directors through an Executive Summary report. From the survey responses, it appears that a significant number of states need help in collaborating with outside parties to collect performance measures.
- LIHEAP Performance Measures Newsletter - an informational newsletter that is distributed to communicate to state grantees about LIHEAP performance measures, PMIWG activities, and best practices from the field.
- NEADA Presentations - presentations have been delivered during National Energy Assistance Directors Association (NEADA) midyear and winter board meetings to build awareness of LIHEAP performance measures, provide PMIWG progress updates, and debut data collection tool and templates for states to use.
- NEUAC Workshops - the Performance Measures Implementation Workgroup has conducted workshops and presentations at the 2011, 2012, and 2013 National Energy and Utility Affordability Conferences (NEUAC). Workshops have provided best practices from work group members on building relationships with stakeholders (e.g., utility companies, vendors, public utility commissions, etc.) to obtain data to measure performance and improve service delivery. Performance measures will be covered again at the 2015 conference as well at the regional trainings being sponsored by NEADA/ACF.
- State Grantee Webinars - webinars have been offered to explain processes, to present data collection tools and templates to states, and to encourage the states to use them in order to collect and report on LIHEAP performance measures.
Data Collection The PMIWG has conducted outreach activities and developed data collection tools to encourage data submission in the most efficient and user-friendly manner. Included below are the outreach activities and data collection tools the work group has completed thus far:
- Outreach - coordinated outreach was done in order to motivate states to submit data on some or all of the LIHEAP performance measures and to describe the advantages of collecting and using this data to improve service delivery. Data collection outreach was conducted through emails, presentations, webinars, and one-on-one exchanges with state grantees.
- Vendors - Below are links to two tables that list vendors that operate across multiple states.
- Process Guidance Tools - tools to assist states with incorporating performance measures data collection into their existing administrative and programmatic processes were created and preliminary prototypes were disseminated. These tools provide guidance that takes states through the data collection process, from the data source to data entry and application.
- Data Definitions - states have been provided with clear guidance on data definitions so there is uniformity and consistency among the states in the collection and calculation of the new LIHEAP performance measures. " Data Repository - an initial online data repository, Zoho, was created to allow easy submission, storage, and reporting of LIHEAP performance measures during the developmental stages of the implementation process. States are being encouraged by the PMIWG members to complete their data reporting.
- PMIWG Website - APPRISE (Applied Public Policy Research Institute for Study and Evaluation) was awarded a contract to develop a website to be used to collect data, share best practices, and generate reports state grantees can use to show the program's effectiveness and impact. The site has been set up and is password protected. It can be accessed directly or through OLDC. NCAT (the Clearinghouse) can provide login assistance.
Work committees, comprised of state coordinators, have devoted a significant amount of time and energy; and ACF has committed dollars to develop realistic, workable, performance measures that will help ACF report the value of LIHEAP to Congress and will help the states explain, report, and demonstrate how our state residents' lives are improved thanks to LIHEAP. States will be required to collect performance measures data beginning in FY 2016; to be reported at the end of that federal fiscal year.
- Performance Measures Work Group Website: https://liheappm.acf.hhs.gov/ (only available to state grantees that have a log-on). Please see your state LIHEAP coordinator for more information.
- Administration for Children and Families, Office of Community Services Website: http://www.acf.hhs.gov/programs/ocs.
- LIHEAP Information specifically use this url: http://www.acf.hhs.gov/programs/ocs/programs/liheap. Scroll down this page until you get to the topics in blue under LIHEAP Guidance, Policies and Procedures to access information on:
- Information Memoranda
- Training Materials and Webinars under "Grantee Training Materials"
- The Performance Measures Website under "Performance Measures"
- The LIHEAP Clearinghouse Website under "LIHEAP Clearinghouse"