The projects included in the Research Forum's database have been carefully selected according to specific criteria used in our review process. Nonetheless, some aspects of these projects may be of better quality than others.

When using the Research Forum's database -- or reviewing any research project-- we recommend that you pay attention to:

• The types of studies within the research project, including impact studies and implementation/process studies;
• The design of these studies, particularly for impact studies;
• The data sources (or data collection methods) used, including response rates and attrition; and
• The timeliness of this database content.

Types of Studies

Impact Study

An impact study aims to detect the effects of policies or programs on particular outcomes, such as earnings, welfare receipt, or child well-being. Impact studies generally compare one or more groups 'exposed' to a particular policy or program with an unexposed group.

Impact Study with Controlled Experiment

Evaluation researchers generally agree that controlled experiments with random assignment are the 'gold standard' for assessing impacts of programs. Under this design, individuals are assigned at random to different types of programs, such as the traditional AFDC program or a TANF program entailing work requirements and time limits. In comparison to other impact study designs, random assignment has an important advantage: one may assume that the experimental and control groups are equivalent on all factors, except the program in which they participate. Findings from experimental designs have strong internal validity. That is, one may assume that differences between experimental and control groups are due to differences in program content.

Experimental designs have a number of limitations. These are succinctly described in a Mathematica publication entitled Approaches to Evaluating Welfare Reform: Lessons from Five State Demonstrations.

Before TANF, the Department of Health and Human Services permitted states to offer alternative programs to AFDC as long as an experimental design was incorporated into an evaluation. Many of these projects are included in the Research Forum's database. The passage of the Personal Responsibility and Work Opportunities Reconciliation Act of 1996 reduces the use of experimental design; states and localities now have the authority to design income support programs within broad parameters established by the federal government (e.g., life time limit of 5 years of support) without the requirement for controlled experimental evaluations. In this policy environment, other types of research design will be increasingly relied upon to evaluate impact.

Impact Study with Quasi-Experimental Design

Quasi-experimental designs are one common alternative to controlled experiments with random assignment. These may either employ control groups selected to be similar to the intervention group (non-equivalent controls) or they may compare groups prior to the intervention with groups after the intervention has occurred (pre-post time periods). The non-equivalence of program participants and control group members constitutes the main threat to the internal validity of quasi-experiments.

The quality of quasi-experimental designs depends upon a number of factors, including

• choice of control group(s),
• the number of time periods observed, and
• the ability to statistically control for possible difference between groups compared.

A well-designed quasi-experimental study generally has an advantage over an experimental study with respect to external validity--i.e., the ability to generalize findings across persons, places, and time. The strengths and weaknesses of various types of quasi-experimental designs are also described in the Mathematica publication referenced above (see also Cook, T. & Campbell, D. Quasi Experimentation: Design and Analysis Issues for Field Settings. Boston: Houghton Mifflin, 1979).

Implementation/Process Study

Implementation studies (also called process evaluations) have become increasingly important in evaluating the effects of social policies and programs. These studies are designed to track how well components of a program are put in place. They have many purposes:

• to determine whether legislative intent has been met,
• to identify management trouble spots, and
• to isolate components of programs that may have positive results.

There are few agreed upon standards for doing good implementation studies, although the Research Forum, Institute for Research on Poverty and other researchers are exploring the development of such standards. As this work progresses, we will add relevant information to this web site.

Data Sources

Administrative Data

Many of the projects in this database use some type of "administrative" data. These are data originally collected for administrative purposes and include caseload counts, child support records, unemployment insurance receipt and the like. When the information is available, we outline the particular types of administrative data used under "Data Source Title." Please be aware that the quality of administrative data may differ between states and between agencies within states. Comparability across states may be limited if different definitions or criteria for reporting are used. As this source of data becomes increasingly used for evaluating welfare reform, attempts to standardize measurement and enhance quality are being initiated . We will keep you informed about these efforts as they progress.

Survey Data

In assessing the quality of survey research, we recommend consulting The American Association of Public Opinion Research's (AAPOR) outline of 'best practices' for survey researchers. In addition to these practices, the database user should be aware of how to interpret reports of response rates and attrition.

Response Rate

Response rate refers to the proportion of eligible persons that researchers interview. For example, in a household survey of persons aged 18-64, the response rate is calculated by dividing the number of persons interviewed by the total number of eligible persons in the sample. In survey research, acceptable response rates vary somewhat by mode of administration. When interviews are conducted in person, researchers generally achieve higher response rates than they do for interviews conducted by telephone or by mail. A response rate of at least 80% is considered desirable for in-person surveys. Standards for mail and telephone surveys are somewhat lower, with rates of 65% or better considered acceptable. (See Babbie, Earl R. The Practice of Social Research, 7th ed,. Belmont : Wadsworth, 1995; Dillman, D. A. Mail and Telephone Surveys : The Total Design Method, New York : Wiley, 1978.)

Attrition

Attrition refers to the loss to follow-up of sample members over time. This becomes important in longitudinal research where persons are interviewed at multiple time points. There is some variation in how attrition is assessed. Some investigators attempt to interview the original (baseline) sample members at each follow-up period while others interview just those who responded to the prior interview.

In our database, we note the response/attrition rate as reported by the evaluator, when available. If provided, we also report the method used to calculate these rates -- namely the characteristics of individuals/units from whom data are obtained (the numerator) and characteristics of individuals/units from whom data were sought (the denominator).

Timeliness of Database Content

The projects included in this database are at various stages of development. With some, data collection has yet to begin, with others final reports have been produced. For some projects the information in the database may not reflect the current status of the research. As of December 2005, the database was archived with no plans to update information in the database.