2008 Open Ended Coding Project

Since its inception, the ANES has asked respondents open-ended questions that they answered in their own words.

During most of the duration of the project, the open-ended text was not made available to users unless a special request was made, and procedures were implemented to protect the confidentiality of the text. Instead, analysts were given the results of “coding” of the responses. This coding process involved assigning each answer to one or more categories for use in analysis.

To provide the user community with coded data whose meaning could be well understood, in 2008, we sought to apply best practices during all of the stages of assigning numeric codes to CFA question responses. These stages included the development of a theoretically defensible coding framework that human coders could implement, the development of instructions telling human coders how to apply the framework, using multiple independent coders to document important properties of the coding process, and public disclosure of all of our procedures and results.

In recent years, social scientists have paid increased attention to developing and implementing such practices. Collectively, the work of these individuals reveals criteria that can increase the credibility and reliability of coded open-ended data. These criteria, many of which were identified by DeBell (2013), include:

1. Development of codes to be applied to open-ended answers
   • A substantive rationale is articulated for the construct validity of the code categories.
   • Code categories are mutually exclusive and collectively exhaustive.
2. Development of instructions by which the codes are applied.
   • Coders follow specific and comprehensive rules for assigning code categories to open-ended data.
   • Coding rules are tested to assess inter-coder reliability with subsets of data prior to being fully implemented.
   • High inter-coder reliability suggests the instructions are effective, coders are following them consistently, and full coding should proceed.
   • Low reliability suggests the instructions are not effective and should be modified, or that coders are not following correctly and should be retrained or replaced. Disagreements among coders should be investigated to diagnose reasons for low reliability.
3. Independent coding by multiple coders
   • Coding is performed by coders working with records of the open-ended responses, rather than on the fly by interviewers during the interview.
   • Two or more coders assign code categories to all open-ended data.
   • All coders work independently and do not discuss their coding work with each other.
   • Any coder question is directed to a single individual who generates an answer that is distributed simultaneously to all coders.
   • After all coders have independently coded all open-ended data, disagreements between coders are identified and returned to the coders for resolution. The original independent coders explain reasons for their original coding to each other, and collaborate to converge on a single coding that both coders agree is accurate.
4. Public disclosure of all procedures and results
   • The logic underlying the code categories and the procedures used to create the code categories are publicly disclosed.
   • Coding rules used during independent coding are documented and publicly disclosed.
   • Inter-coder reliability of full independent coding is measured and publicly disclosed.
   • Source data (transcribed open-ended responses) are publicly disclosed.

How the ANES Coded Open-ended Answers in the Past

We could find no evidence that the ANES had consistently followed these practices prior to the 2008 Time Series Study. Coding instructions were often been vague or undocumented or both. Because the coding was done using open-ended responses that were never released to the public before 2008, it was impossible for scientists to scrutinize the coding. Typically, the coding was performed by just one person, and reliability was not reported and perhaps never calculated.

Moreover, we found that pre-2008 ANES codes were not developed using identical procedures from year-to-year, which limits scholars’ abilities to accurately track trends in over time. Initial evidence for this finding came from examining the codes used for answers to the CFA questions during three preceding ANES presidential election year studies (1996, 2000, and 2004). We found that new codes had been added with every successive study, though the rationale for these changes was neither documented nor clearly consistent across years.

Another problem involves code categories that were used in multiple years but had modified definitions from one year to the next. For example, the 2000 CFA code frame included 22 code definitions that were modified from codes used in 1996. The 1996 code frame included categories for a candidate’s connection to “Poor people/needy people/the unemployed”, while the definition for those same categories refer to “Poor people/needy people/handicapped/disabled” in 2000. 31 of codes in the 2000 code frame were modified from codes used in the 2004 code frame. Changes in code category definitions from one year to the next create problems when scholars attempt to use these categories to track trends over time. Time trends may be distorted by unstable category definitions.

As we explored past coding practices, we also discovered that coders used very few of the nearly 600 available codes with any frequency. For example, the 10 most used codes for the candidate likes/dislikes questions were applied to 29% of the answers in 1996, 26% of the answers in 2000, and 29% in 2004. The 30 most used codes in these years were used for more than half of the responses (60% in 1996, 51% in 2000, and 60% in 2004). Thus, in these years, only about 5% of the available codes were applied to more than 50% of the data. By contrast, 42% of the candidate likes/dislikes codes available for the 1996 study were never used, 47% were never used in 2000, and 41% were never used in 2004. Additionally, of the codes that were available in all three studies, 35% were never used for a single response in a single study. In sum, while the pre-2008 ANES code frame included a large number of choices, coders tended to use only a small number of available codes to characterize responses.

The reports above and the coding done of open-ended responses in 2008 are intended to improve on all this by implementing best practices.