Observational Measurement of Behavior

Dr Paul J. Yoder, Ph.D. Professor, Department of Special Education, Peabody College, Vanderbilt University, Nashville, Tennessee 37203Dr. Paul Yoder has been studying the transition from prelinguistic to linguistic communication in multiple populations with disabilities for over two decades. He is a co-designer of Milieu Communication Teaching and has contributed to several studies examining the efficacy of this treatment. He teaches methods and measurement at Vanderbilt University.Primary research activities of Frank J. Symons, Ph.D., are supported by the National Institute of Child Health and Human Development (NICHD), and they focus on improving the assessment and treatment of severe self-injurious behavior among individuals with developmental disabilities and pervasive developmental disorders. Dr. Symons was a research scientist at the Frank Porter Graham Child Development Center at the University of North Carolina at Chapel Hill and a postdoctoral fellow at the John F. Kennedy Center at the Peabody College of Vanderbilt University in Nashville, Tennessee. He is the co-author of Behavioral Observation: Technology and Applications in Developmental Disabilities (Paul H. Brookes Publishing Co., 2000).

• 1: Introduction and Measurement ContextsOverviewDefinition of Systematic Observation Using Count CodingRationale for Systematic Observation Using Count CodingImportance of Falsifiable HypothesesThe Continuum of State-Likeness to Trait-LikenessContext-Dependent BehaviorPerson CharacteristicsGeneralized Behavioral TendenciesSkillsThe Relative Scientific Value of Different Objects of MeasurementEcological Validity and RepresentativenessConclusions and RecommendationsReferences2: Validation of Observational VariablesOverviewThe Changing Concept of ValidationUnderstanding Which Types of Validation Evidence Are Most Relevant for Different Research Designs, Objects of Measurement, and Research PurposesContent ValidationDefinition of Content ValidationDifferent Traditions Vary on the Levels of Importance Placed on Content ValidationWeaknesses of Content ValidationSensitivity to ChangeDefinition of Sensitivity to ChangeInfluences on Sensitivity to ChangeWeaknesses of Sensitivity to ChangeTreatment UtilityDefinition of Treatment UtilityWeaknesses of Treatment UtilityCriterion-Related ValidationDefinition of Criterion-Related ValidationPrimary Appeal of Criterion-Related ValidationWeaknesses of Criterion-Related ValidationConstruct ValidationDefinition of Construct ValidationDiscriminative ValidationNomological ValidationMultitrait, Multimethod ValidationAn Implicit “Weakness” of Science?RecommendationsReferences3: Measuring Person CharacteristicsOverviewContextual Measurement ErrorDefinition of Measurement ContextA Brief Overview of Measurement TheoryDefinition of Contextual Measurement ErrorRepresentativenessContextual Measurement Error in Measures Of Generalized Behavioral TendenciesAveraging Scores Across Contexts Improves Measures of Generalized Behavioral TendenciesAggregates Tend to Improve Estimates of Known True Score.Aggregates Tend to Improve Construct Validity.Aggregates Tend to Improve Stability.Controlling Influential Contextual Variables Stabilizes Observed Scores for Highly Variable Person CharacteristicsWhy Naturalistic Observations Are Not Necessarily More Representative Than Contrived OnesWhy Skills Are Often Measured in Structured Measurement ContextsWhy Skills Are Often Assessed in Clinics or LabsThe Link Between Stability and Construct ValidityRecommendations and ConclusionsReferences4: Designing or Adapting Coding ManualsOverviewDefinition of a Coding ManualDeciding Whether to Use an Existing Coding Manual or to Construct a New OneRecommended Steps for Modifying or Designing Coding ManualsDefining When to Start and Stop CodingConceptually Defining the Context-Dependent Behavior or the Generalized CharacteristicDefining the Highest Level of Codable BehaviorDetermining the Level of Distinction Coders Have to MakeOrganizing the Coded Categories into Mutually-Exclusive SetsPhysically Based Definitions, Socially Based Definitions, or Both?Defining the Lowest Level CategoriesSource of Conceptual and Operational DefinitionsA Qualitative Approach to Identifying DefinitionsDefining Segmenting RulesThe Potential Value of FlowchartsDo Coding Manuals Need to be Sufficiently Short to be Included in Methods Sections?Recommendations and ConclusionsReferences5: CodingOverviewThe Elements of an Observational Measurement SystemBehavior SamplingThe Superordinate Distinctions: Continuous versus IntermittentThe Subordinate Distinctions: Timed-Event versus Event versus IntervalTimed-Event SamplingEvent SamplingInterval SamplingTypes of Interval SamplingWhole Interval SamplingMomentary Interval SamplingPartial Interval SamplingThe Importance of Knowing What Metric the Investigator Wants to EstimateSummary of Behavior SamplingParticipant SamplingFocal SamplingMultiple Pass SamplingConspicuous SamplingReactivityLive Coding versus Recording the Observation For Later CodingLive CodingCoding from Recorded SessionsRecording Coding DecisionsRecommendations and ConclusionsReferences6: Common Metrics of Observational VariablesOverviewDefinition of MetricQuantifiable Dimensions of BehaviorProportion MetricsProportion Metrics Change the Meaning of Observational VariablesScrutinizing ProportionsAn Implicit Assumption of Proportion MetricsTesting Whether the Data Fit the Assumption of Proportion MetricsConsequences of Using a Proportion When the Data Do Not Fit the AssumptionAlternative Methods to Control Influential Contextual VariablesStatistical ControlProcedural ControlAggregate Measures of Generalized Person CharacteristicsWeighted CountUnit-Weighted AggregatesGroup Analysis of Observational VariablesTransforming the MetricBootstrappingRecommendations and ConclusionsReferences7: Observer Training and Preventing Observer DriftOverviewPoint-by-point Agreement and DisagreementPoint-by-point Agreement of Interval Sampled DataPoint-by-point Agreement of Timed-Event DataDiscrepancy MatricesDiscrepancy DiscussionsUsing Discrepancy Discussions to Train ObserversCreating Criterion-coding Standards.Remaining Steps to Train ObserversPreventing Observer DriftMethod of Selecting Sessions for Agreement ChecksRemaining Steps to Preventing or Addressing Observer DriftRecommendationsReferences8: Interobserver Agreement and Reliability of Observational VariablesOverviewAdditional Purposes of Point-by-Point AgreementAdded Principles When Agreement Checks Are Used to Estimate Interobserver “Reliability” of Observational Variable ScoresExhaustive Coding Spaces RevisitedThe Effect of Chance on AgreementCommon Indices of Point-by-Point AgreementOccurrence Percentage AgreementNonoccurrence Percentage AgreementTotal Percentage AgreementKappaBase Rate and Chance Agreement RevisitedIntraclass Correlation Coefficient (ICC) as an Index of Interobserver Reliability in Group DesignsOptions for Running ICC With SPSSBetween-Participant Variance on the Variable of Interest Affects ICCUsing ICC as a Measure of Interobserver Reliability for Predictors and Dependent Variables in Group DesignsThe Interpretation of SPSS Output for ICCThe Conceptual Relation Between Interobserver Agreement and ICCConsequences of Low or Unknown Interobserver ReliabilityRecommendationsReferences9: Introduction to Sequential AnalysisOverviewDefinition of Terms Used in this ChapterSequential versus Nonsequential VariablesSequential Associations are not Sufficient Evidence for Causal InferencesCoded Units and ExhaustivenessContingency TablesThree Major Types of Sequential AnalysisEvent LagConcurrent IntervalEvent Lag with Pauses (to replace time window method)Explanation for no longer focusing on time windowIndices of Sequential Association: Controlling for ChanceExisting Indices of Sequential Association: Advantages and DisadvantagesTransitional/conditional probabilitiesYule’s QRisk Difference/Operant Contingency ValueOther commonly-used indices and why we do not focus on them (e.g., z)Recommendations and ConclusionsReferences10: Identifying and Addressing Research Questions Involving Sequential AssociationsOverviewSequential Analysis in Group DesignsTypes of Research Questions and Methods to Address ThemIndices of Sequential Association as Dependent VariablesTesting the Significance of a Mean Sequential AssociationTesting the Between-Group Difference in Mean Sequential AssociationsTesting the Within-Subject Difference in Sequential AssociationsSequential Analysis in Single-Case DesignsTypes of Research QuestionsDescriptive Questions Involving Behavior-Environment Associations to Inform Experimental AnalysesDescriptive Questions to Inform Temporal Distribution of One or More Target BehaviorsIndices of Sequential Association as Dependent Variables in Single-Case DesignsIndices of Sequential Association to Inform Procedural FidelityMethods to Address Sequential Analysis Research Questions in a Single Case FrameworkWhy Significance Testing is Inappropriate at the Level of the Individual ParticipantMore on the Term ‘Operant Contingency’Summary of Analysis Methods Used in Behavior Analytic LiteratureConditional ProbabilitiesLag Sequential AnalysisContingency Space AnalysisWhat is “Enough Data” and How Do We Attain It?Proposed Solutions for Insufficient DataSummary and RecommendationsReferences11: Generalizability TheoryOverviewScope of This ChapterOverview of G Theory and Definition of TermsAn Example Observer by Context G and D StudyRationale for Preferring the Absolute G CoefficientExample Applications of D StudiesAn Ongoing ControversyRecommendations and ConclusionsReferences12: Best Practices in Observational MeasurementGlossaryIndex