TY - JOUR
T1 - An extended validity argument for assessing feedback culture
AU - Rougas, Steven
AU - Clyne, Brian
AU - Cianciolo, Anna T.
AU - Chan, Teresa M.
AU - Sherbino, Jonathan
AU - Yarris, Lalena M.
N1 - Publisher Copyright:
© 2015, Taylor & Francis Group, LLC.
PY - 2015
Y1 - 2015
N2 - Construct: This study sought to develop a construct for measuring formative feedback culture in an academic emergency medicine department. Four archetypes (Market, Adhocracy, Clan, Hierarchy) reflecting an organization’s values with respect to focus (internal vs. external) and process (flexibility vs. stability and control) were used to characterize one department’s receptiveness to formative feedback. The prevalence of residents’ identification with certain archetypes served as an indicator of the department’s organizational feedback culture. Background: New regulations have forced academic institutions to implement wide-ranging changes to accommodate competency-based milestones and their assessment. These changes challenge residencies that use formative feedback from faculty as a major source of data for determining training advancement. Though various approaches have been taken to improve formative feedback to residents, there currently exists no tool to objectively measure the organizational culture that surrounds this process. Assessing organizational culture, commonly used in the business sector to represent organizational health, may help residency directors gauge their program’s success in fostering formative feedback. The Organizational Culture Assessment Instrument (OCAI) is widely used, extensively validated, applicable to survey research, and theoretically based and may be modifiable to assess formative feedback culture in the emergency department. Approach: Using a modified Delphi technique and several iterations of focus groups amongst educators at one institution, four of the original six OCAI domains (which each contain 4 possible responses) were modified to create a 16-item Formative Feedback Culture Tool (FFCT) that was administered to 26 residents (response rate D 55%) at a single academic emergency medicine department. The mean score of each item on the FFCT (range D 0-100) was analyzed. Convergent and divergent properties of the four archetypes were assessed using a multitrait-multimethod matrix of Pearson’s coefficients. Expecting that items in one archetype would diverge from the others, whereas items within an archetype should have strong convergent properties, convergent validity was assessed by comparing items across domains that all related to the same archetype. Similarly, divergent validity was assessed by comparing the correlation of items within an archetype to the correlations of those items within a hetero-domain block (i.e., to other items within the same domain). Results: Three of the four domains of the FFCT (Overall Departmental Characteristics 35.4 § 15.4, Departmental Foundation of Feedback 46.1 § 16.7, and Departmental Emphasis of Feedback 30.3 § 17.7) had the highest mean in the Market archetype (results/achievement oriented), whereas the final domain (Departmental Definition of Successful Feedback 34.8 § 22.1) had the highest mean in the Clan archetype (personal growth/team achievement). Item responses in the Clan and Hierarchy archetypes had the strongest convergent and divergent validity, respectively. Item responses in the Adhocracy archetype had the weakest convergent and divergent validity. Conclusions: Although the sample size was small, this initial study demonstrates that a modified organizational culture assessment tool can feasibly be utilized to identify the primary formative feedback archetype of a cohort of residents. This may have future implications for measuring changes in culture after the implementation of strategic programs to address formative the FFCT to other institutions, as well as address the weak feedback. Future studies should examine the generalizability of validity evidence of the Adhocracy archetype in the FFCT.
AB - Construct: This study sought to develop a construct for measuring formative feedback culture in an academic emergency medicine department. Four archetypes (Market, Adhocracy, Clan, Hierarchy) reflecting an organization’s values with respect to focus (internal vs. external) and process (flexibility vs. stability and control) were used to characterize one department’s receptiveness to formative feedback. The prevalence of residents’ identification with certain archetypes served as an indicator of the department’s organizational feedback culture. Background: New regulations have forced academic institutions to implement wide-ranging changes to accommodate competency-based milestones and their assessment. These changes challenge residencies that use formative feedback from faculty as a major source of data for determining training advancement. Though various approaches have been taken to improve formative feedback to residents, there currently exists no tool to objectively measure the organizational culture that surrounds this process. Assessing organizational culture, commonly used in the business sector to represent organizational health, may help residency directors gauge their program’s success in fostering formative feedback. The Organizational Culture Assessment Instrument (OCAI) is widely used, extensively validated, applicable to survey research, and theoretically based and may be modifiable to assess formative feedback culture in the emergency department. Approach: Using a modified Delphi technique and several iterations of focus groups amongst educators at one institution, four of the original six OCAI domains (which each contain 4 possible responses) were modified to create a 16-item Formative Feedback Culture Tool (FFCT) that was administered to 26 residents (response rate D 55%) at a single academic emergency medicine department. The mean score of each item on the FFCT (range D 0-100) was analyzed. Convergent and divergent properties of the four archetypes were assessed using a multitrait-multimethod matrix of Pearson’s coefficients. Expecting that items in one archetype would diverge from the others, whereas items within an archetype should have strong convergent properties, convergent validity was assessed by comparing items across domains that all related to the same archetype. Similarly, divergent validity was assessed by comparing the correlation of items within an archetype to the correlations of those items within a hetero-domain block (i.e., to other items within the same domain). Results: Three of the four domains of the FFCT (Overall Departmental Characteristics 35.4 § 15.4, Departmental Foundation of Feedback 46.1 § 16.7, and Departmental Emphasis of Feedback 30.3 § 17.7) had the highest mean in the Market archetype (results/achievement oriented), whereas the final domain (Departmental Definition of Successful Feedback 34.8 § 22.1) had the highest mean in the Clan archetype (personal growth/team achievement). Item responses in the Clan and Hierarchy archetypes had the strongest convergent and divergent validity, respectively. Item responses in the Adhocracy archetype had the weakest convergent and divergent validity. Conclusions: Although the sample size was small, this initial study demonstrates that a modified organizational culture assessment tool can feasibly be utilized to identify the primary formative feedback archetype of a cohort of residents. This may have future implications for measuring changes in culture after the implementation of strategic programs to address formative the FFCT to other institutions, as well as address the weak feedback. Future studies should examine the generalizability of validity evidence of the Adhocracy archetype in the FFCT.
KW - Feedback
KW - Organizational culture
KW - Validity
UR - http://www.scopus.com/inward/record.url?scp=84954516930&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84954516930&partnerID=8YFLogxK
U2 - 10.1080/10401334.2015.1077133
DO - 10.1080/10401334.2015.1077133
M3 - Article
C2 - 26507990
AN - SCOPUS:84954516930
SN - 1040-1334
VL - 27
SP - 355
EP - 358
JO - Teaching and Learning in Medicine
JF - Teaching and Learning in Medicine
IS - 4
ER -