You want to quantify that?! The science and metrics of partner engagement in research


>>THE COLOR COMMITTEE OF THE TRANS-NIH WORKING GROUP ON WOMEN IN BIOMEDICAL CAREERS, AND WE’RE VERY HAPPY TO BE ABLE TO HOST DR. MELODY GOODMAN TODAY. DR. GOODMAN RECEIVED HER BACHELOR’S IN SCIENCE IN APPLIED MATHEMATICS, STATISTICS AND ECONOMICS FROM STONY BROOK UNIVERSITY. SHE, FOR A PERIOD OF TIME, WAS ON WALL STREET AND THEN SAW THE LIGHT AND DECIDED TO GET HER MASTERS IN BIOSTATISTICS FROM THE HARVARD SCHOOL OF PUBLIC HEALTH AND HER PH.D. IN BIOSTATISTICS FROM HARVARD IN THEORETICAL STATISTICS AND SOCIAL DETERMINANTS OF HEALTH DISPARITIES, AN EMPHASIS THAT SHE DEVELOPED HERSELF FOR HARVARD. SHE’S AN ASSOCIATE PROFESSOR OF BIOSTATISTICS IN THE SCHOOL OF PUBLIC HEALTH AT NEW YORK UNIVERSITY, SHE’S SUPPORTED BY LONG ISLAND COMMUNITY FOUNDATION, PCORI, THE SUSAN G. COMAN FOUNDATION, SHE’S WELL PUBLISHED WITH MORE THAN 90 REFEREED JOURNAL ARTICLES, TWO BOOKS, AND SHE’S KNOWN AS A RESEARCH METHODOLOGIST WITH A LARGE STATISTICAL TOOLBOX. HER RESEARCH INTEREST IS ON IDENTIFYING ORIGINS OF HEALTH DISPARITIES AND DEVELOPING AS NECESSARY EVIDENCE-BASED PRIMARY PREVENTION STRATEGIES TO REDUCE HEALTH DISPARITIES. SO I KNOW DR. GOODMAN HAS A LOT OF REALLY GOOD INFORMATION TO SHARE WITH YOU. I’M GOING TO ASK HER TO COME TO THE PODIUM AND TALK WITH US. [APPLAUSE]>>GOOD AFTERNOON, EVERYONE. THANK YOU FOR HAVING ME. I’M EXCITED TO TALK ABOUT THE SCIENCE AND METRICS OF PARTNER ENGAGEMENT IN RESEARCH. SO I WANT TO START WITH JUST A WORKING DEFINITION OF COMMUNITY ENGAGEMENT THAT I’LL BE USING THROUGHOUT MY TALK, AND THIS IS THE PROCESS OF WORKING COLLABORATIVELY WITH AND THROUGH GROUPS OF PEOPLE AFFILIATED BY GEOGRAPHIC PROXIMITY, SPECIAL INTEREST OR SIMILAR SITUATIONS TO ADDRESS ISSUES OF AFFECTING THE WELL-BEING OF THOSE PEOPLE. IT IS A POWERFUL VEHICLE FOR BRINGING ABOUT ENVIRONMENTAL AND BEHAVIORAL CHANGES THAT WILL IMPROVE THE HEALTH OF THE COMMUNITY AND ITS MEMBERS, AND IT OFTEN INVOLVES PARTNERSHIPS AND COALITIONS THAT HELP MOBILIZE RESOURCES AND INFLUENCE SYSTEMS, CHANGE RELATIONSHIPS AMONG PARTNERS, AND SERVE AS A CATALYST FOR CHANGING POLICY, PROGRAMS AND PRACTICES. SO WE DO STAKEHOLDER ENGAGED RESEARCH FOR A VARIETY OF REASONS, TO ENSURE RESEARCH IS PATIENT OR COMMUNITY CENTERED, NON-ACADEMIC STAKEHOLDERS PROVIDE UNIQUE PERSPECTIVE ON THEIR NEEDS, INCREASED RELEVANCE TO NON-ACADEMIC STAKEHOLDERS INCLUDING PATIENTS AND COMMUNITIES AND POLICY MAKERS, SUPPORT SUSTAINABILITY OF INTERVENTIONS POST FUNDING, BUILD CAPACITY AND TRUST AMONG ALL THE STAKEHOLDERS INVOLVED, LEVERAGES EXISTING RESOURCES WITHIN THE COMMUNITY, IT INCLUDES RECIPROCAL RELATIONSHIPS BETWEEN RESEARCHERS AND NON-ACADEMIC STAKEHOLDERS, AND IT IS AN EVIDENCE-BASED APPROACH FOR ADDRESSING HEALTH DISPARITIES. PART OF WHAT BRINGS ME TO THIS WORK, I THINK, IS MOTIVATED BY AN AFRICAN PROVERB THAT SAYS IF YOU WANT TO GO FAST, GO ALONE, BUT IF YOU WANT TO GO FAR, GO TOGETHER. AND I HOPE THAT WE ALL WANT TO GO FAR IN OUR SCIENCE. SO WHY MEASURE PARTNER ENGAGEMENTS? THE EXTENT TO WHICH STAKEHOLDERS AND RESEARCH PARTNERSHIPS FEEL ENGAGED HAS NOT RECEIVED SCIENTIFIC ATTENTION. IT IS IMPORTANT TO UNDERSTAND HOW ENGAGEMENT LEVEL IN A PARTNERSHIP IS DEVELOPING AND TO WHAT EXTENT ENGAGEMENT LEVEL IS A PREDICTER OF OUTCOMES IN THE LARGER STUDY. SO I’M GOING TO TALK ABOUT OUR WORK IN THIS SPACE AND WALK YOU THROUGH SOME OF OUR APPROACH. THESE ARE THE ELEMENTS WE TOOK. WE STARTED WITH A SYSTEMATIC REVIEW, WE DID SOME INITIAL DEVELOPMENT OF A NEW MEASURE, WE ARE REFINING AND VALIDATING THE NEW MEASURE, AND I’M GOING TO TALK ABOUT SOME UPCOMING WORK INCLUDING DEVELOPING A COMPUTER ADAPTIVE ALGORITHM AND OUR IMPLEMENTATION STUDY WHICH WE’RE GETTING READY TO LAUNCH IN THE NEW YEAR. SO FIRST WE DID A SYSTEMATIC REVIEW OF THE LITERATURE. THIS PAPER IS PUBLISHED IN CLINICAL AND TRANSLATIONAL SCIENCES, AND IT WAS A TEAM OF US, WE LOOKED THROUGH ALL PAPERS THAT HAD EXISTING MEASURES OF STAKE HOLD EVERY ENGAGEMENT AND BASICALLY CLASSIFIED THOSE MEASURES INTO SOME GROUPS. WE STARTED THIS WORK BY THINKING THAT SUCH MEASURES EXISTED AND THAT THEY HAD PROPERTIES THAT WERE UNDERSTOOD. WE FOUND THAT LOTS OF PEOPLE HAD MEASURED SOMETHING BUT REALLY DIDN’T KNOW WHAT THEY HAD MEASURED. AND THE FIELD WAS NOT VERY STRONG METHODOLOGICALLY. THE EXISTING MEASURE WE FOUND CAME IN TWO CAMPS SO WE GROUPED THEM IN THIS WAY, ONE IN WHICH INVESTIGATORS SIMPLY COUNTED THE ATTENDANCE IN VARIOUS EVENTS AND ACTIVITIES AND ASSUMED THAT THAT WAS ENGAGEMENT. ONE OF THE THINGS WE KNOW IS THAT SHOWING UP DOESN’T NECESSARILY MEAN YOU’RE ENGAGED. YOU MAY SHOW UP FOR THE PIZZA AND THE COOKIES BUT NOT REALLY BE ENGAGED IN THE CONVERSATION THAT IS HAPPENING. THEN THE OTHER SET OF MEASURES FELL IN THE CAMP IN WHICH INVESTIGATORS MEASURED SOME CONSTRUCT THAT WAS POSSIBLY RELATED TO ENGAGEMENT BUT NEITHER WAY HAD BEEN VALIDATED OR CORROBORATED, MOST WERE NOT IN OUTCOMES IN THE RESEARCH PROGRESS AND NONE HAD BEEN TRACKED OVER TIME. SO I JUST WANTED TO GIVE YOU A COUPLE OF EXAMPLES OF EACH OF THE METHODS THAT WE FOUND. SO IN THE COUNTING METHOD, WE FOUND THINGS LIKE COUNTING THE NUMBER OF PEOPLE WHO ATTENDED A BOARD MEETING OR COUNTS OF ATTENDEES AT A COMMUNITY MEETING, FREQUENCY OF ATTENDANCE AT PROCESS REPORTS. AND IN THE CONSTRUCT MEASUREMENT METHOD, WE FOUND PEOPLE MEASURED THE DEGREE TO WHICH PARTICIPANTS FELT THEY WERE A PART OF A POSITIVE COMMUNITY, THE DEGREE TO WHICH THEY FELT COMFORTABLE SHARING THEIR THOUGHTS AND OPINIONS AND THE LEVEL OF CONFIDENCE IN PARTICIPATE IN THE PROBLEM SOLVING PROCESS. ALL OF THESE THINGS ARE RELATED TO ENGAGEMENT BUT ARE NOT NECESSARILY MEASURING ENGAGEMENTS DIRECTLY. SO WE DEVELOPED AN INITIAL MEASURE, A QUANTITATIVE MEASURE OF DEVELOPMENT, AND I’LL TALK A LITTLE BIT ABOUT THAT. IT WAS BASED ON 11 ENGAGEMENT PRINCIPLES WHICH CAME FROM THE LITERATURE, BUT WE ALSO WORKED WITH A COMMUNITY ADVISORY BOARD TO MAKE SURE THAT THESE WERE THE THINGS THAT WE SHOULD MEASURE, INCLUDING THINGS LIKE FOCUS ON LOCAL RELEVANCE AND DETERMINANTS OF HEALTH, ACKNOWLEDGING THE COMMUNITY, DISSEMINATING FINDINGS AND KNOWLEDGE GAINED TO ALL PARTNERS, SEEK AND USE INPUT OF COMMUNITY PARTNERS, INVOLVING A CYCLICAL AND ITERATIVE PROCESS IN THE PURSUIT OF OBJECTIVE, TO FOSTER CO-LEARNING, CAPACITY BUILDING AND CO-BENEFIT FOR ALL PARTNERS, TO BUILD ON THE STRENGTHS AND RESOURCES WITHIN THE COMMUNITY, TO FACILITATE COLLABORATIVE AND EQUITABLE PARTNERSHIPS, TO INTEGRATE AND ACHIEVE A BALANCE OF ALL PARTNERS, TO INVOLVE ALL PARTNERS IN THE DISSEMINATION PROCESS, AND LASTLY, TO PLAN FOR A LONG TERM PROCESS AND COMMITMENT. OUR ITEM IS MEASURED ON TWO SCALES, SO THERE’S THREE TO FIVE ITEMS THAT ASSESSED EACH OF THOSE ENGAGEMENT PRINCIPLES. THEY USE LIKERT RESPONSE OPTIONS. WE HAVE ONE SCALE THAT MEASURES QUANTITY, HOW MUCH, AND ONE SCALE THAT MEASURES QUALITY, HOW WELL. THIS WAS ALSO DRIVEN BY OUR COMMUNITY STAKEHOLDERS THAT SAID YOU CAN DO A LOT OF SOMETHING POORLY OR YOU CAN DO A LITTLE BIT OF SOMETHING WELL AND SO THAT IT WAS IMPORTANT TO MEASURE BOTH QUANTITY AND QUALITY. JUST A HISTORICAL ASIDE, THE LIKERT SCALE WAS DEVELOPED AT NYU IN 1932, I WAS ABLE TO PULL THE ORIGINAL PAPER FROM THE ARCHIVE, I’M A NERD AND I THOUGHT IT WAS AMAZINGS TO
AMAZING TO FIN
D THAT. A SINGLE STREAMLINED QUESTION WITH A SCALED SET OF ANSWERS WAS IDEAL FOR STANDARDIZED QUESTIONS. WE FEL THAT EVEN THOUGH WE HAD DONE THIS SYSTEMATIC REVIEW THAT OTHER PEOPLE PROBABLY HAD BEEN MEASURING ENGAGEMENT BUT JUST NOT PUBLISHING IT IN THEIR WORK. SO WE WROTE THIS POSITION PAPER IN TRANSLATIONAL BEHAVIORAL MEDICINE AND IT WAS REALLY A CALL TO ACTION TO TRY TO GET PEOPLE TO START DISSEMINATING THE WORK THAT THEY WERE DOING, AND REALLY A SOAP BOX IRK EU FOR
ISSUE FOR ME IS TREATING PARTNER ENGAGEMENT ATION AS A SCIENCE,
HOW DO WE IMPLEMENT IT AND EVALUATE IT. SO IN THE PAPER WE PUT THIS FIGURE THAT WAS SOMEWHAT CONTROVERSIAL TO PEOPLE AT THE TIME BECAUSE WE GROUPED PARTICIPATION IN THREE CATEGORIES THAT WE CALLED NON-PARTICIPATION, SYMBOLIC PARTICIPATION AND ENGAGED PARTICIPATION. PEOPLE WHO WERE DOING OUTREACH IN EDUCATION DIDN’T LIKE THAT THEY WERE IN THE NON-PARTICIPATION WORK, BUT WE’RE NOT SAYING THAT PEOPLE SHOULDN’T BE DOING THAT WORK, AND NOT EVERY PROJECT REQUIRES ENGAGED PARTNERSHIP. SOME PROJECTS REALLY ONLY REQUIRE OUTREACH ENGAGEMENT SO IT REALLY DEPENDS ON YOUR STUDY AND THE PROJECT AND THE POPULATIONS THAT YOU’RE WORKING WITH. THE NICE THING ABOUT GETTING ACADEMICS FIRED UP IS THEN THEY START PUBLISHING SOME STUFF AND WORKING IN A SPACE THAT YOU WANT TO SEE OTHER PEOPLE MOVE SCIENCE FORWARD. WE SUBSEQUENTLY HAD A PATIENT-CENTERED OUTCOMES RESEARCH INSTITUTE STUDY THAT WAS FUNDED WITH FOUR SORT OF MAIN SPECIFIC AIMS TO EXAMINE THE CONSTRUCT VALIDITY ENGAGEMENT PRINCIPLE ON BOTH SCALES AND USE A STANDARDIZED DELPHI PROCESS FOR EXPERT VALIDATION BASED ON ADVOCACY GROUPS, PATIENT, FAMILY FEEDBACK AND PRIORITIZATION OF DOMAINS WE SHOULD BE MEASURING. WE ALSO WANT TO EXAMINE THE PSYCHOMETRIC PROPERTIES, WE’RE WORKING ON A SHORTER, MORE CONDENSED VERSION OF THE MEASURE BECAUSE IT’S REALLY TOO LONG TO BE USED WIDELY, AND THEN AT THE BEGINNING OF THE YEAR, WE’RE GOING TO EVALUATE ITS IMPLEMENTATION INTO PATIENT-CENTERED OUTCOMES RESEARCH STUDIES THAT ARE& ONGOING. SO OUR PHASE 1 WAS A DELPHI PROCESS. THERE’S A METHOD FOR COLLECTING AND ORGANIZING INFORMED OPINIONS FROM A GROUP OF EXPERTS USING AN ITERATIVE PROCESS, OFTEN USED IN SURVEY DEVELOPMENT. THIS APPROACH IS MOST APPROPRIATE TO ENSURE THAT FEEDBACK IS OBTAINED FROM ALL STAKEHOLDERS WITH ALL EXPERTS BEING TREATED EQUALLY. THIS WAS IMPORTANT TO US BECAUSE WE HAD BOTH ACADEMIC STAKEHOLDERS BUT ALSO COMMUNTY AND OTHER SORT OF TYPES OF PARTNERS AND WE WANTED EVERYONE’S VOICE TO BE HEARD EQUALLY, NOT JUST THE PEOPLE WITH THE LOUDEST VOICE OR THE PEOPLE WHO SPEAK THE MOST. THIS DELPHI TECHNIQUE AFFORDED A STAKEHOLDER ENGAGED MEASUREMENT DEVELOPMENT PROCESS WHICH WE ACTUALLY THINK IS REALLY IMPORTANT AND ONE OF THE BIGGEST OUTCOMES OF OUR STUDY, IS THAT IF YOU’RE DEVELOPING A MEASURE, YOU SHOULD REALLY ENGAGE THE POPULATION THAT WILL BE TAKING THAT MEASURE IN THE MEASURE DEVELOPMENT. WE USE WEB-BASED SURVEYS FOR ROUNDS 1 THROUGH 3 AND ALSO IN OUR FIFTH AND FINAL ROUND, BUT THE ROUND 4 WAS A 2-DAY IN-PERSON MEETING WHICH WAS REALLY CRITICAL TO HELP US REACH CONSENSUS ON THINGS THAT WERE HARD TO REACH CONSENSUS JUST BASED ON SURVEYS. THE RESPONSES WERE ANALYZED BY THE INVESTIGATOR TEAM AND EACH MEMBER OF THE DELPHI PANEL RECEIVED A SUMMARY REPORT THAT SHOWED THEIR RESPONSES ALONG WITH RESPONSINGS FROM OTHER MEMBERS OF THE DELPHI PANEL STRATIFIED BY WHETHER THOSE MEMBERS WERE ACADEMIC MEMBERS OR COMMUNITY OR PATIENT STAKEHOLDERS SO POEM PEOPLE
COULD SEE THE DIFFERENT VOICES AND WHO WAS SAYING WHAT, BUT THAT WAS ANONYMOUS SO THEY ONLY SAW THEIR RESPONSE AND GROUPED RESPONSES FROM EVERYONE ELSE. SO WE USED THIS FIVE ROUND DELPHI PROCESS AND WE STARTED WITH THINGS LIKE DEMOGRAPHICS JUST TO SORT OF KNOW WHO WAS IN OUR DELPHI PROCESS AND WE ACTUALLY STARTED WITH THE ENGAGEMENT PRINCIPLES THEMSELVES, DID WE AGREE THAT THESE WERE THE THINGS THAT SHOULD BE MEASURED, AND SOME ITEMS AND WE ALSO LOOKED AT SOME OTHER EXISTING MEASURES. ONE OF THE THINGS THAT BECAME REALLY CLEAR TO US IN THIS PROCESS WAS THAT DEFINITIONS WERE IMPORTANT SO THAT WE COULD BE ON THE SAME PAGE SO WE WORKED THROUGH DEFINITIONS FOR EACH OF THE ENGAGEMENT PRINCIPE TITLES, AND WE ALSO DEVELOPED SOME SCENARIOS SO WE HAVE LEVELS OF ENGAGEMENT LIKE I SHOWED YOU IN THAT FIRST FIGURE, AND WE HAVE SCENARIOS THAT ALIGN WITH EACH OF THOSE ENGAGEMENT LEVELS SO THAT PEOPLE COULD BE ON THE SAME PAGE ABOUT WHAT DOES IT MEAN TO BE DOING OUTREACH, WHAT DOES IT MEAN TO BE CONSULTATION, WHAT DOES IT MEAN TO BE DOING COOPERATION LEVEL RESEARCH, AND THEN ALL OF THESE THINGS WERE ITERATIVE, SO ANYTHING THAT DIDN’T REACH CONSENSUS IS THEN BROUGHT TO THE NEXT ROUND. AND LIKE I SAID, THE TWO-DAY IN-PERSON MEETING WAS AMAZING, THESE WERE HIGHLY ENGAGED GROUP OF PEOPLE BUT THEY GAVE US REALLY GREAT FEEDBACK. AND WE LEARNED THAT A LOT OF IT WAS ABOUT LANGUAGE. AND WE HAD THE GREAT FORTUNE OF HAVING AN EDITOR PARTICIPATE IN THAT TWO-DAY MEETING SO SHE REALLY HELPED US CRAFT LANGUAGE THAT THEN ENABLED US TO REACH CONSENSUS FOR THE PARTICIPANTS. AND BEFORE EACH ROUND, THEY WERE GIVEN A SUMMARY OF SORT OF WHAT HAPPENED IN THE LAST ROUND AND THEY WERE PREPARED TO TAKE THEIR WEB-BASED SURVEY OR WE DID SOME POLLING AT THE IN-PERSON MEETING. SO PANELISTS IN EACH ROUND ARE ENCOURAGED TO RECONSIDER THEIR PREVIOUS RESPONSES AND ONLY IF APPROPRIATE TO CHANGE THEIR PREVIOUS RESPONSES IN LIGHT OF THE REPLIES AND COMMENTS FROM OTHER PANELISTS. WE WERE NOT TRYING TO FORCE CONSENSUS, WE THOUGHT IT WAS IMPORTANT FOR PEOPLE TO STAND THEIR GROUND WHEN THEY THOUGHT THAT THAT WAS IMPORTANT BUT WE WANTED THEM TO GIVE US STRONG JUSTIFICATIONS OF WHY THEY DISAGREED. SO WE WERE ALSO REALLY ABLE TO REACH CONSENSUS, LIKE I SAID, A LOT OF IT WAS ABOUT LANGUAGE WHICH WE WERE ABLE TO FIX. OUR VERSION OF CONSENSUS WAS 80% AGREEMENT AMONG OUR DELPHI PANELISTS SO WE WERE JUST TRYING TO REACH 80% AGREEMENT. SO THESE ARE THE PARTICIPANTS THAT PARTICIPATED IN THAT TWO-DAY MEETING. IT WAS APRIL IN NEW YORK CITY, IT WAS A GREAT TIME, AND THEY WERE LOCKED IN A CONFERENCE ROOM FOR TWO DAYS TALKING ABOUT ENGAGEMENT, SO WE WERE RIGHT ACROSS THE STREET FROM THE PARK, SO THIS WAS A GREAT GROUP OF DEDICATED PEOPLE WHO WERE REALLY SORT OF INTERESTED AND INVESTED IN THIS WORK. OUR DELPHI PANELISTS WERE PRIMARILY FEMALE, PRIMARILY AFRICAN-AMERICAN, MOST HAD GRADUATE LEVEL EDUCATION. WE PURPOSELY HAD MORE COMMUNITY STAKEHOLDERS THAN ACADEMIC STAKEHOLDERS JUST BECAUSE WE WANTED THAT VOICE TO BE AMPLIFIED A LITTLE BIT MORE. WE WANTED REPRESENTATION FROM ALL THE REGIONS. MOST HAD LOTS OF EXPERIENCE DOING RESEARCH, PARTICULARLY COMMUNITY ENGAGED RESEARCH, BUT WE DID HAVE ONE PARTICIPANT WHO WAS NEW TO THIS TYPE OF WORK, BECAUSE WE THOUGHT IT WAS IMPORTANT TO HAVE THE PERSPECTIVE OF SOMEONE WHO HAD NOT BEEN ENGAGED IN THIS TYPE OF WORK BEFORE TO SEE HOW THEY WOULD RESPOND TO SOME OF THE ITEMS. SO WE RECENTLY RELEASED SOME OF THE WORK FROM THIS DELPHI PANEL, WHICH IS REALLY TO DO A CONTENT VALIDATION, ARE WE MEASURING THE RIGHT THINGS, ARE WE THINKING ABOUT THESE THINGS IN THE RIGHT WAY. SO I’M NOT GOING TO GO THROUGH ALL THIS INFORMATION BUT I DO WANT TO SHOW YOU THAT WE KEPT METICULOUS TRACK OF ALL THE ITEMS WE STARTED WITH, WHICH WAS 48, AND THEN HOW MANY ITEMS WERE NOT CHANGED, DROPPED, WE WERE TRYING NOT TO ADD ITEMS BUT ITEMS WERE ADDED IN ROUND ONE AND I’LL TALK ABOUT WHY THAT WAS, AND HOW MANY ITEMS WERE MODIFIED, INCLUDING SOME ITEMS REMOVED FROM ONE ENGAGEMENT PRINCIPLE TO ANOTHER. WE DID THIS FOR EACH ROUND TO SORT OF KEEP TRACK OF HOW THINGS WERE SORT OF PROGRESSING AS WE WENT THROUGH THE FIVE ROUND DELPHI PROCESS, SO WE WENT FROM 48 ITEMS AT THE BEGINNING OF THE PROCESS AND WE ULTIMATELY ENDED ON 32 ITEMS, STILL TOO LONG. WE’RE WORKING ON CUTTING IT DOWN, BUT THIS WAS A WAY FOR US TO SORT OF START TO HONE IN ON WHAT THE KEY POINTS WERE. THE MAIN THING THAT CAME OUT OF THE PROCESS IS WE WENT FROM MEASURING 11 ENGAGEMENT PRINCIPLES DOWN TO 8. THE FIRST SEVEN ARE FROM THE 11 ALTHOUGH SOME OF THEM HAVE BEEN RE-WORDED. THE LAST ONE WAS ADDED WHICH WAS AN IMPORTANT ONE OUR COMMUNITY PARTNERS BROUGHT OUT WHICH WAS ABOUT TRUST SO THE LAST PRINCIPLE IS BUILD TRUST, SOMETHING THEY FELT WAS HINTED AT IN THE OTHER PRINCIPLES BUT WASN’T SPOKEN TO DIRECTLY AND FELT THEY NEEDED TO DO IT. WE ALSO NAMED OUR MEASURE SO IT’S CALLED THE RESEARCH ENGAGEMENT SURVEY TOOL. WE CALL IT REST, WE THINK EVERY P.I. SHOULD STOP AND TAKE A REST AT SOME POINT IF THEY ARE DOING ENGAGEMENT-TYPE WORK. WE ALSO LOOKED AT CONSENSUS OF ALL OF THE TITLES AND WE WERE ABLE TO REACH HIGH LEVELS OF CONSENSUS FOR ALL OF THE TITLES, MEANING WHAT WE WERE TRYING TO MEASURE, AND THEN WE WENT ITEM BY ITEM TO MAKE SURE THAT THERE WAS CONSENSUS ON EACH OF THE ITEMS. WE REALLY WENT THROUGH ALL EIGHT ENGAGEMENT PRINCIPLES, LIKE I SAID, THEY ALL ARE MEASURED BY THREE TO FIVE ITEMS, TO MAKE SURE THAT THERE WAS CONSENSUS. WE HAD REALLY STRONG CONSENSUS AMONG STAKEHOLDERS BUT IF ANY OF YOU HAVE WORKED WITH RESEARCHERS OR ACADEMICS, ILTS REALLY HARD TO REACH CONSENSUS AMONG THAT GROUP SO WE THINK THESE ARE REALLY GOOD NUMBERS EVEN THOUGH THERE WERE OFTEN ONE OR TWO PEOPLE WHO DIDN’T NECESSARILY AGREE, OVERALL WE HAD OVER 80% FOR ALL OF THE ITEMS. SO ONCE WE HAD OUR MEASURE, WE THOUGHT IT WAS IMPORTANT TO TEST THIS MEASURE IN PRACTICE. WE HAD A LONGITUDINAL STUDY WHICH WAS FOUR WAVES. OUR GOAL WAS TO ENROLL 500 PEOPLE THAT HAD BEEN ENGAGED IN SOME SORT OF RESEARCH STUDY. WE WERE ABLE TO CONSENT 487. WE HAD A VARIETY OF NUMBER OF PEOPLE WHO COMPLETED EACH OF THE SURVEYS WHICH WE STARTED TO RELEASE IN JULY OF 2017, AND WE JUST CLOSED OUR SURVEYS AT THE END OF SEPTEMBER OF 2019. WE HAD 391 PEOPLE COMPLETE AT LEAST ONE SURVEY AND 324 PEOPLE COMPLETE ALL SURVEYS, AND AS YOU WOULD EXPECT, THERE’S SOME ATTRITION ACROSS THE FOUR WAVES BUT THEY’RE ALL HIGHER THAN 300 SO WE THINK WE DID A GOOD JOB. YOU’LL SEE THAT THERE’S A BIG GAP BETWEEN WHEN SURVEY THREE AND WHEN SURVEY FOUR WAS RELEASED, SO SURVEY THREE WAS RELEASED IN MARCH OF 2018 AND IT TOOK US A YEAR ALMOST BEFORE WE RELEASED SURVEY FOUR, AND THAT’S BECAUSE WE DID SOME COGNITIVE RESPONSE TESTING IN BETWEEN SURVEY THREE AND SURVEY FOUR, AND THIS IS REALLY TO ENSURE — SO ONE OF THE THINGS I MENTIONED WAS THAT OUR DELPHI PANELS WERE HIGHLY EDUCATED BUT WE WANTED TO MAKE SURE THAT OUR ITEM REALLY WORKED FOR ALL POPULATIONS AND THAT WAS AT A LEVEL THAT WAS REALLY READABLE AND UNDERSTANDABLE TO EVERYUP WITH,
EVERYONE. AND EVEN THOUGH I’M A SURVEY RESEARCH METHODOLOGIST, OUR COGNITIVE TESTING PICKED UP THINGS THAT WERE COMPLEX, BUT WE ALSO HAD SOME TERMS THAT PEOPLE JUST THOUGHT WERE REALLY COMPLEX. WE USED THE TERM DISSEMINATION AND GOVERNANCE AND INTELLECTUAL PROPERTY AND CAPACITY AND WHOSE CAPACITY AND ALL THOSE TYPES OF THINGS, SO WE WORKED ON THOSE TYPES OF COMPLEX QUESTIONS. THERE WAS SOME CONFUSION IN THE QUESTION STEM. I SHOULDN’T SAY CONFUSION. LOTS OF PEOPLE DRN THE ORIGINAL QUESTION STEM WAS WHAT ACADEMICS ARE DOING, SO IT WAS UNIDIRECTIONAL, SO NOW THE QUESTION IS ALL PARKT NERS SO
PARTNERS SO EVERYONE WHO’S PARTICIPATING IN THE STUDY DOES, AND I’M A STATISTICIAN SO I DON’T LOVE THIS BUT ONE OF THE THINGS WE DID ADD WAS A “NOT APPLICABLE” OPTION FOR ALL OF THE ITEMS, YOU MAY NOT BE THINKING OF DISSEMINATION YET OR KNOW A LOT ABOUT HOW DISSEMINATION WILL WORK. SO WE MODIFIED THE MEASURE BASED ON COGNITIVE RESPONSE TESTING AND THAT’S WHY THIS SURVEY FOUR WAS RELEASED A LITTLE BIT LATER. WE ORIGINALLY PROPOSED THAT WE WOULD JUST RECRUIT PEOPLE FROM WASHINGTON UNIVERSITY AND ST. LOUIS, AND THE UNIVERSITY OF WASHINGTON, BUT ONE OF THE THINGS THAT CAME OUT IN OUR DELPHI PROCESS IS THAT ENGAGED IS DIFFERENT BY POPULATION BUT ALSO BY REGION, AND SO WE DECIDED TO ACTUALLY OPEN THIS UP AND DO A BROADER NATIONAL STUDY SO THIS IS A MAP OF WHERE PARTICIPANTS ARE FROM, YOU SEE MOST ARE IN MISSOURI BECAUSE THAT’S WHERE WE STARTED BEFORE WE THE EXPANSION. WE ALSO HAVE ONE PARTICIPANT FROM THE VIRGIN ISLANDS, PUERTO RICO AND ALASKA. THOSE WERE NOT DISPLAYED ON THE MAP BUT THEY DID PARTICIPATE, AND THEN THERE WERE 12 STATES THAT HAD NO PARTICIPANTS. BUT WE THINK WE GOT A GOOD REPRESENTATION, PARTICULARLY FROM PLACES WHERE THERE’S LOTS OF ENGAGEMENT WORK HAPPENING. SO WE HAD PEOPLE WHO ENGAGED WITH 177 DIFFERENT UNIVERSITIES ACROSS THE COUNTRY. WE’VE LISTED HERE UNIVERSITIES THAT HAD MORE THAN 15 PEOPLE. WHEN THIS PROJECT WAS STARTED, I WAS AT WASHINGTON UNIVERSITY IN ST. LOUIS, I’M NOW AT NEW YORK UNIVERSITY SO THOSE ARE THE TOP TWO BECAUSE A LOT OF THESE ARE MY PARTNERS, BUT WE ALSO HAD GREAT RESPONSE FROM THE UNIVERSITY OF ALABAMA, NORTHWESTERN, THE MAYO CLINIC, ST. LOUIS UNIVERSITY, AND THE UNIVERSITY OF WASHINGTON, WHERE ONE OF OUR COLLABORATORS IS LOCATED. OUR PARTICIPANTS HAD A GOOD MIX BETWEEN WHITE AND BLACK BUT WE DIDN’T DO A JOB WITH HISPANICS, ASIANS AND OTHER MULTIETHNIC GROUPS. AGAIN FEMALE, ONE OF THE THINGS WE FIND WAS WOMEN WERE MORE LIKELY TO PARTICIPATE IN STUDIES BUT THEY’RE REALLY A LOT MORE LIKELY TO BE ENGAGED AS RESEARCH PARTNERS, SO EVEN THOUGH THIS IS SKEWED HEAVILY FEMALE, WE DO THINK THIS IS REPRESENTATIVE OF THE POPULATION IN WHICH WE WERE TRYING TO REACH, AND IN THIS CASE, LESS THAN HALF HAD A GRADUATE DEGREE, SO MORE VARIABILITY IN THE EDUCATION LEVEL. SO ONE OF THE FIRST THINGS WE WANTED TO DO IN TERMS OF EXAMINING THE PSYCHOMETRIC PROPERTIES WAS TO MAKE SURE THAT THE ITEMS, 3 TO 5 ITEMS THAT ASSESS ANY ONE ENGAGEMENT PRINCIPLE WERE ENTIRELY CONSISTENT WITH ONE ANOTHER, SO THEY’RE MEASURING THE SAME THING. SO WE USED A STATISTIC CALCULATOR FOR PAIR WISE CORRELATION BETWEEN ITEMS AND IT MEASURES WHETHER SEVERAL ITEMS PRODUCE CONSISTENT RESPONSES. IN MOST FIELDS, AN ALPHA OF .7 AND HIGHER IS CONSIDERED ACCEPTABLE. SO WE LOOKED AT THE INTERNAL CONSISTENCY ON BOTH SCALES, BOTH QUALITY AND QUANTITY, AND WE HAD REALLY GOOD INTERNAL CONSISTENCY, MEANING THAT THE ITEMS WE’RE MEASURING, EACH ENGAGEMENT PRINCIPLE LOOKED TO BE CONSISTENT WITH ONE ANOTHER. I HAVE EP7 STARRED BECAUSE THERE’S ONE ITEM WE PLAN TO DROP, AND THIS ITEM IS — ALL PARTNERS HAVE OPPORTUNITY TO BE CO-AUTHORS WHEN THE WORK IS PUBLISHED. ONE OF THE THINGS WE HEARD FROM OUR DELPHI PANELISTS IS THEY MAY OR MAY NOT BE INTERESTED IN BEING AUTHORS ON PUBLISHED WORK, IT REALLY DEPENDS ON WHERE THEY SIT AND WHETHER PUBLICATION IS REALLY IMPORTANT TO THEM FOR THEIR OWN WORK. THE OTHER THING WE WANTED TO LOOK AT IS CORRELATIVE VALIDITY WITH OTHER EXISTING MEASURES. BECAUSE OUR DATA, IN SOME INSTANCES, IT’S A LICK ERT SCALE SO IT’S HIGHLY SKEWED, I’M GOING TO SHOW YOU THE LOWER VALUES FIRST. SO WE LOOKED AT IT IN CORRELATION WITH THE MEDICAL MISTRUST SCALE, A SCALE OF TRUST IN MEDICAL RESEARCHERS. THERE’S A COMMUNITY ENGAGEMENT IN RESEARCH INDEX, THERE’S A MEASURE CALLED THE PARTNERSHIP ASSESSMENT AND COMMUNITY-BASED RESEARCH, AND THEN THE COALITION SELF ASSESSMENT SURVEY, WE SOME OTHER PARTNER RS I’LL SHOW YOU SOON. SO YOU’LL SEE, WE WERE ACTUALLY REALLY EXCITED TO SEE THEM, MOST OF THEM WERE NEGLIGIBLE OR LOW BECAUSE WE DO BELIEVE THAT THESE THINGS ARE TO BE CORRELATED BUT WE’RE TRYING TO CREATE A NEW MEASURE THAT WE THINK IS MEASURING SOMETHING DIFFERENT. SO WE DON’T WANT TO BE TOO CORRELATED BECAUSE IT MEANS IT’S MEASURING THE SAME THING. SO ON THE MODERATE SCALE, THERE’S A MEASURE BY KAGAN, A PARTNERSHIP SELF ASSESSMENT TOOL, SYNERGY SCALE AND ALSO A SATISFACTION SCALE AND THEN ALSO A COLLABORATION SCALE. AGAIN, WE WERE EXCITED THAT WE DIDN’T HAVE REALLY HIGH LEVELS OF CORRELATION BECAUSE THAT MEANS WE’RE MEASURING PRETTY MUCH THE SAME THING. THIS SHOWS YOU WE’RE MEASURING SOMETHING THAT IS ASSOCIATED WITH THESE THINGS BUT IS NOT NECESSARILY THE SAME SO WE THINK THIS IS A GOOD THING FOR OUR ENGAGEMENT MEASURE. WE’RE DOING SOME EXPLORATORY FACTOR ANALYSIS, REALLY LOOKING TO SEE HOW OUR MEASURE IS SORT OF LAYING OUT IN PRACTICE, SO WE HAVE SOME OBSERVED VARIABLES WHICH ARE THE ONES THAT WE MEASURED, AND THEY’RE ASSOCIATED WITH LATENT VARIABLES THAT ARE NOT DIRECTLY MEASURED. IN OUR CASE, WE HAVE SEVERAL MEASURED VARIABLES, THOSE ARE THE ITEMS THAT WE’RE ASKING PEOPLE, AND WE HYPOTHESIZE THAT THEY MAKE UP THESE LATENT VARIABLES WHICH WE’RE CALLING THE ENGAGEMENT PRINCIPLES. AND ALL OF THIS FALLS UNDER THE LARGER UMBRELLA OF COMMUNITY ENGAGEMENT IN RESEARCH. WE USE HOW MUCH VARIANCE IS CAPTURED BY A COMPONENT OR FACTOR. YOU TYPICALLY RETAIN FACTORS WITH VALUES GREATER THAN 1. BECAUSE OUR SCALE IS NOT CONTINUOUS, IT’S REALLY ORDINAL, WE USE POLYCORE COMMONLY USED WITH VARIABLES. SO ONE OF THE THINGS WE NOTICED IS THE SCALES IN TERMS OF FACTOR ANALYSIS DON’T LINE UP THE SAME WAY AND I THINK THIS BODES WELL FOR MEASURING BOTH QUANTITY AND QUALITY. WE’RE STILL DIGGING THROUGH THIS TO SEE IF WE CAN GET A SENSE OF WHAT THIS MEANS BUT IN TERMS OF FACTOR ANALYSIS, THE QUALITY SCALE REALLY LOADS ON TWO FACTORS AND WE’RE WORKING THROUGH TO SEE IF WE CAN GET AN UNDERSTANDING OF WHAT IS ON WHAT FACTOR. ONE OF THE THINGS WE WERE REALLY SORT OF INTERESTED IN IS ARE ALL THE ITEMS FROM ONE ENGAGEMENT PRINCIPLE ALL ALIGNED WITH ONE FACTOR OR THE OTHER. THAT’S NOT THE CASE, WE’RE SEEING ITEMS FROM MANY OF THE PRINCIPLES ALIGN ON ONE FACTOR OR BOTH FACTORS ACTUALLY. THERE’S SOME ITEMS THAT ACTUALLY LOADED ON BOTH ITEMS SO WE’RE LOOKING AT WHICH ONE THEY LOADED HEAVIER ON, ONE ABOUT PARTNERS HAVING THE OPPORTUNITY TO SHARE IDEAS, INPUT AND LEADERSHIP RESPONSIBILITIES AND TO SHARE IN THE DETERMINATION OF THE PROJECT STRUCTURE AND THE LIKE. SO REALLY TRYING TO UNDERSTAND DO THESE ITEMS BELONG ON ONE FACTOR OR THE OTHER. SIMILARLY FOR QUANTITY, INSTEAD OF FALLING ON TWO FACTORS, IT ACTUALLY FALLS ON FOUR, AND WE HAVE SOME AT LEAST INCLINATION HERE WHERE THESE FALL. THERE’S A SET OF ITEMS THAT REALLY — THE LENGTH OF ITEMS, SET OF ITEMS LOOKING AT TRUST, AT VALUE, AND EVERYTHING ELSE IS GROUPED INTO THAT FOURTH FACTOR. BUT AGAIN, WE WERE TRYING TO SEE WHICH ENGAGEMENT PRINCIPLES WERE ALIGNED WITH WHICH FACTORS AND IF THEY WERE ON MULTIPLE FACTORS LIKE YOU CAN SEE ENGAGEMENT PRINCIPLE 1 IS SOME ITEMS ARE ON FACTOR 1 BUT SOME ITEMS ARE ON FACTOR 4, SO WE’RE TRYING TO WORK THROUGH THIS IN TERMS OF THE WAY OUR SCALE IS PERFORMING IN PRACTICE. AND HERE WE’RE LOOKING AT ITEMS THAT CROSS LOADED ON BOTH SCALES. ONE IS JUST AN OPPORTUNITY TO DROP ITEMS, ARE THEY NOT REALLY PARTICULARLY THERE WERE TWO ITEMS THAT DIDN’T LOAD ON ANY OF THE SCALES SO WE THINK THIS MAY BE A PLACE THAT WE CAN MAYBE CUT SOME ADDITIONAL ITEMS. SO THE FIRST FIGURE I SHOWED YOU SORT OF HAD A STACKED WAY OF THINKING ABOUT ENGAGEMENT LEVELS, IN TERMS OF BINS OF NON-ENGAGEMENTS, SYMBOLIC ENGAGEMENTS, AND SORT OF FULL ENGAGEMENT, AND WE WERE REALLY ORIGINALLY THINKING ABOUT ENGAGEMENT AS A CONTINUUM, SO YOU COULD BE ANYWHERE ON THIS CONTINUUM, AND NOT THAT BEING ON ONE END OR THE OTHER IS MORE VALUABLE, BUT JUST RILEY THINKING ABOUT WHERE YOUR PROJECT FALLS, THAT’S LIKELY TO DRIVE THE SCORES THAT YOU’RE GETTING FOR ENGAGEMENT. BUT IN PRACTICE, SO ONE OF THE THINGS WE ASKED THE PEOPLE IN THE SURVEY TO TALK ABOUT A SPECIFIC PROJECT AND WE REMINDED THEM EACH WAVE WHICH PROJECT WE WERE TALKING ABOUT, THEN WE GAVE THEM THE DEAF ANYTHINGS WE HAVE BEEN USING FOR THE CLASSIFICATION LEVELS AND ASKED THEM TO CLASSIFY WHERE THEY FELT THEIR PROJECT FELL ON THIS ENGAGEMENT SCALE. OUTREACH AND EDUCATION, WHICH WE HAD BEEN CONSIDERING AT ONE END OF THE SPECTRUM ACTUALLY HAD HIGHER MEAN SCORES THAN CONSULTATION AND IN SOME CASES COOPERATION. SO IT WASN’T PERFORMING EXACTLY THE WAY WE THOUGHT ABOUT IT. THEN WE STARTED TO THINK HOW WE REALLY WERE LAYING OUT THE CONTINUUM, THIS IS WHERE YOU’LL SEE THAT I’M A BIOSTATISTICS BECAUSE NOW WE HAVE TWO AXES INSTEAD OF ONE BECAUSE I THINK IN TERMS OF ENGAGEMENT, IT’S IMPORTANT TO THINK ABOUT IT IN TWO ASPECTS. ONE IS THE ACADEMIC RESEARCHER’S OUTPUT TO NON-ACADEMIC STAKEHOLDERS AND THE OTHER IS NON-ACADEMIC INPUT INTO ACADEMIC RESEARCH. WE THINK OUTREACH AND ENGAGEMENT IT’S NOT HIGH ON NON-ACADEMIC- INPUT INTO RESEARCH, IT IS REALLY HIGH IN WHAT COMMUNITIES CAN BENEFIT FROM ACADEMICS GOING OUT AND PUTTING THEIR WORK OUT INTO THE COMMUNITIES THEY’RE TRYING TO SERVE. SO NOW WE’RE THINKING ABOUT THIS ON SORT OF A TWO AXIS LEVEL AND YOU’LL SEE THAT THEY’RE NOT EXACTLY IN LINE BUT THEY’RE SORT OF LINED UP IN A GREAT WAY, SO OUTREACH AND EDUCATION IS SLIGHTLY HIGHER ON WHAT WOULD THE Y AXIS AND LOWER ON THE X AXIS, WHEREAS CONSULTATION IS HIGHER ON THE X AXIS BUT LOWER ON THE Y AXIS AND THEN SORT OF BOX THAT COLLABORATION, COOPERATION AND PARTNERSHIP. COLLABORATION IS EXACTLY WHAT YOU THINK IT IS, IS FULL, STRONG COLLABORATION AND PARTNERSHIP IS SUSTAINED COLLABORATION, SO FOR MORE THAN ONE PROJECT, SO BASICALLY COLLABORATION HAPPENING OVER TIME. SO WE ARE NOW HEADED INTO PHASE III OF OUR STUDY, AND THIS IS SOME OF THE STUFF THAT I AM AS A STATISTICIAN REALLY EXCITED ABOUT. SO WE ORIGINALLY THOUGHT THAT WE WOULD BE ABLE TO REDUCE THIS MEASURE IN THE DELPHI PROCESS, AND WE DID A BIT BUT NOT AS MUCH, AND WE THOUGHT THROUGH SOME OF THE PSYCHOMETRIC ANALYSIS, BUT IT LOOKS LIKE AN ITEM HERE AND THERE. SO ONE OF THE THINGS THAT WE ARE WORKING ON NOW IS TO DEVELOP A COMPUTER ADAPTIVE ALGORITHM SO THIS WILL ALLOW US TO REDUCE PARTICIPANT BURDEN ON THE FLY. AND THINK ABOUT IT LIKE TAKING THE S.A.T. OR THE G.R.E., YOUR NEXT QUESTION IS BASED ON WHETHER YOU GET THE FIRST QUESTION RIGHT OR WRONG. IN THIS INSTANCE, WE DON’T HAVE RIGHT OR WRONG, BUT WE WANT TO KNOW, DOES YOUR RESPONSES TO SOME QUESTIONS PREDICT HOW YOU WILL RESPOND TO OTHERS, MEANING THAT I DON’T NEED TO NECESSARILY ASK YOU THOSE QUESTIONS BASED ON WHAT — ON THE RESPONSES THAT YOU’VE GIVEN ME ALREADY. SO COMPUTER ADAPTIVE ALGORITHM IS A METHOD THAT MERGES COMPUTER TECHNOLOGY WITH MODERN MEASUREMENT THEORY TO INCREASE EFFICIENCY AND REDUCE PARTICIPANT BURDEN. OUR MEASURES SO FAR HAVE BEEN GIVEN USING A WEB BASED SURVEY TOOL SO THIS ALLOWS US TO REALLY BUILD THIS IN TO THE SURVEY INSTRUMENT, AND WE’RE WORKING ON CREATING AN ITERATIVE ALGORITHM USING OUR FULL MEASURE OF ITEMS AS AN ITEM BANK AND THEN TAILORING IT BASED ON PEOPLE’S INDIVIDUAL RESPONSES. SO ALL ITEMS THAT HAVE NOT YET BEEN ADMINISTERED OR EVALUATED TO DETERMINE WHICH ONE WOULD BE THE NEXT BEST ONE TO GIVE AND THAT ITEM IS SORT OF GIVEN IN THE PARTICIPANT RESPONSE, A NEW ESTIMATE FOR THE LEVEL OF ENGAGEMENT IS MEASURED, THEN WE COMPUTE BASED ON THAT AND DETERMINE WHAT THE NEXT ITEM IS TO GIVE AND WE SORT OF REPEAT THIS PROCESS UNTIL WE DON’T HAVE TO GIVE THE PARTICIPANT ANY ADDITIONAL ITEMS. USUALLY ITEM RESPONSE THEORY IS USED IN THIS TYPE OF WORK BUT THAT’S USUALLY WHEN YOU HAVE CASES WHERE THERE’S A RIGHT ANSWER. IN TERMS OF ENGAGEMENT, THERE IS NO RIGHT ANSWER. SO AS A STATISTICIAN, WE’RE THINKING ABOUT FRAMING THIS AS YOU WOULD FRAME A MISSING DATA PROBLEM. THE GREAT THING ABOUT THIS MISSING DATA PROBLEM IS I HAVE AURAL THE DATA. SO ALL THE DATA SO I CAN RANDOMLY DROP THE OBSERVATIONS AND COMPUTE BACK TO GET THE RIGHT ANSWER SO I’M SORT OF CHEATING A LITTLE BIT, I KNOW WHAT THE REAL ANSWER IS, I CAN DROP THEM OUT AND ESTIMATE WHETHER I CANPREDICT THEM AND THAT WILL ALLOW US TO HAVE AN ALGORITHM, PEOPLE ARE HIGHLY ENGAGED BUT OFTEN ASKED TO DO THINGS OVER AND OVER SO WE DON’T WANT TO BURDEN THEM BY TAKING ON LOTS OF NEW ITEMS, AND WE ALSO WANT INVESTIGATORS TO PUT THIS ON THEIR EXISTING SURVEYS AND THEY’RE NOT GOING TO DO THAT IF IT’S TOO LONG. SO WE’RE REALLY SORT OF EXCITED ABOUT THAT. AND THE KEY QUESTION THAT WE’RE TRYING TO GET AT IS, IS THERE A SUBSET OF ITEMS THAT PRODUCE THE SAME MEAN SCORE AS ALL OF THE ITEMS WOULD, AND WE ARE WORKING TO DEVELOP THIS ALGORITHM AND WE’RE PILOT TESTING IT ON OUR COMMUNITY ADVISORY BOARD FOR THIS PROJECT. SO WE KNOW HOW ENGAGED THEY HAVE BEEN, AND SO WE’RE ALSO ABLE TO LOOK AT IT IN OUR PROJECT AND THEN WE’LL BE IMPLEMENTING IF IN OTHER PC OSM R AND COR AND
EFFECTIVENESS STUDIES. SO WE’RE CURRENTLY RECRUITING AND ROLL OUT IMPLEMENTATION PHASE AT THE BEGINNING OF NEXT YEAR. THE IDEA IS TO EVALUATE THE STAKEHOLDER ENGAGEMENT MEASURE IN PRACTICE, AND WE’RE WORKING WITH A LOT OF PATIENT-CENTERED OUTCOMES RESEARCH AND COMPARATIVE EFFECTIVENESS STUDIES, PARTICULARLY OTHER STUDIES THAT HAVE BEEN FUNDED BY PCORI. WE’RE GOING TO DO WEB BASED SURVEYS OF PROJECT TEAM MEMBERS FIRST TO REALLY GET THEIR SENSE OF WHAT ARE THE BARRIERS AND FACILITATOR FACTORS FOR YOU MEASURING PARTNER ENGAGEMENTS IN YOUR STUDIES. SO FAR WE’VE HAD 34 PROJECT TEAMS COMPLETE THE BASELINE SURVEYS AND 14 HAVE AGREED TO IMPLEMENT OUR MEASURE IN THEIR STUDIES. SO REALLY FROM THEM, WE’RE TRYING TO FIGURE OUT HOW DOES THIS WORK IN PRACTICE, DOES IT REALLY ADD A LOT OF TIME TO YOUR EXISTING MEASURES, CAN YOU TACK IT ON TO A SURVEY THAT YOU WERE ALREADY GIVEN, WHAT ARE THOSE IMPLEMENTATION CHALLENGES TO ADDING AN ASSESSMENT TO YOUR EXISTING SURVEY TOOLS, DOES IT MAKE YOUR SURVEY WAY TOO LONG. AND IN BETWEEN OUR BASELINE AND FOLLOW UP, WE’LL BE DOING SOME PHONE INTERVIEWS WITH PROJECT TEAMS THAT HAVE DECIDED TO COLLABORATE WITH US. WE’RE USING THE CONSOLIDATED FRAMEWORK FOR IMPLEMENTATION RESEARCH AS A FRAME FOR OUR IMPLEMENTATION STUDY WHICH ALLOWS US TO LOOK AT INTERVENTION CHARACTERISTICS, THE OUTER SETS, WHAT ARE THE EXTERNAL FACTOR, THE INNER SETTINGS, WHAT ARE THE INTERNAL PROJECT TEAM FACTORS, COLORADO SPRINGS CHARACTERISTICS OF INDIVIDUALS INVOLVED IN THE PROCESS OF IMPLEMENTING OUR MEASURE IN THEIR STUDY, AND WE THINK THAT THIS WILL HELP US REFINE THE TOOL, PARTICULARLY IN TERMS OF HOW IT SHOULD BE ADMINISTERED FOR GUIDELINES, PROCEDURES FOR IMPLEMENTING THE TOOL IN PRACTICE, AND REALLY ALLOW US TO PACKAGE AND DISSEMINATE THE TOOL IN A WAY THAT IS USEFUL FOR RESEARCH TEAMS. SO PLEASE DON’T BLAME PCORI FOR ANYTHING THAT I SAID, BECAUSE THEY’RE NOT RESPONSIBLE FOR IT, BUT I HOPE YOU DID ENJOY. I DO HAVE TO REALLY THANK MY COLLEAGUES WHO HAVE BEEN INVOLVED IN THIS STUDY BOTH FROM THE SYSTEMATIC REVIEW PROCESS AND THOSE WHO ARE CURRENTLY INVOLVED IN OUR PCORI-FUNDED STUDY, AND ARE IMMENSELY GRATEFUL TO THE MEMBERS OF OUR DELPHI PANEL WHO WERE HIGHLY ENGAGED AND PARTICIPATED AT REALLY HIGH LEVELS THROUGHOUT THE DELPHI PROCESS. I SHOULD MENTION THAT WE STARTED WITH 19 DELPHI MEMBERS AND WE HAD 18 COMPLETE ALL FIVE ROUNDS OF THE PROCESS WHICH IS REALLY GOOD CONSIDERING ACADEMICS AND RESEARCHERS AND COMMUNITY PEOPLE AND HEALTH CENTERS AND DIRECTORS WHO WERE ALL SUPER, SUPER BUSY PEOPLE. AND I WILL TAKE QUESTIONS. [APPLAUSE]>>WE’LL ASK PEOPLE TO COME TO THE MICROPHONES IN THE AISLE TO ASK QUESTIONS AND I’LL START OFF WITH A QUESTION. THIS IS REALLY FASCINATING, AND CLEARLY YOU’RE STILL REFINING IT. BUT WHAT IS YOUR THOUGHT, YOUR VISION AS TO HOW THIS TOOL MIGHT BE USEFUL FOR YOUR STANDARD NIH CLINICAL TRIAL WHERE YOU’RE TRYING TO ASSESS THE LEVEL OF INCLUSION AND RETENTION TECHNIQUES THAT ARE BEING DEVELOPED?>>SO WE REALLY THINK THIS IS AN IMPORTANT STEP TO TAKE BACK, BUT WE REALLY HOPE THAT PEOPLE — TO DEVELOP THE MEASURE, BUT WE REALLY HOPE PEOPLE USE IT FOR THINGS LIKE DID ENGAGING STAKEHOLDERS HELP ME RECRUIT FASTER, DID IT HELP ME RECRUIT A MORE DIVERSE SAMPLE OF PARTICIPANTS, DID IT MAKE ME REFINE MY RESEARCH QUESTION IN MEANINGFUL WAYS, DID IT HELP ME IMPLEMENT MY WORK IN PRACTICE IN A PRACTICE SETTING IN A BETTER CASE? SO WE REALLY HOPE THAT PEOPLE WILL BE ABLE TO USE IT AND REALLY LOOK AT IT IN TERMS OF HOW IT REALLY IMPACTS SOME OF THE KEY COMPONENTS OF RESEARCH THAT ARE NOT NECESSARILY THE EFFECTIVENESS OF MY INTERVENTION, WHICH PEOPLE ALREADY SORT OF MEASURE, BUT DOES IT REALLY HELP ME IN IMPLEMENTING THE RESEARCH THAT I’M WORKING ON.>>I REALLY ENJOYED YOUR PRESENTATION. THANK YOU VERY MUCH, DR. GOODMAN. YOU TALKED A LITTLE BIT ABOUT THE ENHANCING ENGAGEMENT OF WOMEN EARLY IN YOUR PRESENTATION. CAN YOU TALK A LITTLE BIT MORE ABOUT WHAT YOU SEE BETWEEN DIFFERENCES BETWEEN MEN AND WOMEN GENERALLY AND WHAT THE CONSEQUENCES OF THOSE DIFFERENCES MIGHT BE?>>SO THERE’S ALREADY SOME RESEARCH THAT SHOWS THAT WOMEN ARE MORE LIKELY TO PARTICIPATE IN RESEARCH STUDIES IN GENERAL. I DON’T THINK ANYONE HAD REALLY LOOKED AT IT IN TERMS OF ENGAGEMENT. BUT WHEN WE STARTED TO SEE HIGH NUMBER OF WOMEN IN OUR SAMPLE, WE WERE A LITTLE BIT CONCERNED THAT OUR DATA WOULD BE SKEWED, SO WE ACTUALLY — SO THE WAY WE RECRUITED PEOPLE WAS WE WENT THROUGH PRINCIPLE INVESTIGATORS AND WE SAID HI, WE REALLY LOVE YOUR STUDY BECAUSE COMPLIMENTS ALWAYS HELP, AND WE’RE TRYING TO REACH OUT TO PEOPLE WHO MAY HAVE BEEN ENGAGED IN YOUR PROCESS, CAN YOU FORWARD THIS EMAIL TO YOUR PARTICIPANTS. SO THEN WE WENT BACK TO THOSE P.I.s AND WE SAID, CAN YOU GIVE US A SENSE OF WHAT THE MAKEUP WAS OF YOUR PARTNERS? AND THAT’S WHEN WE REALLY STARTED TO FIGURE OUT THAT THERE WERE A LOT MORE WOMEN IN THIS ENGAGED PROCESS THAN THERE WERE MEN. SO IN ONE WAY, I THINK THAT HELPED VALIDATE WHAT WE WERE SEEING, THAT THE POPULATION THAT WE WERE TRYING TO REACH, OUR SAMPLE WAS REPRESENTATIVE OF THAT POPULATION. IN ANOTHER WAY, IT’S REALLY IMPORTANT TO ENGAGE BOTH SECTIONS IN SEXES IN THE RESEARCH PROCESS, PARTICULARLY AS RESEARCH PARTNERS, AND SO IN SOME WAYS, IT’S A LITTLE BIT DISHEARTENING. YES, IT’S NICE THAT WE’RE GETTING A GOOD GOOD SAMPLE OF
THE POPULATION, BUT IS THIS EVEN THE RIGHT POPULATION TO START WITH IS ONE OF OUR QUESTIONS, AND SO ONE OF THE THINGS WE THINK IS REALLY IMPORTANT FOR THOSE WHO DO ENGAGEMENT WORK IS TO REALLY ENGAGE MEN MORE AS PARTNERS IN THE RESEARCH PROCESS. AND THE INTERESTING THING WAS, WE CONTACTED A P.I. WHO DID A LOT OF WORK IN PROSTATE CANCER AD HE STILL HAD MORE WOMEN PARTNERS AND THEY WERE LIKE REPRESENTING THEIR HUSBANDS OR THEIR — YOU KNOW, SO IT’S ALL REALLY INTERESTING BUT I THINK IT’S IMPORTANT, YOU KNOW, ENGAGEMENT IS SORT OF MY SOAP BOX, IT’S REALLY IMPORTANT TO ENGAGE THE PEOPLE THAT ARE MOST IMPACTED BY OUR WORK. AND SO IT IS A LITTLE BIT DISHEARTENING THAT IT’S MAINLY WOMEN, AND I REALLY HOPE THAT AS WE RELEASE SOME OF THIS DATA, THAT PEOPLE REALLY START TO CONSIDER WHO THEY’RE ENGAGING. YES, SOMETIMES IT’S NICE TO HAVE A SPOUSE IF ONE SPOUSE IS ACTUALLY HAVING AN ILLNESS, IT’S SOMETIMES NICE TO HAVE THE OTHER SPOUSE BEING ENGAGED, BUT IT’S REALLY A LOT MORE MEANINGFUL TO HAVE THE PERSON IMPACTED TO BE THE PERSON THAT IS REALLY ENGAGED IN DEVELOPING OUR STUDY. SO I’M GIVING THIS TALK AT NIH. THOSE OF YOU THAT ENGAGE PARTICIPANTS IN YOUR WORK, I THINK YOU SHOULD BE REALLY THOUGHTFUL AND CONSCIOUS OF WHO YOUR ACADEMIC PARTNERS ARE, OR YOUR NON-ACADEMIC PARTNERS ARE, AND PART OF WHAT PEOPLE WERE REALLY HONEST WITH US ABOUT IS THAT WHOEVER COMES IS WHO THEY TAKE. SO IF THE WOMEN SHOW UP, THAT’S WHO THEY’RE GOING TO USE AS PARTNERS, BUT I DO THINK WE HAVE TO BE MORE MEANINGFUL, SO IN MY OWN STUDIES NOW, I’M REALLY MORE MEANINGFUL ABOUT LOOKING AT THE DEMOGRAPHICS OF MY PARTNERS, AND REALLY SAYING THERE’S SOME GAPS, LET’S TRY TO GO — SO I’M WORKING ON A STUDY WITH PEOPLE STUDYING THE — GENE, WE FORMED A BOARD AND IT WAS A LOT WOMEN AND I SAID THEY ALL CHECK OFF A BUNCH OF BOXES, NOW LET’S FIND MEN WHO CHECK OFF THESE SAME BOXES. BECAUSE THINK IT’S IMPORTANT TO HAVE BOTH VOICES HEARD AND WE’RE NOT DOING A GOOD JOB OF THAT RIGHT NOW.>>HI. SO THAT BRINGS UP ANOTHER QUESTION. DO YOU THINK THAT YOUR WORK CAN HELP IN TARGETING MESSAGES TO PROMOTE ENGAGEMENT FROM DISENGAGED POPULATIONS?>>I DON’T KNOW IF IT COULD HELP IN DEVELOPING MESSAGES, BUT I DO THINK IT CAN — THE REASON WE’RE LOOKING AT THESE EIGHT ENGAGEMENT PRINCIPLES, HAVING A CERTAIN SCORE TELLS YOU WHERE YOU ARE, CONSULTATION, COOPERATION, BUT KNOWING, LIKE, I’M DOING REALLY WELL ON THIS ENGAGEMENT PRINCIPLE BUT NOT SO GOOD OVER HEERYLY TELLS YOU
HERE REALLY T
ELLS YOU HOW TO STARGT YOUR TARGET YOUR
ENGAGEMENT STRATEGIES. I THINK THAT REALLY TAKES MORE THOUGHTFULNESS AND MORE MINDFULNESS ABOUT WHO WE’RE ENGAGING AND NOT USING PEOPLE JUST BECAUSE THEY’RE THERE AND THEY’RE PRESENT AND WILLING TO COME BUT REALLY TAKING MORE EFFORT TO ENGAGE THOSE WHO NEED TO BE ENGAGED, AND YES, THAT MAY REQUIRE A LITTLE BIT MORE WORK AND A LITTLE BIT MORE TIME, BUT I THINK THE SCIENCE WILL GREATLY BENEFIT FROM THAT.>>THANK YOU.>>I HAVE ANOTHER QUESTION. AS YOU KNOW, WE HAVE THIS NEW POLICY AT NIH ON INCLUSION ACROSS LIFESPAN, AND I NOTICE THAT IT APPEARED FOR YOUR DELPHI PROCESS THAT THE AVERAGE AGE OF PEOPLE IS ABOUT 40-SOMETHING. HOW DO YOU THINK YOUR DATA MIGHT BE INFLUENCED BY HAVING A LARGE NUMBER OF PEOPLE OVER THE AGE OF 65?>>SO WE ACTUALLY HAD A LARGE — OUR AVERAGE AGE — WE HAD A LARGE RANGE IN TERMS OF AGE, WHAT WE MISSED WAS YOUNGER FOLKS. THAT’S PROBABLY BECAUSE THEY HAVEN’T REALLY BEEN ENGAGED IN RESEARCH PROJECTS THAT MUCH. USUALLY CHILDREN ASSENT BUT THEY DON’T CONSENT SO THEY’RE OFTEN NOT ON — SO WHEN I TALK ABOUT RESEARCH PARTNERS, I’M NOT TALKING ABOUT RESEARCH PARTICIPANTS, I’M REALLY TALKING ABOUT PARTNERS THAT ARE IN THE DRIVING FORCE OF DEVELOPING THE RESEARCH STUDY, DESIGNING IT AND THE LIKE, SO WHERE I REALLY THINK WE NEED TO DO MORE WORK IS AT THE YOUNGER END OF THE LIFE COURSE. WE’RE TRYING TO REFINE THIS MEASURE AS IT IS NOW AND WE’RE REALLY TRYING TO DO A GOOD JOB OF DEVELOPING A GENERIC MEASURE SO IT HAS NO DISEASE FOCUS, IT HAS NO POPULATION FOCUS, BUT I DO THINK THE NEXT STEP, IF WE GET THIS RIGHT, IS TO REALLY START TO THINK ABOUT HOW TO TAILOR THIS TO CERTAIN GROUPS, AND THAT INCLUDES BOTH AGE AND RACE, ETHNICITY AND GEOGRAPHIC REGION AND THE LIKE.>>HI. THANK YOU FOR YOUR TALK, DR. GOODMAN. I WAS THINKING ABOUT THE EXPERIENCE OF ACTUALLY PARTICIPATING IN A SURVEY, AND WHETHER OR NOT YOU ALWAYS ADMINISTER IT ONE WAY OR IS THERE ANY VARIETY? SO IS IT ELECTRONIC, IS IT EVER GIVEN AS SOMEONE SPEAKING TO ANOTHER PERSON AND COLLECTING THE RESULTS, OR JUST ON PAPER?>>SO WE HAVEN’T DONE PAPER. WE’VE DONE ELECTRONIC, BOTH SELF ADMINISTERED AND WHERE RESEARCHERS ARE USING AN iPAD AND ASKING THE QUESTIONS, SO I’M REALLY COGNIZANT OF LITERACY AND HEALTH LITERACY OVERALL. ONE OF THE THINGS THAT I DIDN’T MENTION, THERE IS THIS IDEA THAT BY ASKING YOU A FEW QUESTIONS ABOUT ENGAGEMENT AND WILL INFLUENCE MAYBE HOW YOU RESPOND TO OTHER QUESTIONS, AND SO WE HAVE CHANGED THE ORDER OF THE WAY WE ASK QUESTIONS ON OUR SURVEY TO MAKE SURE THAT’S NOT SORT OF — LIKE BY ASKING THESE INITIAL QUESTIONS UP FRONT, IT CHANGES THE WAY PEOPLE RESPOND TO QUESTIONS AT THE END, SO WHEN WE DID OUR COGNITIVE RESPONSE TESTING, WE HAD FOUR DIFFERENT QUESTIONS OF THE SURVEY SO QUESTIONS WERE ASKED IN A DIFFERENT ORDER, SO IN OUR IMPLEMENTATION STUDY, PEOPLE WILL RANDOMLY GET QUESTIONS IN DIFFERENT ORDER, SO WE THINK THE ORDERING WILL MATTER. RIGHT NOW IT’S BEING WEB-ADMINISTERED JUST BECAUSE I’M TRYING TO CONVINCE OTHER PEOPLE TO USE IT, OTHER RESEARCHERS AND I DON’T WANT TO BURDEN OTHER RESEARCHERS MORE THAN WHAT THEY HAVE TO, AND ONE OF THE THINGS WE ASK PEOPLE IS, HOW ARE YOU CURRENTLY GIVING SURVEYS, AND MANY OF THEM ARE ALREADY DOING WEB SURVEYS, SO THIS ALLOWS US TO TACK IT ON TO AN EXISTING SURVEY. PEOPLE HAVE TAKEN IT BY PAPER BEFORE, WE’RE NOT AGAINST IT, BUT ONCE IT’S ENTERED BY PAPER, THAT MAKES IT HARDER FOR ME TO CONVINCE A RESEARCH TEAM TO DO THIS ON TOP OF ALL THE STUFF THEY’RE ALREADY DOING. BUT IF I SAY IT’S A WEB SURVEY, YOU DON’T HAVE TO DO ANYTHING, YOU JUST ADD IT ON TO YOUR EXISTING SCALE, IT’S JUST BEEN A BETTER SELL TO GET PEOPLE TO DECIDE TO USE IT.>>I HAD A BRIEF QUESTION. HAS YOUR INSTRUMENT EVER BEEN ADMINISTERED NOT IN ENGLISH?>>NO. SO THERE’S A POSTDOC WHO IS CURRENTLY TRANSLATING IT INTO SPANISH, BECAUSE SHE WANTS TO USE IT AND WE’RE EXCITED. ONE OF THE THINGS WE REALLY FEEL IS WE NEED TO GET IT RIGHT IN ENGLISH FIRST AND WE’RE NOT THERE YET, BUT WE DO THINK THE NEXT STEP IS TRANSLATING INTO OTHER LANGUAGES, AND WE EVEN HAVE SOMEONE WHO WORKS WITH THE DEAF POPULATION THAT’S REALLY INTERESTED IN USING OUR MEASURE. SO WE HOPE THAT IT GOES BEYOND ENGLISH BUT WE THINK WE NEED TO SORT OF REFINE IT IN ENGLISH AND THEN WORK ON TRANSLATING INTO OTHER LANGUAGES.>>SO FOLLOW-UP QUESTION. DO YOU FEEL LIKE THERE WOULD BE ANY DIFFERENCES, CULTURALLY OR OTHERWISE THAT WOULD COME UP IN KIND OF GOING THROUGH THIS PROCESS IN ANOTHER LANGUAGE AND NOT JUST DOING AS WE ALWAYS DO, CREATE EVERYTHING IN ENGLISH, TRANSLATE TO ANOTHER LANGUAGE BECAUSE OF COURSE IT WILL WORK.>>SO WE THOUGHT ABOUT THIS A LOT. THIS IS THE NICE THING ABOUT HAVING THE EDITOR IN THE ROOM AND REALLY FOCUSING ON PLAIN LANGUAGE. I DO HAVE SOME CONCERNS THAT SOMETIMES SOMETHING IS AT A READABLE SCALE IN ENGLISH AND THEN WE DIRECTLY TRANSLATE INTO ANOTHER LANGUAGE AND IT MOVES FROM, LIKE, SIXTH GRADE LEVEL TO NINTH GRADE LEVEL JUST THROUGH THE TRANSLATION PROCESS. I DON’T KNOW THE ANSWER TO THAT YET AND I WON’T KNOW UNTIL WE START TESTING IN OTHER LANGUAGES, BUT WE’RE REALLY COGNIZANT OF MAKING SURE THAT OUR MEASURE IS AT A REALLY READABLE LEVEL. THEY’RE ALL SORT SHORT ITEMS THAT REALLY NOW USE PLAIN LANGUAGE, SO THE HOPE IS THAT IT WILL TRANSLATE WELL, BUT I CAN’T ANSWER YOUR QUESTION UNTIL WE ACTUALLY TRANSLATE IT. BUT WE ARE CONSCIOUS OF THAT, PARTICULARLY IN ENGLISH, WE’RE CONSCIOUS OF THE FACT THAT WE HAVE BEEN USING AN EDUCATED POPULATION THAT WE REALLY NEED TO MAKE SURE THAT IT WORKS FOR A VARIETY OF POPULATION. PARTICULARLY TRYING TO GET PEOPLE WITH VARIED LEVELS OF EDUCATION TO LOOK THROUGH THE MEASURE TO SEE HOW WELL IT PERFORMED IN ALL GROUPS.>>BEFORE I ASK YOU TO AGAIN THANK DR. GOODMAN FOR A REALLY MARVELOUS PRESENTATION, I WANT TO REMIND YOU THAT FAES HAS VERY KINDLY PROVIDED A RECEPTION RIGHT OUTSIDE OF THESE DOORS, AND PLEASE DO JOIN ME IN THANKING DR. GOODMAN.
00:46:47.071,00:00:00.000
>>THANK YOU VERY MUCH.

Michael Martin

Leave a Reply

Your email address will not be published. Required fields are marked *

Post comment