Getting to the core of customer satisfaction
in skilled nursing and assisted living.
Satisfaction Questionnaire
&
User’s Manual
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
Questionnaire Development
Nick Castle, Ph.D., from the University of Pittsburgh led an effort with input from an
AHCA/NCAL workgroup to develop a short, reliable and valid questionnaire that could be added
to existing survey instruments or used by itself to collect information to calculate a set of
quality measure(s) related to individuals discharged following short-term rehabilitation
services, for long-stay resident and their family members and for residents and their families in
assisted living. The SNCC related measures were endorsed by the National Quality Forum (NQF)
in 2016. The AL measures is still undergoing some final testing before submitting an application
to NQF.
The draft questionnaire was administered to over 25,000 residents in SNCCs (both short-term
and long- stay residents) and ALCs across several organizations and states with approximately
10,000 responses. Dr. Castle, and Matt O’Connor, Ph.D., HCR ManorCare, conducted extensive
psychometric testing. The results identified three core questions that were reliable and valid
across SNCCs and ALCs for both short-stay and long stay residents. There was one extra
question about the discharge process for the short-stay discharges and one extra question for
assisted living (see appendix A for the final validated questionnaires). These questions yielded
the same information, when aggregated together and summed as a quality measure at the
center/community level, as using additional questions. In other words, additional questions did
not change a center/community’s score or ranking. Additional questions did help identify areas
for improvement.
Several satisfaction vendors have been involved with this work; keeping them informed of the
process at each step. Many of the vendors have added the questions to their existing
instruments for both skilled nursing and assisted living. To date, Rhode Island has incorporated
the long-stay CoreQ questionnaire into its state-wide satisfaction data collection for their public
reporting program.
Satisfaction Quality Measures
There are five center-level satisfaction measures (NQF submission pending) that were
developed to assess the customer satisfaction/experience in SNCCs and ALCs. The measures will
be based on the CoreQ questionnaires (see appendix A). The validity of these measures are
described in Appendix B.
Measure #1: Short-Stay Discharges Satisfaction defined as the percentage of individuals
discharged from short-term rehab who were satisfied (defined as an
average rating of the center as either a 3, 4 or 5 on the CoreQ Short-Stay
Discharges questionnaire).
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
Measure #2: Long-Stay Resident Satisfaction defined as the percentage of individuals
who are long-stay residents who were satisfied (defined as an average
rating of the center as either a 3, 4 or 5 on the CoreQ Long-Stay Resident
questionnaire).
Measure #3: Long-Stay Family Satisfaction defined as the percentage of families or
designated responsible party for long-stay residents who were satisfied
(defined as an average rating of the center as either a 3, 4 or 5 on the CoreQ
Long-Stay Family questionnaire).
Measure #4: Assisted Living Resident Satisfaction defined as the percentage of AL
residents or designated responsible party who were satisfied (defined as an
average rating of the community as either a 3, 4 or 5 on the CoreQ AL
Resident questionnaire).
Measure #5: Assisted Living Family Satisfaction defined as the percentage of families or
designated responsible party for AL residents who were satisfied (defined as
an average rating of the community as either a 3, 4 or 5 on the CoreQ AL
Family questionnaire).
Sample Selection
The sample selection ideally should be drawn by the organization from their administrative
and/or clinical database and the list of names with contact information provided to the survey
vendor. If a Center, cannot identify any of the exclusions, it is ok to send a survey to those
individuals who would be excluded if such data was available (e.g. those with
dementia/cognitive impairment), however as a result it is likely that they will either not get a
response (and thus lower their response rate) or get a questionnaire back that has been
completed by someone other than the resident or the intended recipient (which will result in
exclusion from the measure calculation).
CoreQ Assisted Living Resident. All current residents in the community are initially
eligible. Residents who meet any of the following criteria are excluded from either
receiving or being administered the CoreQ questionnaire:
• Dementia impairing their ability to answer the questionnaire defined as BIMS
score of 7 or lower or MMSE score of 12 or lower. [Note: we understand that
many ALCs may not have information on cognitive function. We will suggest
administering the survey to all AL residents and assume that those with cognitive
impairment will not complete the survey or have someone else complete on
their behalf which in either case will exclude them from the analysis.]
• Have a legal court appointed guardian.
• Residents on hospice.
• Residents who have been in the ALC for less than two weeks.
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
Ideally, the sample of residents should be drawn from the ALC’s current census records
for all current residents in the building at the time of survey. All residents should be
included minus any meeting the above exclusions.
CoreQ Assisted Living Family. The resident representative (i.e., the primary contact) for
each current resident is initially eligible regardless of their being a family member or
not. Only one primary contact per resident should be selected. Primary contacts who
meet any of the following criteria are excluded from receiving the CoreQ questionnaire:
• Court appointed guardian.
• Family members who reside in another country.
• Residents on hospice.
• Residents who have been in the ALC for less than two weeks.
Ideally, the sample of family members and contact information should be drawn from
the assisted living’s resident contact lists. All primary contacts should be included
except for those meeting exclusion criteria.
Core Q Long-Stay Resident. All current residents are initially eligible. Residents who
meet any of the following criteria are excluded from receiving/administered the CoreQ
questionnaire:
• Dementia impairing their ability to answer the questionnaire defined as having a
BIMS score on the MDS as 7 or lower. [Note: we understand that some SNCCs
may not have information on cognitive function available to help with sample
selection. In that case, we suggest administering the survey to all residents and
assume that those with cognitive impairment will not complete the survey or
have someone else complete on their behalf which in either case will exclude
them from the analysis.]
• Have a legal court appointed guardian.
• Patients on hospice as recorded on the MDS as Hospice (O0100K1 = 1).
• Residents who have lived in the SNCC for less than 100 days.
CoreQ Long-Stay Family. The resident representative (i.e., the primary contact) for each
current resident is initially eligible regardless of their being a family member or not.
Only one primary contact per resident should be selected. Primary contacts who meet
any of the following criteria are excluded from receiving the CoreQ questionnaire:
• Court appointed guardian.
• Family members who reside in another country.
• Patients on hospice as recorded on the MDS as Hospice (O0100K1 = 1).
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
• Residents who have lived in the SNCC for less than 100 days.
Ideally, the sample of family members and contact information should be drawn from
the resident contact records.
CoreQ Short-Stay Discharges. All residents admitted from a hospital to the SNCC
regardless of payor who were discharged back to the community within 100 days of
admission are initially eligible. Discharged back to the community includes discharges to
home or ALC. Residents who meet any of the following criteria are excluded from
receiving/administered the CoreQ questionnaire:
• Discharged to a hospital, another SNCC, psychiatric facility, Inpatient
Rehabilitation Facilities (IRF), or Long Term Care Hospital (LTCH).
• Discharged on hospice.
• Have dementia impairing their ability to answer the questionnaire defined as
having a BIMS score on the MDS as 7 or lower. [Note: we understand that some
SNCCs may not have information on cognitive function available to help with
sample selection. In that case, we suggest administering the survey to all
residents and assume that those with cognitive impairment will not complete
the survey or have someone else complete on their behalf which in either case
will exclude them from the analysis.]
• Have a legal court appointed guardian.
• Left Against Medical Advice (AMA).
• Died during their SNCC stay.
Questionnaire Administration
Timing of Administration. The frequency in which the questionnaires should be
administered can be left up to the provider or vendor except for the CoreQ short-stay
discharges (see below) but should at least be administered once a year.
CoreQ Assisted Living: No specifications on frequency but assume it will be a
cross-sectional collection at a point in time of current resident with some
specified follow-up period.
Core Q Long-Stay Resident: No specifications on frequency but assume it will be
a cross-sectional collection at a point in time of current resident with some
specified follow-up period.
CoreQ Long-Stay Family: No specifications on frequency but assume it will be a
cross-sectional collection at a point in time of current resident with some
specified follow-up period.
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
CoreQ Short-Stay Discharges: Should be initially administered within two weeks
of discharge from the center. The data collection should continue for the next six
months or you may stop once you receive 125 or more valid responses. Please
note you must have at least 20 valid responses to meet the minimum
requirements for the minimum sample size (see Reporting Center’s Results
section for more information). These must be consecutive returns and cannot be
the “best 125 responses” (See Figure 1). The questionnaire should be
administered AFTER discharge not the day of discharge similar to the
requirement for the Consumer Assessment of Health Care Providers and Systems
(CAHPS) survey.
Figure 1. Short-stay survey administration
Follow-up Reminders. The use of follow-up reminders are encouraged to boost the
response rate, but are not required. Incentives to complete the survey may also be
used, but are not required.
Additional Questions. Several additional questions about the respondent completing
the questionnaire should be collected as they are required in the analysis of the
measure (see Figure 2). This does NOT apply to the family CoreQ questionnaires. These
include questions about any help the respondent received in completing the
questionnaire and who helped complete the questionnaire. If these are already
included in an existing questionnaire that the CoreQ is being added to, then use those
questions. They do not have to be exactly worded the same way as the example below.
Additional similarly worded questions are not necessary. Additional questions about the
respondents’ demographics such as age and gender are optional.
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
Figure 2. Example questions to assess help completing the survey
Administering CoreQ by Itself. The CoreQ can be administered by itself as a stand-alone
questionnaire which must include, at a minimum, the CoreQ questions and when given
to residents, the two additional questions about assistance in completing the
questionnaire above.
Including CoreQ in Other Questionnaires. These questions, when added to a vendor’s
questionnaire, should always be grouped together with an introduction sentence: “The
following three/four questions are part of a national initiative to measure the quality of
skilled nursing care centers/assisted living communities”. The Association recommends
that the CoreQ questions appear first on the questionnaire, when used as part of a
larger longer satisfaction questionnaire.
Survey Completion Window. We recommend using only surveys returned within two
months of the resident initially receiving the survey. Responses received after two
months are excluded.
Scoring Survey. Responses to each question will be translated into a numeric response.
• Excellent = 5
• Very good = 4
• Good = 3
• Average = 2
• Poor = 1
See the section on quality measure calculation for how each person’s average response
is calculated and used in the quality measure.
Quality Measure Calculation
Overview
For each respondent, the numeric responses to the three or four questions are summed
together and divided by the total number of questions. This yields an average score for each
respondent. The total number of respondents with a score greater than or equal to 3.0 is
divided by the total number of respondents who are not excluded.
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
Figure 3. Algorithm for calculating a SNCC’s/ALC’s satisfaction CoreQ score
Calculating Respondents Average Score. Each respondent’s answers to the CoreQ need
to be averaged. Table 1 shows an example of how the average answers are calculated
for three different responses. The numeric values are summed and divided by the
number of questions to calculate the average score.
Table 1. Example of calculating average resident’s ratings
Imputing Missing Data. If information from one question is missing, we will impute
missing data by taking the average of the other two (or three) questions. If there is
missing information for two or more questions; the respondent’s information is
excluded.
Table 2 below demonstrates how imputation works for four different scenarios.
Resident B is the only scenario where imputation would occur since data is missing for
only 1 question on the CoreQ. The average of the other two questions is imputed for
the missing question and then the resident’s average is calculated using the imputed
value.
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
Table 2. When and how to impute data when information on a question is missing
Calculating a Center’s/Community’s Satisfaction Rating. The number of respondents with an
average score greater to or equal to 3.0 are counted (numerator) and divided by the total
number of valid responses to the same questionnaire type (denominator) to yield the
center’s/community’s satisfaction score (e.g. percent of respondents who are satisfied).
For example, a center has four respondents with average scores of 2.9, 3.1, 4.0, and 4.2
on the CoreQ long-stay resident questionnaire (see Figure 4). Three residents have an
average score of greater than 3.0. Thus, the numerator is 3 and the denominator is 4 so
the center’s long-stay resident satisfactions core is 75 percent (3 divided by 4).
Figure 4. Demonstrating the calculation of a center/community’s satisfaction rating
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
Excluding Resident Surveys from the Calculation. Surveys that meet any exclusion criteria
should be omitted from the quality measure calculations. These include all:
a. Surveys received outside the time window (two months after the administration
date).
b. Residents’ surveys that have missing data (more than one CoreQ question missing
should be excluded).
c. Residents who indicate that someone else answered the questions for the resident
should also be excluded or added to the family responses. If a resident had help
answering the questions such a reading the questions or writing down their
responses should be included in the calculation.
Reporting Results. In order to report results for a center/community, two criteria must be met.
1. Minimum sample size
A center/community must have a minimum of 20 valid responses (e.g. the
denominator must be > 20); we will accept results for centers/communities with
<20 valid responses but will not include them in any aggregate calculations or
analyses. A valid response is considered to be from a respondent who:
a. Has all items answered or only one question missing on the CoreQ; AND
b. Does not meet any of the exclusion criteria (see exclusion criteria for each CoreQ
questionnaire in questionnaire administration); AND
c. Responses received to the questionnaire within two months of the person being
sent the questionnaire.
2. Minimum response rate
A center/community must also have a minimum response rate of valid responses
of at least 30 percent or greater. The response rate is calculated by counting all
the valid responses (see above) divided by the number of people who were given
the survey to complete. Figure 5 depicts a typical center with 100 residents, of
which 25 meet exclusion criteria (see exclusions above). The remaining 75 are
given the survey, of which 35 are completed; however, ten of them are excluded
(e.g. left questions blank, completed outside the allowed time window, another
person completed the survey for the resident, etc). This leaves 25 usable surveys
from the 75 who received the survey. Thus, the response rate will be 25/75 or
33 percent. We will accept results with response rates less than 30 percent but
will exclude those centers/communities in any aggregate analyses or
calculations. For vendors utilizing phone surveys, the response rate would be the
number of people who completed the survey divided by the number of people
they actually reached by phone.
Long-Stay and Assisted Living surveys can be aggregated for up to six months to meet
minimum response rate and sample size requirements. When submitting data for a six-
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
month sampling period, report the month the last survey sample was drawn. For
example, if a vendor draws a sample in January and continues drawing samples through
June to meet minimum requirements, the month indicated on the upload should be
June not January. Surveys should only be used if they are returned within two months of
the resident initially receiving the survey. If the vendor is not aggregating data over an
extended period to meet sample size and response rate requirements, the month
selected should be the month the sample was drawn. For example, if the vendor draws
a sample of residents in March and continues to collect responses over the allowed two
month period, the month entered on the upload for this sample would be March not
May.
Figure 5. Example to demonstrate calculating response rate
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
Appendix A: CoreQ questions and rating scales
These questions, when added to a vendor’s questionnaire, will always be grouped together
with the introduction sentence: “The following three/four questions are part of a national
initiative to measure the quality of skilled nursing care centers/assisted living communities”.
We recommend that the CoreQ questions appear first on the questionnaire, when used as part
of a larger satisfaction questionnaire.
CoreQ – Short-Stay Discharge
1. In recommending this facility to your friends and family, how would you rate it overall?
Poor Average Good Very Good Excellent
2. Overall, how would you rate the staff?
Poor Average Good Very Good Excellent
3. How would you rate the care you receive?
Poor Average Good Very Good Excellent
4. How would you rate how well your discharge needs were met?
Poor Average Good Very Good Excellent
CoreQ – Long-Stay Residents & Family Members
1. In recommending this facility to your friends and family, how would you rate it overall?
Poor Average Good Very Good Excellent
2. Overall, how would you rate the staff?
Poor Average Good Very Good Excellent
3. How would you rate the care you received?
1
Poor Average Good Very Good Excellent
Core Q – Assisted Living Residents & Family Members
1. In recommending this facility to your friends and family, how would you rate it overall?
Poor Average Good Very Good Excellent
2. Overall, how would you rate the staff?
Poor Average Good Very Good Excellent
3. How would you rate the care you receive?
1
Poor Average Good Very Good Excellent
4. Overall, how would you rate the food?
2
Poor Average Good Very Good Excellent
1
When question #3 is administered to family members, reword the question to say: “How would you rate the care
your family member received?” to reflect the person answering the question is not the resident who received care.
2
Drop question #4 when administered to family members
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
Appendix A1. Examples of CoreQ questionnaire with instructions
Ideally, the formatting will be the same in each survey but we understand that it may need to
be modified to be consistent with the rest of a vendor’s questionnaire. See examples below of
different formatted methods that have been used. Regardless of formatting, the wording,
ordering and grouping of the questions together needs to be identical across all questionnaires.
Example 1. CoreQ short-stay with rating below question using numbers and words
Example 2. CoreQ long-stay resident using grid format
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
Appendix B: Development of questionnaire and measures
Measurement Development and Testing
Two large national corporations have pilot tested the questionnaire in approximately 250
centers; one for individuals discharged following rehabilitation and the other for long- stay
residents and their family members (or person responsible for the resident). We evaluated
multiple methods of calculating the measure and recommended the above approach based on
the statistical evaluation and review by the CEC workgroup on satisfaction measure
development.
The analyses conducted to arrive at this proposed outcome measure can be categorized in the
following phases:
1) Develop the questionnaire
2) Validate the questionnaire
3) Develop the quality measures
4) Test and validate the proposed measures
1) Development of questionnaire
Focus groups with residents, family members and staff from both skilled nursing and
assisted living were conducted. Existing customer satisfaction questionnaires were
reviewed. This resulted in a questionnaire with approximately 25 questions using a 5 point
Likert rating scale.
2) Validation of questionnaire
The draft questionnaire was administered to over 25,000 skilled nursing, both short-term
and long-stay residents, and assisted living residents across several organizations and states
with approximately 10,000 responses. Dr. Castle and Dr. Matt O’Connor conducted
extensive psychometric testing of all the responses. The results identified three core
questions that were reliable and valid across skilled nursing and assisted living. There was
one extra question about the discharge process for the short- stay discharges and one extra
question for assisted living (see appendix A for the final validated questionnaires). These
questions yielded the same information when aggregated together and summed at the
center/community level as analyses using additional questions. Additional questions did not
change a score or ranking. Additional questions did help identify areas needing
improvement but from a quality measure’s standpoint – added no additional information.
3) Development of quality measures
The workgroup tested different methods of summing the scores from the questions for
each person and aggregating their responses to create a center/community score. All the
methods were extremely highly correlated (0.85 to 0.95) and the relative ranking of
center/communities to each other did not change between different methods; indicating
that each method of calculating a center/community measure was statistically, nearly
Getting to the core of customer satisfaction in skilled nursing and assisted living.
Learn more at coreQ.org
identical. We elected to use the method most commonly used in skilled nursing and other
health care settings that is familiar to most health care providers and policy makers. We
calculated the percentage of respondents who were satisfied, defined as responses that
were either 3, 4 or 5. This resulted in an overall score in the 80-90 range, similar to other
measures of satisfaction currently in use.
4) Test and validate selected measure
The questionnaire was used in 280 centers in one corporation for all discharges home over
a six month period and to 220 centers in another corporation to all long-stay residents and
families. The results were calculated for each center and are being correlated with other
quality measures such as hospital readmission rate and discharge to community, a
requirement of the NQF submission process.
