Bark’s Bytes #34 | What DHS Thinks It Knows From the NCI Survey
Published April 3, 2017
In March 2017, DHS released a data brief on “Employment Choices for People with Disabilities in Minnesota” that summarized – “What We Know” about people with Intellectual or Developmental Disabilities (I/DD). Recently, DHS has been publicly stating that we know that 50% of people with disabilities want competitive integrated employment; and, in a recent article, Commissioner Piper was quoted as saying – “50% of people with I/DD and 40% of people with physical disabilities who are unemployed want to work”. I have been told both assertions are based on the most recent results of the National Core Indicators (NCI) Survey and, after reviewing the Minnesota NCI report, can find no data that supports either statement.
Before sharing some of the report results it is important that the reader understands what the NCI Survey is and the following is the language from the “NCI Adult Consumer Survey Outcomes – Minnesota Report 2014 – 2015 Data”.
The National Core Indicators survey is a voluntary effort by state developmental disability agencies to track their performance using a standardized set of consumer and family/guardian surveys with nationally validated measures. The effort is coordinated by the National Association of State Directors of Developmental Disabilities Services (NASDDDS) and the Human Services Research Institute (HSRI).
NCI has developed more than 100 standard performance measures (or ‘indicators’) that states use to assess the outcomes of services for individuals and families, including outcomes in the areas of employment, rights, service planning, community inclusion, choice, health, and safety. In 2015-16 a total of 45 states, the District of Columbia, and 22 sub-state entities were participating in NCI.
The NCI Family/Guardian Survey is used to gather data on family outcomes. It is mailed to families who have an adult family member who lives outside of the family home and receives services from the State Developmental Disabilities (DD) Agency (that is, the survey respondent does not live with the person receiving supports). The survey collects demographic information on the individual receiving services and the person who fills out the survey (the ‘respondent’; usually a parent) as well as information on services and supports received. The survey is continually refined and tested to ensure that it is valid and reliable.
In 2015-16, a total of 6,173 Family/Guardian Surveys were completed across 14 states. The survey contained six groupings of questions (“sub-domains”) that probe specific areas of quality service provision: information and planning, access and delivery of services, choice and control, community connections, satisfaction, and outcomes. Respondents also had the option of writing open-ended comments concerning their family’s participation in the service system.
States were asked to administer the survey to a random sample of at least 1,200 families, all of whom have an adult family member with a developmental disability who lives outside of the family home and receives at least one direct service or support other than service coordination.
All states mailed out a paper survey to families selected in their sample. A sample size of 1,200 was recommended with the expectation of a 40% return rate or greater (yielding 400 or more usable responses per state). However, most states decided to sample more than 1,200 families, expecting a lower response rate than 40%. A final sample size of 400 would guarantee a +/- 5% margin of error and a 95% confidence level when interpreting the results. Both the confidence interval and margin of error used are widely acceptable standards for reviewing results, regardless of population size. States with fewer than 1,200 potential respondent families were instructed to send surveys to all eligible families. With response rates lower than expected, we also included in our national reports those states that submitted fewer than 400 surveys up to a margin of error of no greater than +/- 7%.
The NCI Family/Guardian Survey is a tool for assessing system-wide performance. It is not intended to be used to monitor individuals or providers. The NCI Statewide Average should not be interpreted as defining or providing a benchmark for “acceptable” levels of performance or satisfaction. Instead, it describes average levels of performance or satisfaction across the State; it is up to public managers, policy-makers, and other stakeholders to use the data to determine programmatic and policy-related priorities. (End of description)
While Minnesota was asked to survey at least 1,200 families, it is unknown how many surveys were actually sent to families by DHS. At most, it appears they had 404 respondents to the employment domain which is .0027% (404/150,262) or less than 3/10ths of 1% of the state’s total population of people with I/DD (2nd bullet below) and only 1.6% (404/26,000) of the people served in supported employment settings. So, based on the NCI report, what does DHS think it knows?
1. Apparently DHS believes that – “Throughout the world, paid work is a crucial aspect of culture and identity. Many people organize their lives around their jobs. Employment helps define a person’s place in the community.” Because I am not sure that paid work is a crucial aspect of the aborigine culture and identity, or many others, I will confine my thoughts to the State of Minnesota beginning with findings from two independent public sources.
- From the “StateData: The National Report on Employment Services and Outcomes 2015” published by the Institute for Community Inclusion at the University of Massachusetts Boston; in 2014, Minnesota had a total population of 5,457,000 of which 3,217,676 were of working age (16-64) with only 2,603,597 or 81% employed.
- From the same report, a total of 150,262 or 2.7% of the total population had a cognitive disability (I am going to use the term I/DD). When applied to the working age group of 3,217,676 (bullet above), the result is an estimate that 86,877 people between 16 – 64 have an I/DD. If the norm is that only 81% of working age adults without a disability are employed, then only 70,370 Minnesotans with I/DD would be the norm expected to work.
- In school year 2010 – 2011 the Minnesota Department of Education reported an unduplicated count of 24,130 special education students ages 16 – 21 (more recent data could not be found). Subtracting that group from the 70,370 people with an I/DD projected to be working (2nd bullet) leaves 46,240 people with I/DD between the ages of 22 – 64 that should, by all accounts, be employed.
- From a 3/27/17 presentation by Vikki Wachino, former Director of CMS, 2 of 5 people with disabilities are funded by Medicaid (MA). Applied to the 22 – 64 age group results in 18,496 working age adults with disabilities funded by MA and closely matches the number of people with disabilities served in Day Training & Habilitation programs. It also likely means that the balance of 27,744 working age adults (22 – 64) with disabilities not funded by MA (60%) are working independently without formal I/DD support services.
2. DHS message – “Only 26% of them had a paying job in the community”.
Report finding – of 404 respondents, 105 had a paid job in the community which is 26% and 9% higher than national average.
My interpretation – why doesn’t DHS acknowledge how successful Minnesota providers have been in finding competitive integrated work for people with I/DD long before they approved the employment first policy.
3. DHS message – “Among those without a paid job in the community, 59% wanted one”.
Report finding – of 140 respondents without a paid job in the community, 82 wanted a paid job in the community (10% higher than national average).
My interpretation – if 105 respondents have a job in the community (item 2), then 299 do not (404 – 105), and if only 82 “wanted one” this is 27% of the respondents not the 59% reported. Also, it is important to remember that most of the respondents are likely to be the most capable and motivated to complete the survey creating a self-selection bias that results in higher “wanted” responses than would be found in a non-biased group.
4. DHS message – “Most people told us they like their paid community job (91%), but when asked if they’d rather work somewhere else, many (26%) said they would rather work somewhere else”.
Report finding – If 105 respondents have a paid job in the community (item 2), and 26% of them would rather work somewhere else, that is 27 people which is only 7% (27/404) of all respondents.
My interpretation – in my view 7% of the respondents does not rise to the level of “many”.
5. DHS message – “Similarly, most agreed they liked their day program (86%), but even though they like their program, 35% of those told us they would rather be doing something else”.
Report finding – of the 267 respondents attending a day program (17% lower than national average), 35% or 93 people, would rather be doing something else which is 23% (93/404) of all respondents.
My interpretation – Again, why doesn’t DHS acknowledge that our numbers served in supported employment services is significantly less than the national average and; exactly what is “something else” because if it has anything to do with playing golf count me as part of the 23%.
As most know, anyone can twist the numbers to support their position – as I have in this editorial – and the NCI report makes it clear that the data is “not intended to be used to monitor individuals or providers” and it is up to “public managers, policy-makers, and other stakeholders to use the data to determine programmatic and policy-related priorities”. With results from only .0027% of the population, and instructions that findings are not to be used to monitor individuals or providers, why the message from DHS that “50% of people with I/DD and 40% of people with physical disabilities who are unemployed want to work”? Also, I don’t remember any meetings between public managers, policy-makers, and other stakeholders on how to use the NCI data to determine programmatic and policy-related priorities?
Instead, what we have is a bureaucracy determined to prop-up its flawed philosophy found in the employment first policy that “all working age Minnesotans with disabilities can work, want to work, and can achieve competitive integrated employment” despite an independent data source showing that only 81% of working age adults without disabilities in Minnesota are employed. Furthermore, from the early outcomes of the WIOA section 511 process, anecdotal results suggest that less than 15% of clients served in supported employment programs are interested in even considering competitive integrated work; and, I suspect, fewer still will actually pursue competitive integrated work when they fully understand the realities of that choice.
Let me be clear, I support individualized services and when a client expresses a desire to explore competitive integrated work that preference should be honored as it is routinely occurring in supported employment programs. With less than 26,000 people served by less than 150 providers one would think that if DHS truly wanted to know how people with I/DD felt about the quality of their lives we could easily get that data and not rely on some national report based on .0027% of the population. The question is – when is DHS going to stop issuing byzantine quotes and genuinely listen to what people with I/DD and their families want so that new services can be added without diminishing current options?