THE POISON GARDEN website       

Sunday 29th July 2012

I started a recent blog by explaining;

‘When I use a number to support my argument, it is robust, accurate and completely reliable. When you use a number to refute my point it is a statistic and we all know the one about ‘lies, damn lies and statistics’.’

Today, I want to look at two lots of statistics and see if either of them contain numbers.

I don’t doubt the intention of those who produce official surveys. They have a difficult task and, demonstrably, try and improve the quality of their data with each survey. Thus the British Crime Survey (BCS) has become the Crime Survey for England and Wales (CSEW) to reflect what has always been its geographical range. I wish they had gone further, however, and called it the Crime Survey of Households in England and Wales because it is the idea that it is a full survey that leads to the problems.

In addition to its general survey of experience of crime and perceptions about its prevalence, the CSEW surveys illicit drug use on a self-report basis and produces separate documents about what it calls ‘Drug Use Declared’.

There’s another difference this time; the data has been published online with supporting downloadable Excel spreadsheets as well as in a single pdf report. There’s an entry page¹ giving details of all the different sections and access (at the bottom of the page) to the pdf.

The way the CSEW works is that interviewers go to a sample of households and ask questions about a variety of matters related to crime. The interviewer then asks if the interviewee is willing to take part in the drug use section and, if they agree, the interviewee is asked to complete a survey on the interviewer’s laptop so that the responses are anonymous.

The drug use section only interviews people in the age range 16 to 59 because there are other ways that drug use by the under 16s is assessed and, the belief is, drug use by over 60s is too rare to make a survey economically justifiable. The Office for National Statistics is responsible for the CSEW but the Home Office extracts and analyses the ‘drug use declared’ information.

Like the terms and conditions for online purchases or software downloads, one of the most important documents of the whole publication, the ‘User Guide’ containing an explanation of the method and giving its limitations², is unlikely to be read by the majority of those interested in the report as a whole.

That’s unfortunate because it does contain some very significant information that should colour the way the survey is read and used. The User Guide notes;

‘…the CSEW does not cover some small groups, potentially important given that they may have relatively high rates of drug use: notably the homeless and those living in certain institutions such as prisons. Nor, in practice, will any household survey necessarily reach those problematic drug users whose lives are so busy or chaotic that they are hardly ever at home or are unable to take part in an interview.’

In other words, this is a survey of drug use amongst people who are least likely to be drug users.

It is very easy to demonstrate the flaws that arise just by looking at heroin, methadone and crack cocaine use. The CSEW finds that 92,000 people used one or other of these substances in the previous month. It gives a range of 50-142,000 for its estimate. I’ve taken the last month prevalence figures because I think it is fair to assume that problem drug users will come from the population with the highest levels of drug use.

Problem drug users are, of course, of interest to other bodies than just the Home Office. The National Treatment Agency for Substance Abuse (NTA) is part of the National Health Service (NHS) and, according to its ‘About Us’ page ‘was created as a Special Health Authority in 2001 to improve the availability, capacity and effectiveness of drug treatment in England’. As part of its work, it funds combined research by the University of Glasgow and the University of Manchester into what a private company would call its target market.

The NTA publishes the work of the two universities on its site. ‘Estimates of the Prevalence of Opiate Use and/or Crack Cocaine Use, 2009/10’ is the most recent such report.³ This work found that there were an ‘estimated 306,150 opiate and/or crack cocaine users aged 15 to 64 in England in 2009/10’. I’m not comfortable with the estimate being to the nearest one, especially when giving the 95% confidence interval as 299,094 – 316,916, but I’ll settle for 300k, give or take, say, 20k.

Now, the period covered is different and the age range is slightly different but I think we can be fairly sure such differences don’t explain the discrepancy between the NTA’s 300k and the CSEW’s 92k.

Trying to explain the difference gets into more semantic conundrums. The CSEW User Guide talks about the ‘homeless’ but, officially, there are ‘homeless households’, that is households in temporary accommodation. You might expect the CSEW to be able to survey this group but let’s suppose they don’t and that the approximately 60,000 households in temporary accommodation⁴ at any time are excluded. In addition, estimates suggest around 2,000 people sleep rough in England⁵. The prison population is no more than 90,000 (based on the September 2010 total of 84,000⁶)

So the total excluded from the CSEW is around 152,000 (apart from ‘those problematic drug users whose lives are so busy or chaotic that they are hardly ever at home or are unable to take part in an interview’) and if all of them were using opiates or crack cocaine we’d still be around 50k short. Of course, they are not all using. Government figures suggest that less than 10% of prisoners use ‘drugs’ though campaign groups believe the figure is closer to 35%. That would suggest 30k prisoners are using. The highest reported overall prevalence in the world is 2.8% so if we said 10% of the 62k ‘homeless’ were using we’d still be grossly over-estimating.

By these over-estimates, you can get the CSEW’s 92k up to just under 150k, still only half the NTA figure.

I’m not saying the NTA’s figure is completely correct but the actual number of people receiving treatment is much more certain and this is around 200k per year from the NTA’s own figures.

Very clearly, the CSEW is meaningless when it comes to the number of users of opiates and crack cocaine, the substances accepted as leading to almost all problem drug use. It may well be just as meaningless in all other areas.

I’ve had these concerns about the crime survey data for some years but assumed it was something I wasn’t understanding as most people, on all sides of the drug debate, seemed to accept them. I was very pleased, therefore, to see Russell Newcombe point out this discrepancy on Twitter.

Why does this matter? Because these numbers are accepted at face value by most people and policy can be based on them. Even those who may be concerned about the absolute values seem to be willing to accept that the surveys provide a useful measure of trend over time.

I’m not sure they do. In fact, I’m pretty convinced that the Drug Use Declared figures from the CSEW offer nothing of value. I'd be interested to hear your opinion.

In a future piece, I’ll look at what they say about Catha edulis, khat, a substance added to the survey this year.

References

1. Drug Misuse Declared: Findings from the 2011/12 Crime Survey for England and Wales Home Office 25th July 2012
2. User Guide to Drug Misuse Declared: Findings from the Crime Survey for England and Wales Home Office 25th July 2012
3. Estimates of the Prevalence of Opiate Use and/or Crack Cocaine Use, 2009/10 The National Treatment Agency for Substance Abuse 5th September 2009
4. Homelessness The Poverty Site
5. Number of rough sleepers in England rises by a fifth The Guardian 23rd February 2012
6. Offender Management Statistics Quarterly Bulletin Ministry of Justice January 2011  

'Is That Cat Dead? - and other questions about poison plants' is now also available in Kindle form from Amazon.