menu

Monday, 5 February 2018

Revisiting a Classic Probability Puzzle: the Two Envelopes Problem


Many people have heard about the Monty Hall problem. A similar (but less well known and more mathematically interesting) problem is the two envelopes problem, which Wikipedia describes as follows:
“You are given two indistinguishable envelopes, each containing money, one contains twice as much as the other. You may pick one envelope and keep the money it contains. Having chosen an envelope at will, but before inspecting it, you are given the chance to switch envelopes. Should you switch?”
The problem has been around in various forms since 1953 and has been extensively discussed (see, for example Gerville-Réache for a comprehensive analysis and set of references)  although I was not aware of this until recently.

We actually gave this problem (using boxes instead of envelopes) as an exercise in the supplementary material for our Book, after Prof John Barrow of University of Cambridge first alerted us to it. The ‘standard solution’ (as in the Monty Hall problem) says that you should always switch. This is based on the following argument:
If the envelope you choose contains $100 then there is an evens chance the other envelope contains $50 and an evens chance it contains $200. If you do not switch you have won $100. If you do switch you are just as likely to decrease the amount you win as increase it. However, if you win the amount increases by $100 and if you lose it only decreases by $50. So your expected gain is positive (rather than neutral). Formally, if the envelope contains S then the expected amount in the other envelope is 5/4 times X (i.e. 25% more).
In fact (as pointed out by a reader Hugh Panton), the problem with the above argument is that it equally applies to the ‘other envelope’ thereby suggesting we have a genuine paradox. In fact, it turns out that the above argument only really works if you actually open the first envelope (which was explicitly not allowed in the problem statement) and discover it contains S. As Gerville-Réache shows, if the first envelope is not opened, the only probabilistic reasoning that does not use supplementary information leads to estimating expectations as infinite amounts of each envelope. Bayesian reasoning can be used to show that there is no benefit in switching, but that is not what I want to describe here.

What I found interesting is that I could not find - in any of the discussions about the problem - a solution for the case where we assume there is a finite maximum prize, even if we allow that maximum to be as large as we like. With this assumption it turns out that we can prove (without dispute) that there is no benefit to be gained if you stick or switch. See this short paper for the details:
Fenton N E, "Revisiting a Classic Probability Puzzle: the Two Envelopes Problem" 2018, DOI10.13140/RG.2.2.24641.04960

Thursday, 18 January 2018

Criminally Incompetent Academic Misinterpretation of Criminal Data - and how the Media Pushed the Fake News



On 17 Jan 2018 multiple news sources (e.g. see here, here, and here) ran a story about a new research paper ‎ that claims to expose both the inaccuracies and racial bias in one of the most common algorithms used for parole and sentencing decisions to predict recidivism (i.e. whether or not a defendant will re-offend).

The research paper was written by the world famous computer scientist Hany Farid (along with a student Julia Dressel).

But the real story here is that the paper’s accusation of racial bias (specifically that the algorithm is biased against black people) is based on a fundamental misunderstanding of causation and statistics. The algorithm is no more ‘biased’ against black people than it is biased against white single parents, ‎ old people, people living in Beattyville Kentucky, or women called ‘Amber’. In fact, as we show in this brief article, if you choose any factor that correlates with poverty you will inevitably replicate the statistical ‘bias’ claimed in the paper. And if you accept the validity of the claims in the paper then you must also accept, for example, that a charity which uses poverty as a factor to identify and help homeless people is being racist because it is biased against white people (and also, interestingly, Indian Americans).

The fact that the article was published and that none of the media running the story realise that they are pushing fake news is what is most important here. Depressingly, many similar research studies involving the same kind of misinterpretation of statistics result in popular media articles that push a false narrative of one kind or another.

22 June Update

Our article (5 pages): Fenton, N.E., & Neil, M. (2018). "Criminally Incompetent Academic Misinterpretation of Criminal Data - and how the Media Pushed the Fake News" http://dx.doi.org/10.13140/RG.2.2.32052.55680  Also available here.

The research paper: Dressel, J. & Farid, H. The accuracy, fairness, and limits of predicting recidivism. Sci. Adv. 4, eaao5580 (2018). 

Thanks to Scott McLachlan for the tip off on this story.

See some previous articles on poor use of statistics:

Thursday, 11 January 2018

On lawnmowers and terrorists again: the danger of using historical data alone for decision-making


The short paper and blog posting we did last week generated a lot of interest, especially after Nicholas Taleb retweeted it. An edited version (along with a response from a representative of the Royal Statistical Society) is going to appear in the February issue of Significance magazine (which is the magazine of the RSS and the American Statistical Association). In the mean time we have produced another short paper that explores further problems with the 'lawnmower versus terrorist risk' statistics - in particular the inevitable limitations and dangers of relying on historical data alone for risk assessment:
Fenton, N.E., & Neil, M. (2018). "Is decision-making using historical data alone more dangerous than lawnmowers?", Open Access Report DOI:10.13140/RG.2.2.20914.71363. Also available here.

Wednesday, 3 January 2018

Are lawnmowers a greater risk than terrorists?


Kim Kardashian, whose tweet comparing the threats of lawnmowers and terrorists, led to RSS acclaim
In December 2017 the Royal Statistical Society (RSS) announced the winner of its “International Statistic of the Year”. The statistic was simply "69" which it said was "the annual number of Americans killed, on average, by lawnmowers - compared to two Americans killed annually, on average, by immigrant Jihadist terrorists".  The full RSS citation says that the statistic tweeted by Kim Kardashian ‘highlights misunderstandings of risk’ and ‘illuminates the bigger picture’. Unfortunately, we believe it does exactly opposite as we explain in this brief paper:
Fenton, N.E., & Neil, M. (2018). "Are lawnmowers a greater risk than terrorists?" Open Access Report DOI:10.13140/RG.2.2.34461.00486/1 
As you can see from the tweet by Taleb, this use of statistics for risk assessment was not universally welcomed.


See update to this story here.

Monday, 11 September 2017

An objective prior probability for guilt?



One of the greatest impediments to the use of probabilistic reasoning in legal arguments is the difficulty in agreeing on an appropriate prior probability that the defendant is guilty. The 'innocent until proven guilty' assumption technically means a prior probability of 0 - a figure that (by Bayesian reasoning) can never be overturned no matter how much evidence follows. Some have suggested the logical equivalent of 1/N where N is the number of people in the world. But this probability is clearly too low as N includes too many who could not physically have committed the crime. On the other hand the often suggested prior 0.5 is too high as it stacks the odds too much against the defendant.

Therefore, even strong supporters of a Bayesian approach seem to think they can and must ignore the need to consider a  prior probability of guilt (indeed it is this thinking that explains the prominence of the 'likelihood ratio' approach discussed so often on this blog).

New work - presented at the 2017 International Conference on Artificial Intelligence and the Law (ICAIL 2017) - shows that, in a large class of cases, it is possible to arrive at a realistic prior that is also as consistent as possible with the legal notion of ‘innocent until proven guilty’. The approach is based first on identifying the 'smallest' time and location from the actual crime scene within which the defendant was definitely present and then estimating the number of people - other than the suspect - who were also within this time/area. If there were n people in total, then before any other evidence is considered each person, including the suspect, has an equal prior probability 1/n of having carried out the crime.

The method applies to cases where we assume a crime has definitely taken place and that it was committed by one person against one other person (e.g. murder, assault, robbery). The work considers both the practical and legal implications of the approach and demonstrates how the prior probability is naturally incorporated into a generic Bayesian network model that allows us to integrate other evidence about the case.

Full details:
Fenton, N. E., Lagnado, D. A., Dahlman, C., & Neil, M. (2017). "The Opportunity Prior: A Simple and Practical Solution to the Prior Probability Problem for Legal Cases". In International Conference on Artificial Intelligence and the Law (ICAIL 2017). Published by ACM. Pre-publication draft.
See also

Thursday, 7 September 2017

Recommendations for Dealing with Quantitative Evidence in Criminal Law


From July to December 2016 the Isaac Newton Institute Programme on Probability and Statistics in Forensic Science in Cambridge hosted many of the world's leading figures from the law, statistics and forensics with a mixture of academics (including mathematicians and legal scholar), forensic practitioners, and practicing lawyers (including judges and eminent QCs). Videos of many of the seminars and presentation from the Programme can be seen here.


A key output of the Programme has now been published. It is a very simple set of twelve guiding principles and recommendations for dealing with quantitative evidence in criminal law for the use of statisticians, forensic scientists and legal professionals. The layout consists of one principle per page as shown below.



Links:

Monday, 14 August 2017

The likelihood ratio and its use in the 'grooming gangs' news story


This blog has reported many times previously (see links below) about problems with using the likelihood ratio. Recall that the likelihood ratio is commonly used as a measure of the probative value of some evidence E for a hypothesis H; it is defined as the probability of E given H divided by the probability of E given not H.

There is especially great confusion in its use where we have data for the probability of H given E  rather than for the probability of E given H. Look at the somewhat confusing argument here in relation to the offence of 'child grooming' which is taken directly from the book McLoughlin, P. “Easy Meat: Inside Britain’s Grooming Gang Scandal.” (2016):



Given the sensitive nature of the grooming gangs story in the UK and the increasing number of convictions, it is important to get the maths right. The McLoughlin book is the most thoroughly researched work on the subject.  What the author of the book is attempting to determine is the likelihood ratio of the evidence E with respect to the hypothesis H where:

H: “Offence is committed by a Muslim” (so not H means “Offence is committed by a non-Muslim”)

E: “Offence is child grooming”

In this case, the population data cited by McLoughlin provides our priors P(H)=0.05 and, hence, P(not H)=0.95. But we also have the data on child grooming convictions that gives us P(H | E)=0.9 and, hence, P(not H | E)=0.1.

What we do NOT have here is direct data on either P(E|H) or P(E|not H). However, we can still use Bayes theorem to calculate the likelihood ratio since:

So, in the example we get:



Hence, while the method described in the book is confusing, the conclusion arrived at is (almost) correct (the slight error in the result, namely 170.94 instead of 171, is caused by the authors rounding  10 divided by 95% to 10.53).

See also