If you like my blog, please tell others.
If you don’t like it, please tell me.
Figures don’t lie, but liars figure. So do people who figure with good intentions but inadequate skill or thought.
It was all over the news. “U.S. gun stores and gun shows are the source of more than 90 percent of the weapons being used by Mexico's ruthless drug cartels, according to U.S. and Mexican law enforcement officials.” However the story had a bit of a problem – it was dead wrong, the result of misuse of data.
William La Jeunesse and Maxim Lott of Fox News published the facts. That number comes from a limited and abnormal sample of guns recovered by Mexican authorities. During 2007 and 2008 Mexico recovered 29,000 guns at crime scenes. They submitted 11,000 of those to the US for tracing. Only 6,000 could actually be traced with 90% of those 6,000 coming from the US. Over 23,000 of the guns either could not be traced or weren’t even submitted because there was no reason to do so. Many obviously came from Russia, South America, China etc
This is one example of the errors caused by poor understanding of what data means. The claim that 90% of those guns came from the US fooled a lot of people who should have known better, even the US secretary of state. Some may have been deliberately trying to mislead us, but most were probably just victims of the normal human failure to understand statistics. We tend to think that what is easily visible is representative of the whole.
Worse yet, we often think that anything precisely calculated must be correct. The claim that 90% of Mexican criminal weapons came from the U.S. was very precise – and very wrong. That number comes from equations that somebody manipulated to perfection, based on poor input. The source seems to have forgotten that applied mathematics is much more than the manipulation of equations, it requires understanding of the “physics” behind those equations and of the information fed to the equations. Even if the equations are perfect, they will give the wrong answer if the numbers used in the calculation are bad, Garbage In Garbage Out (GIGO).
This is not limited to politics. Two winners of the Nobel Memorial Prize for Economics served on the board of Long Term Capital Management, an investment firm which failed spectacularly. Their calculation was impeccable but accuracy in calculation was not enough to warn them of the oncoming problem.
I admit that I have a certain affinity for mathematics. I can enjoy grinding through equations and calculating results. However in this blog it would be a mistake to provide a treatise on how to calculate probabilities (though I would have fun doing that). First, I would probably bore any readers I have and lose them. Secondly, the bigger issue is not the actual calculation but determination of what goes into those equations. That is why Taleb* claims that “mathematics is a tool to meditate, not compute.” Anyone interested in the means of calculation can take any number of classes or read any of a variety of good books on the subject. Let us here confine ourselves to the “meditation” aspect of the problem.
What ought we to “meditate” on? First we should think about why we even consider any theory or model of the world in the first place. What makes it attractive to us? Its simplicity? Its usefulness? It’s beauty? (Make no mistake, scientists and mathematicians prefer beauty in their theories.) Are we being mislead because of our preferences?
Second, what other theories might explain the facts? Is there any reason to prefer one over another?
Third, are we considering all the facts? Are the facts we consider pertinent to the question at hand?
Forth, how can we know if the theory or model works? I’ve previously indicated that in any situation there are probably an unlimited number of theories that fit the facts. We need a way of distinguishing between those likely to be true and the rest of them. This is a knotty problem, one has troubled science for centuries and will almost certainly continue to cause trouble. The best answer we know comes from philosopher of science Karl Popper. His answer is that the theory must be falsifiable. By that me meant that it must make predictions that can be tested with clear results that might disagree with the theory.
For example, Newton’s famous second law of motion says that the acceleration of a body is directly proportional to the applied force. We can measure the force and the acceleration on a wide variety of objects under many circumstances. When we do so we find that in most cases the law holds to as accurately as we are able to measure. However at speeds approaching the velocity of light the data contradict that law. Newton’s second law is wrong, though it is very useful at ordinary speeds. At higher speeds we must use Einstein’s theory of relativity.
In the political arena we have many theories of how government should work. Frankly, none of them are perfect, probably because human beings are imperfect. However some work better than others. The unfortunate problem is that politicians seldom pay attention to the results when they try to modify a system. Market economies consistently out-produce controlled economies, yet politicians (and many voters) consistently ignore that fact as they increase government power and restrict individual and market freedom.
Next issue I intend to address one reason that government control causes problems with the economy.
*Nassim Nicholas Taleb, Fooled by Randomness, p216
Showing posts with label fact. Show all posts
Showing posts with label fact. Show all posts
Wednesday, July 1, 2009
Wednesday, June 24, 2009
The Theory Fits Perfectly, So What’s Wrong?
If you like my blog, please tell others.
If you don’t like it, please tell me.
“Hey! Look at this!” Your coworker shows you a chart of the closing price of a certain stock every day for the last month. Then he shows you a graph of a mathematical equation that follows the ups and downs of that stock price almost perfectly. He continues, “I’ve discovered the equation of how the price of this stock goes up and down. It’s going to go up tomorrow. All we have to do is buy now, then sell tomorrow to make some money.” Would you follow his advice? If you do you will probably set yourself up for disappointment.
“But wait,” you might ask. “How can the stock price fit that equation so well up through today and not predict what it will do tomorrow?” The answer is that there can be more than one theory that fits the data and those theories can be quite different. In fact there are an unlimited number of equations that fit your coworker’s data, and they can’t all be right. Some of those equations show the price increasing tomorrow; some show it decreasing. We could even find an equation that shows the stock price as negative tomorrow while still fitting the existing data!*
The same applies to non-mathematical theories. Chicken farmers are well aware that the sun comes up shortly after the rooster crows. They might theorize that the rooster crowing is what causes the sun to rise, but that is hardly proof of the theory. We cannot accept that a theory is correct just because it fits existing data. That is true for things like the coworker’s equation and for other theories. The theory may be true, it may be true in certain circumstances, or it may be only chance that allowed it to fit the data.
The history of science is littered with the ruins of theories later shown to be only approximations or in some cases completely false. Yet the human mind still tends to regard agreement with data as proof of theoretical correctness. Why? Almost certainly because we try to make sense of the world. To do so, we make theories to explain the facts. Then we believe those theories, sometimes even in the face of new facts that contradict them.
That is not all bad. Having a theory of how the world works allows us to live better lives. For example, if the farmer did not have a theory of how the seasons work, he would not be able to effectively plant and harvest his crops. More sophisticated theories allow us to build everything from homes to supercomputers, even rockets to take us to the moon. Those theories can be very useful – but only to the extent that they reflect reality. The theory that vaccines can prevent disease has helped wipe out smallpox and otherwise greatly improved our health. However the old theory that disease was often caused by too much blood in the body led to physicians draining badly needed blood from sick people. That theory probably killed thousands of people.
Humans and other animals seem programmed to seek theories or mental models to explain the world. That can be useful but it can also go too far, leading us into incorrect and sometimes dangerous actions as we try to control important aspects of our lives. B.F. Skinner showed that pigeons fed at random times will develop strange ways of trying to control the food delivery. Whatever action they happen to be doing when food arrives becomes associated in their minds with the food, so they repeat that action when they get hungry. One bird would turn counter-clockwise; others would swing their heads back and forth in a pendulum motion. The birds did that consistently, long after the event that caused them to associate those actions with food. Their theories about what caused the food to appear simply didn’t fit reality.
Humans too are subject to such misleading theories. What sports fan has not tried repeating something he happened to be doing when things went well for his team? I have to admit having been tempted to do such myself. My team was doing poorly while I listened to the game. I had to go do something else and when I got back the team had done much better. It occurred to me that I should turn off the radio so the team would continue to do well. Come on sports fans out there, admit it. What have you done to try to help your team? Is there any rational reason to believe it would work?
Think is limited to people who don’t understand statistics? Think again. Nassim Nicholas Taleb is probably as statistically sophisticated a person as walks the face of this earth. He worked as a trader and one day the taxi dropped him at a different door than where he usually entered the building. That day his account skyrocketed, one of his best days ever. Next morning, without even thinking about it, he asked the taxi driver to drop him at that same door. Then he noticed that he had unconsciously put on the same stained tie he wore the day before! He knows enough to realize that neither the tie nor the door had anything to do with his success, but his natural tendency was still to repeat irrelevant actions from that successful day.
Following theory without or even in spite of evidence is probably nowhere more prevalent than in politics and government. The big government, controlled economy model remains very popular in the world today. In many circles it is considered naïve to even consider anything else. Yet which type of economy has consistently caused problems by over-production and which consistently creates a surplus only of misery? In spite of that abundant evidence, many people still want government to control the economy and make centralized decisions for all of us. (For more on why that does not work well, see my blog on this site, “Who Pays, Who Uses, Who decides?”)
So how can we know that our mental models reflect reality, not just happenstance or some partial data? How can we be certain new data won’t overturn them? Actually we can’t, but with care we can be much more confident and at least weed out the worst of our erroneous theories. Next blog I intend to discuss how to do that.
*For those of mathematical bent and interested in fitting equations to data, we can always fit n + 1 data points with a polynomial of order n. We can fit two data points with a polynomial of the form A + BX. For three points we can use A +BX +CX^2. If our stock price data is for 20 days, a polynomial including values up to X^19 can fit those data perfectly. Then by adding one more term, for X^20, we can fit any next point we want, including negative numbers.
If you don’t like it, please tell me.
“Hey! Look at this!” Your coworker shows you a chart of the closing price of a certain stock every day for the last month. Then he shows you a graph of a mathematical equation that follows the ups and downs of that stock price almost perfectly. He continues, “I’ve discovered the equation of how the price of this stock goes up and down. It’s going to go up tomorrow. All we have to do is buy now, then sell tomorrow to make some money.” Would you follow his advice? If you do you will probably set yourself up for disappointment.
“But wait,” you might ask. “How can the stock price fit that equation so well up through today and not predict what it will do tomorrow?” The answer is that there can be more than one theory that fits the data and those theories can be quite different. In fact there are an unlimited number of equations that fit your coworker’s data, and they can’t all be right. Some of those equations show the price increasing tomorrow; some show it decreasing. We could even find an equation that shows the stock price as negative tomorrow while still fitting the existing data!*
The same applies to non-mathematical theories. Chicken farmers are well aware that the sun comes up shortly after the rooster crows. They might theorize that the rooster crowing is what causes the sun to rise, but that is hardly proof of the theory. We cannot accept that a theory is correct just because it fits existing data. That is true for things like the coworker’s equation and for other theories. The theory may be true, it may be true in certain circumstances, or it may be only chance that allowed it to fit the data.
The history of science is littered with the ruins of theories later shown to be only approximations or in some cases completely false. Yet the human mind still tends to regard agreement with data as proof of theoretical correctness. Why? Almost certainly because we try to make sense of the world. To do so, we make theories to explain the facts. Then we believe those theories, sometimes even in the face of new facts that contradict them.
That is not all bad. Having a theory of how the world works allows us to live better lives. For example, if the farmer did not have a theory of how the seasons work, he would not be able to effectively plant and harvest his crops. More sophisticated theories allow us to build everything from homes to supercomputers, even rockets to take us to the moon. Those theories can be very useful – but only to the extent that they reflect reality. The theory that vaccines can prevent disease has helped wipe out smallpox and otherwise greatly improved our health. However the old theory that disease was often caused by too much blood in the body led to physicians draining badly needed blood from sick people. That theory probably killed thousands of people.
Humans and other animals seem programmed to seek theories or mental models to explain the world. That can be useful but it can also go too far, leading us into incorrect and sometimes dangerous actions as we try to control important aspects of our lives. B.F. Skinner showed that pigeons fed at random times will develop strange ways of trying to control the food delivery. Whatever action they happen to be doing when food arrives becomes associated in their minds with the food, so they repeat that action when they get hungry. One bird would turn counter-clockwise; others would swing their heads back and forth in a pendulum motion. The birds did that consistently, long after the event that caused them to associate those actions with food. Their theories about what caused the food to appear simply didn’t fit reality.
Humans too are subject to such misleading theories. What sports fan has not tried repeating something he happened to be doing when things went well for his team? I have to admit having been tempted to do such myself. My team was doing poorly while I listened to the game. I had to go do something else and when I got back the team had done much better. It occurred to me that I should turn off the radio so the team would continue to do well. Come on sports fans out there, admit it. What have you done to try to help your team? Is there any rational reason to believe it would work?
Think is limited to people who don’t understand statistics? Think again. Nassim Nicholas Taleb is probably as statistically sophisticated a person as walks the face of this earth. He worked as a trader and one day the taxi dropped him at a different door than where he usually entered the building. That day his account skyrocketed, one of his best days ever. Next morning, without even thinking about it, he asked the taxi driver to drop him at that same door. Then he noticed that he had unconsciously put on the same stained tie he wore the day before! He knows enough to realize that neither the tie nor the door had anything to do with his success, but his natural tendency was still to repeat irrelevant actions from that successful day.
Following theory without or even in spite of evidence is probably nowhere more prevalent than in politics and government. The big government, controlled economy model remains very popular in the world today. In many circles it is considered naïve to even consider anything else. Yet which type of economy has consistently caused problems by over-production and which consistently creates a surplus only of misery? In spite of that abundant evidence, many people still want government to control the economy and make centralized decisions for all of us. (For more on why that does not work well, see my blog on this site, “Who Pays, Who Uses, Who decides?”)
So how can we know that our mental models reflect reality, not just happenstance or some partial data? How can we be certain new data won’t overturn them? Actually we can’t, but with care we can be much more confident and at least weed out the worst of our erroneous theories. Next blog I intend to discuss how to do that.
*For those of mathematical bent and interested in fitting equations to data, we can always fit n + 1 data points with a polynomial of order n. We can fit two data points with a polynomial of the form A + BX. For three points we can use A +BX +CX^2. If our stock price data is for 20 days, a polynomial including values up to X^19 can fit those data perfectly. Then by adding one more term, for X^20, we can fit any next point we want, including negative numbers.
Subscribe to:
Posts (Atom)
