Lies, Damn Lies, and Statistics! Is that Chart Lying to You?
Experienced experts know how easy it is for the naïve viewer to be fooled by a misleading chart or, worse, to inadvertently lie with their own work. If this is revealed during testimony, the best case scenario is some uncomfortable cross examination; the worst case is destruction of the expert’s credibility and serious loss of value to the client. Here I draw on my own experience as an economic expert and use some stylized examples to expose five common traps in quantitative graphics. These are a great starting point when examining opposing expert materials and also to check your own work.
Trap #1. A vertical axis that does not start at zero.
Say you have a bar chart showing the dollar amounts from following hypothetical investment strategies. This one would net $40 million, this one $60 million, that one $80 million, etc. We all know that $80 million is twice as much as $40 million, and $60 million is 50% more, but what if the chart started the axis at, say, $20 million? You know, to zoom in on the area of interest and not have so much empty white space. The relationships in the chart will no longer reflect the actual relationships of the underlying data because $60 will appear twice as large as $40 and $80 will look three times the size. Messing with the axis scale is a classic chart deception, as this Onion articleshows.
Trap #2. A vertical axis that changes from one chart to the next?
A variation on Trap #1 is showing side by side charts with different scales (one chart goes from, say, 0 to 100 while the other goes from 0 to 150). This happens frequently when the chart creator fails to change the default settings provided by a certain chart-making program which-shall-not-be-named. Take those hypothetical investment strategies. You’ve run two sets of simulations showing results under different assumptions. Strategy A did a lot worse under the second set of assumptions, earning $40 million with the first set of assumptions vs. only $30 million with the second set, but the change in axis scale obscures this.
Trap #3. A chart that shows levels when it ought to show changes.
Take two companies whose stock prices both nearly double over a certain period of time. One stock went from 10 to 19.2 and the other stock went from 50 to 99. If you plot the stock prices directly, the change from 10 to 20 looks far less impressive than it really is and performance likely looks worse than the change from 50 to 99 (not true). Plotting stock price returns, or indexing (pegging) both stock prices so they start at the same level will give a more accurate picture of the change that took place.
Trap #4. A chart that shows something that does not exist.
With time series data, like securities prices, line charts are commonly used to show what happened to prices over time. For more thinly traded securities, it may not be appropriate to draw a line between each and every data point, as those lines imply continuous trading in the market. Take an example of a thinly traded stock with a trading halt for one day of a week. Imagine the trading halt relates to an issue important to the litigation, and you can see it is misleading to connect daily price points.
Trap #5. A chart that fails to show something that does exist.
In other words, is there something the chart omits, like an “unknown” or “other” category? I don’t think you need a picture to understand how different something might look, or what kinds of questions your finder of fact might ask, when faced with a more inclusive picture.
Of course, there are almost as many ways to deceive with charts as there are ways to create charts, but looking out for these common traps will heighten your awareness and ability to spot any type of problem.
About The Author
Stephanie Lee is the President and Founder of Levinson Lee Consulting, a boutique consulting firm that helps clients tell stories and persuade with data. Stephanie specializes in economic and financial graphics and offers litigation and expert witness support. After almost a decade of economic expert work at NERA Economic Consulting, Stephanie founded her firm to close the gap between trial graphics consultants, who typically have a legal or design background, and economists, who often prefer to focus on…economics, not on designing materials to explain their work. Stephanie has a BA in economics from Dartmouth College, an MBA from NYU’s Stern School of Business, and she is a CFA Charterholder. She lives in San Francisco, California.