The views of behaviorists (that markets are driven by fear and greed) and rationalists (that fundamentals dictate the market) actually are opposite sides of the same coin, according to early research by Andrew W. Lo.
Basically, he says emotion is a part of rationality, and questions whether professional investors can do their jobs without emotion taking an important role.
Mr. Lo, who is the Harris & Harris Group Professor at MIT's Sloan School of Management and director of its Laboratory for Financial Engineering, is at the early stages of research that he hopes will establish a "new paradigm" for financial economics. Ultimately, if he is successful, these theories could help investors hone their strategies, although Mr. Lo acknowledges this objective is a long way off.
At the root of the differences between the two schools is a major philosophical distinction. Going back to Nobel Laureate Paul A. Samuelson's 1947 Ph.D. thesis, economists have argued that individuals have acted rationally, in a way to maximize their "expected utility." This economic model is based on physics and its clear-cut rules.
"All of us economists suffer from physics envy," Mr. Lo said in an interview. "We want three laws that explain 99% of (economic) behavior, but instead we have 99 laws that explain 3% of behavior."
"But what I have finally concluded, after years of frustration in attempting to reconcile the standard economics paradigm with the realities of financial markets, is that physics may not be the right metaphor for economic systems," Mr. Lo wrote in a paper called "Bubble, Rubble, Finance in Trouble?" The paper is to be published in the "Journal of Psychology and Financial Markets."
Conversely, advocates of behavioral finance, who have poked numerous holes into traditional finance theory, have compiled "a collection of anecdotes," but they don't offer any theories as to why people behave as they do, Mr. Lo said.
Nor do behaviorists explain how biases interact with institutional restrictions, such as when people do not act in line with probabilities. Nor do they say how competing biases interact, nor how behavioral biases change over time and with changing market conditions to create such phenomena as market bubbles.
Sometimes, behavioral tendencies dominate security prices, and arbitrageurs step in to take advantage of price discrepancies. "There are market forces that will bring prices back to their rational levels once again. Such forces will tend to prevent these kinds of behavioral biases from becoming too significant," Mr. Lo said.
"The bottom-line question - the question that I've been struggling with for some time now - is how to reconcile the two. How can we reconcile the existence of behavioral biases with the existence of institutional and market forces that impose limits to irrationality, and how do they balance out?"
Mr. Lo is turning to evolutionary biology or ecology for an answer.
The importance of emotion in rational behavior, Mr. Lo continues, makes sense in light of the late Herbert Simon's work on "bounded rationality," for which Mr. Simon received the Nobel Memorial Prize in Economic Sciences in 1978.
Mr. Simon posited that agents often do not make optimal decisions because of an uncertain future and the costs of acquiring information. Instead, they use "satisficing heuristics"- that is, they develop rules of thumb that provide satisfactory levels of achievement, although not necessarily the best.
In actuality, this practice makes a lot of sense because lacking the full amount of time needed to make the best decisions, people instead create shortcuts.
As Mr. Lo explained in his paper, he owns five jackets, 10 pairs of pants, 20 ties, 10 shirts, 10 pairs of socks, four pairs of shoes and five belts. Given that wardrobe, Mr. Lo has exactly 2 million possible clothing combinations from which to choose - especially if he doesn't mind mixing his plaid jacket with his striped shirt or wearing his Hush Puppies with his pin-striped suit.
His real problem, though, is that it would take him 23.1 days to evaluate completely every possible combination of clothing at hand.
But Mr. Lo can't spend all day - much less 23.1 days - picking the optimal outfit. Instead, he must develop a method -a satisficing heuristic - that yields a suitably speedy, if not the best possible, solution.
But how do people do that without first figuring out the optimal solution? How do they weigh the costs and benefits?
The answer, Mr. Lo said, is that they don't. Instead, "we evolve and adapt. Adaptation and evolution dictate the balance of bounded rationality, and it's the process of natural selection that determines where you are in the spectrum of how much you optimize," he wrote.
In a market context, it means that disparities in pricing that can be arbitraged eventually disappear, as other investors learn they are losing their shirts and seek to correct the imbalance.
"The ability to learn, to adapt heuristics to changes in our environment and new information, is probably the single most important evolutionary event to have occurred in the history of Homo sapiens," he wrote.
This principle can be applied to earnings forecasts, financial bubbles and a host of behavioral finance issues such as overconfidence, overreaction and loss aversion, he added.
It also means the conventional definition of market efficiency needs to be rewritten. "Market efficiency to me means two things: Markets are highly adaptive; and markets are highly competitive. It doesn't mean that markets are perfect all the time, that markets reflect all available information. It means that it's hard to make money," Mr. Lo said.
Mr. Lo is testing his theory in several research projects now under way:
* How behavioral biases evolve. "We're all hard-wired to survive, but not to be risk-averse or preference-seeking," he said.
* How to help burned-out traders - who can be like victims of post-traumatic stress syndrome - return to the market.
* How important natural selection - the survival of the fittest - is in finance.
* How traders and investors use emotions to their advantage. For one study, Mr. Lo and Dmitry Repin, a Boston University neuroscientist, wired 10 professional traders to electronic sensors, finding that emotions played a key role in forming intuitive judgments.
Mr. Lo's work is generating significant controversy in academia.
Richard Thaler, the Robert P. Gwinn Professor of Behavioral Science and Economics at the University of Chicago's Graduate School of Business, doubts that using evolution to understand the origin of heuristics and biases will help predict stock prices.
"The problem with evolutionary explanations of behavioral phenomena is that they are mostly post hoc explanations of known facts. People can invent evolutionary stories for why humans are overconfident, but we are no better able to predict behavior as a result of that story," he wrote in an e-mail to Pensions & Investments.
Others applauded Mr. Lo's efforts but fretted how huge a task Mr. Lo has set for himself.
Said Richard O. Michaud, president of New Frontier Advisors, Boston, who has criticized behaviorists for faulty analysis: "I'm a big fan of Andy Lo's work, but my concern is that it's such a basic place to start, it will take a very long time" to develop basic principles.
Meir Statman, the Glenn Klimek Professor of Finance at Santa Clara University and a leading behaviorist, said evolutionary psychology is the way that behavioral finance "really has been going."
Traditionalist Mark Rubinstein, who is the Paul Stephens Professor of Applied Investment Analysis at the University of California at Berkeley's Haas Business School, said he had tried to apply a Darwinian survival of the fittest model to markets a couple of years ago, but it proved "too hard to do."