Reuters - Fears that high-speed traders have been rigging the U.S. stock market went mainstream last week thanks to allegations in a book by financial author Michael Lewis, but there may be a more serious threat to investors: the increasing amount of trading that happens outside of exchanges.

Some former regulators and academics say so much trading is now happening away from exchanges that publicly quoted prices for stocks on exchanges may no longer properly reflect where the market is. And this problem could cost investors far more money than any shenanigans related to high frequency trading.

When the average investor, or even a big portfolio manager, tries to buy or sell shares now, the trade is often matched up with another order by a dealer in a so-called "dark pool," or another alternative to exchanges.

Those whose trade never makes it to an exchange can benefit as the broker avoids paying an exchange trading fee, taking cost out of the process. Investors with large orders can also more easily disguise what they are doing, reducing the danger that others will hear what they are doing and take advantage of them.

But the rise of "off-exchange trading" is terrible for the broader market because it reduces price transparency a lot, critics of the system say. The problem is these venues price their transactions off of the published prices on the exchanges - and if those prices lack integrity then "dark pool" pricing will itself be skewed.

Around 40 percent of all U.S. stock trades, including almost all orders from "mom and pop" investors, now happen "off exchange," up from around 16 percent six years ago.

This trend is "a real concern," John Ramsay, former head of the U.S. Securities and Exchange Commission's (SEC) Trading and Markets division, said on the sidelines of a conference in February. "We have academic data now that suggests that, yes, in fact there is a point beyond which the level of dark trading for particular securities can really erode market quality."

Given the $21.4 trillion worth of U.S. stocks that were traded in 2012, even a small mispricing can move the needle by tens of billions of dollars.

Lewis' new book - "Flash Boys: A Wall Street Revolt" - says that high speed traders bilk that kind of money from investors every year. He focuses on how high-frequency trading firms use ultra- fast telecom links, microwave towers and special access to exchanges to gain an edge over other traders.

The U.S. Justice Department is investigating high-speed trading for possible insider trading, Attorney General Eric Holder told lawmakers on Friday. Other regulators and the FBI have also confirmed they are looking into potential wrongdoing by high-frequency stock traders.

But whether or not high-speed trading is sinister, revenues for these firms have been declining for years: in 2013, they were about $1 billion, after peaking at around $5 billion in 2009, according to estimates by Rosenblatt Securities. If, as Lewis says, these traders are doing nothing more than ripping off the rest of the market, it's a shrinking problem.

Meanwhile, as the revenue from high frequency trading has waned, trading outside of public exchanges has been on the rise, threatening to roll back decades of progress towards more transparent markets.

EXCHANGES ARE LAST RESORT

A brokerage has several ways to fill customers' orders. It can match buy and sell orders from its own customers, known as "internalizing," or sell its orders to another broker that can do the same.

Brokers also send trades to "dark pools," which are similar to exchanges, except the fees are lower and they are anonymous, with orders going unreported until after they have been executed. And finally, they can send trades to exchanges, where they will have to pay higher fees.

"The exchanges have basically become the liquidity venue of last resort," said Manoj Narang, chief executive of HFT firm and technology vendor Tradeworx.

Around 45 dark pools and as many as 200 internalizers compete with 13 public exchanges in the U.S.

Top internalizers include units of KCG Holdings, Citadel, UBS, and Citigroup. Dark pool operators include Credit Suisse and Morgan Stanley. All of the firms declined to comment, or did not respond to requests for comment for this story.

With the incentive to use the public market eroding, many traders increasingly see exchanges, which are often described as "lit" markets because of the pricing transparency, as battlegrounds for high frequency traders, said Rhodri Pierce, of the CFA institute.

The result is an increasingly splintered market.

"So much of the U.S. equity order flow is in now in the dark, or siphoned off, that it never hits the lit exchanges, and there is just a lot less in the way of trading opportunities," said Mark Gorton, CEO of high frequency trading firm Tower Research Capital LLC.

In an attempt to win back some of the retail orders, exchanges such as IntercontinentalExchange Group's New York Stock Exchange, Nasdaq OMX Group, and BATS Global Markets, have allowed brokerages to place dark pool-style orders on their platforms, with the trade hidden until after it is executed. NYSE, Nasdaq, and BATS declined to comment.

There is no doubt that trading costs on U.S. markets are low, and that retail investors get a better deal than they did two decades ago. But U.S. and global trading transaction costs have actually been rising for the past two years, according to a Credit Suisse report on Feb. 20. That may suggest the benefits from off-exchange trading are no longer accruing to investors as much as they previously did.

TRADE SECRECY

A major concern with off-exchange trading is that brokers who internalize trades and offer dark pools do not provide any data to the market before the trade is executed. On a stock exchange, when an order is sent in, the price of the stock is adjusted and everyone with a data feed sees it.

Dark pools only report data after a trade has occurred. At that stage, information about the trade has little influence on the price.

The pools were originally created for institutions to trade large blocks of stock without creating a large impact in the market. If an order of 1 million shares was tracked, people on the other side of the trade could quickly jack up prices and the original investor could easily pay more than expected.

But much of the trading isn't like that now - the average size of orders in dark pools has shrunk to around 200 shares, similar to levels on public exchanges.

"There are potential costs from this trend of having more and more trading being traded away from exchanges," said the CFA's Pierce.

IEX, a new trading platform heralded in Lewis's "Flash Boys" as a fairer place to trade, aims to become an exchange once it gains more volume, but is currently a dark pool. Its average order size is around 750 shares.

"If the shift becomes too egregious to off-exchange markets, then there are no lit exchanges to price against," said Brad Katsuyama, CEO of IEX. "I don't know if we are necessarily at the imbalance yet," he said.

Pierce released a study on off-exchange trading in November that showed that once more than half of the trading volume in a particular security is done on dark venues, the ability to properly price that security becomes difficult. The price discovery process can begin to erode when off-exchange trading in a security surpasses as little as 10 percent, according to a study by Carole Comerton-Forde and Talis Putnins of the University of Melbourne.

Regulators in Canada and Australia have taken steps to curb the growth of dark trading in recent years by requiring, for instance, that off-exchange trades be of a minimum size or have a significantly better price than can be found on an exchange. Authorities in Europe and Hong Kong are eying similar rules.

"Observing some of the trends in U.S. markets and elsewhere and seeing that dark trading activity in Canada was slowly growing we felt that we wanted to put some very important principles in place," said Wendy Rudd, head of market regulation at the Investment Industry Regulatory Organization of Canada.