Like all exchange ratios in the market, interest rates are determined by demand and supply. So, to anticipate what would happen, one must conjecture about both sides of the market.
If the Fed quit buying MBS, their prices would fall and yields would increase. Private demand would likely fall because investors would attach a more realistic risk premium to them without the Fed’s support. There would be little incentive to arbitrage funds to other investments and so there interest rates would not be significantly affected.
If the Fed quit buying Treasuries, their prices would fall and yields would increase. Given that private investors did not reassess the credit worthiness of the securities, the difference in interest rates would generate arbitrage opportunities and therefore, change interest rates on other securities. Whether the effect would be large or small depends on the relative size of the Fed’s intervention. The average daily volume of trade in U.S. Treasuries in 2012 was $519 billion.
Of course, the above comments assume that banks do not change their credit creation as their reserves change.