Here is the Bureau of Labor Statistics data on textile workers:
If you go back to 1990, the level of employment in textiles was around 500,000. It started to decline in 1995. It has fallen steadily since then until 2009 when it leveled off.
One aspect of the explanation for this is that as the Chinese were allowed to enter into the international division of labor through market oriented reforms there was a huge cost differences between production of textiles there and here (with our protected companies and unionized labor force). By purchasing textiles in China and selling them in America, Wal-Mart and other retailers set in motion a process of reallocating resources and capital investment to take advantage of the greater efficiency of production in China. It appears that the adjustment was complete by 2009. Since then, as you suggest, the cost structures seem to be the same as no more downsizing of our textile production has occurred.
Of course, a complete explanation would have to look at the details of the history of the industry over the last twenty years.