In general, American exports in 1812-15 were southern products on northern vessels. Therefore, southerners did have skin in the game.
Madison’s war aims were an end to impressment and a favorable trade treaty with the British. The latter obviously would benefit southerners.
Don’t overlook the strong anti-British sentiment then current in the South. Napoleon had few friends there, but anti-British feeling associated with the American and French Revolutions had not died out.