Your computation uses discrete units and the range of your computation is large relative to the entire demand curve. If you graph the entire demand curve implied by your calculation, it would have a y-axis intercept at $5 and 0 units and an x-axis intercept at ) dollars and 5 units. So your computation uses half of the demand curve. But the demand curve, technically, is only unit elastic at its mid-point.
If you shifted the demand curve parallel outward and to the right, the disparity in your elasticity computations would decrease. For example the demand curve with the y-intercept at $10 and the x-intercept at 10 units would have equal expenditures at at price of $6 and quantity of 5 and at a price of $5 and a quantity of 6. The elasticity computations would be 6/5 for lowering the price from $6 to $5 and 5/6 for raising it from $5 to $6, which are both closer to one. Technically, a linear demand curve is only unit elastic at its mid-point.
The way to reconcile the two computations when using discrete numbers is to use the average of the two numbers for calculating the base in the percent computation. This is a standard technique for such computations when using discrete numbers. For example, the percent change in quantity in your example would be 2-3 divided by the average of the numbers, that is 2+3 divided by 2. So the percent change in quantity is 0.4 (ignoring the minus sign) and the percent change in price would be 3-2 divided by 3+2/2, which is also 0.4.