The differences can be tremendous.
Typically, headphones are fed from the power amplifier sections by using a resistor voltage divider, with a typically 220 ... 270 Ohm series resistor. Thus, the output impedance is set by the said resistor. If you have a typical modern headphone, its own input impedance will be anything from 30 to 100 Ohms. Since the output impedance fails to meet the basic principle of signal transfer, that the output impedance of the source must be at least 10 time smaller than the input impedance of a device, you receive a faulty signal.
For reasons I am completely dumbfounded with, somebody has decided that headphones do not behave as loudspeakers do, and thus the usual rules do not apply to them. This is at absolute odds with the entire field of electronics, but hey, it's cheap'n'cheerful.
Assuming a correct design, a dedicated headphone amplifier should really open your eyes to what headphones are all about. You should end up hearing a much superior sound, with more life and air, with a more natural and convincing sound stage, and with the entire bandwidth present in correct form.
The idea behind them is to provide such a source as to make the headphones themselves the ultimate limiting factor in terms of music quality. This obviously implies that going for a costly ultra high quality headphones amp makes no sense if you listen to them only occasionally, or if you have some "mere mortal" cans. As a rough guide, cans costing say 200 quid and upwards will make a dedicated amp worthwhile, below that you would be wasting your time and money.
But still only if you spend enough time with them.
