Your amplifier is loud enough at nine o'clock on the dial, the usable range is a sliver, and down at the bottom one channel is quieter than the other. That is not a fault — it is a volume pot being asked to work at the very bottom of its travel, where its two halves stop tracking each other.
The fix is to throw away some level before the pot has to, so you can turn it up into the range where it behaves. This works out how much, and what to build.
The obvious one is a resistor in series with one leg of the cable. It is one component and it works — but it adds its own value straight onto the amplifier's output impedance. Dropping a 32Ω in-ear by 12 dB needs about 95Ω in series, which turns a 1Ω output into a 96Ω one. For a headphone with flat impedance that is merely a level change. For a multi-driver armature in-ear it is a tone control you did not ask for, and the interaction simulator will show you how big.
The better one is an L-pad: a resistor in series and a second one across the headphone. It needs two components and a moment with a soldering iron, but the output impedance the headphone sees is now set by the shunt leg, so it can be made lower than the amplifier's own. This is exactly what an iFi iEMatch is, and it is why calling that device an “impedance adapter” is misleading — a plain adapter raises output impedance, an L-pad does not have to.
Attenuation placed after the amplifier drops the amplifier's own hiss by exactly the same number of decibels, because signal and noise pass through the same divider. That is why a pad fixes hiss with sensitive in-ears when a digital volume cut does not: turning the source down quietens the music but leaves the amplifier's noise floor exactly where it was.
A gain switch does the same job with no soldering, if you have one. Going from a 6× setting to unity is 15.6 dB, which is usually more than enough. It works by moving the pot's operating point, not by changing anything about the imbalance itself.
And if you would rather solve it in software: cutting digitally is free and reversible, but check what it costs in bit depth first — on a 24-bit path the answer is usually nothing at all.