Withdrawing money from a pension fund can cost savers up to 10% of their retirement fund, according to Citizens Advice.
The regulator, The Financial Conduct Authority (FCA) has already warned that up to 670,000 consumers face such fees.
In a survey of people withdrawing cash from their pension, Citizens Advice found those with smaller funds were paying proportionately larger charges.
Those with pension pots of less than £20,000 typically paid £1966 in fees.
The regulator is planning to introduce a cap of 1% on such fees, from March 2017.
But Citizens Advice said even that cap would be too high.
It is calling for a standard £50 charge to cover the providers’ administration costs.
“The threat of excessive charges can also put people off making the right pension choices for them,” said Gillian Guy, the chief executive of Citizens Advice.
“A standard £50 exit fee across all types of pensions will mean consumers can make the most of the pension freedoms.”
The survey was based on 500 people who had taken money out of their pensions since April 2015, when the pension freedoms began.