Around 7000 prisoners, many of them sub-Saharan African mercenaries hired by Gaddafi, are still being held in detention across Libya, the UN says.
The men are held without access to legal services as courts are not yet fully working in the country.
"While the (National Transitional Council) has taken some steps toward transferring responsibility for the detainees from brigades to proper state authorities, much remains to be done to regularize detention, prevent abuse and bring about the release of those whose detention should not be prolonged," the report says.
Mr Ban said: "I believe that the leaders of the new Libya are indeed committed to building a society based on the respect for human rights."
"Achieving this requires the earliest possible action, however difficult the circumstances, to end arbitrary detention and prevent abuses and discrimination, against third country nationals as well as against any group of Libya's own citizens," he added.
