Some $506 million has been returned to victims of Bernard Madoff's Ponzi scheme, the largest in history, according to a Monday announcement by the office of trustee Irving Picard and the Securities Investor Protection Corp.

That means that a total of $5.44 billion has been returned to victims of Madoff.

But more than four years after Madoff's conviction, more than two-thirds of the stolen money is either unaccounted for, or is yet to be returned to burned investors.

Investigators say that $17.5 billion was lost to Madoff's scam, which crumbled with his arrest in December 2008. Since that time, investigators have recovered assets -- including property, jewelry and a yacht -- and reached settlements totaling some $9.32 billion.

Related: Madoff ruined thousands, including his son

Many of the victims are still awaiting compensation. The trustee said that 1,106 accounts have been fully compensated, out of a total of 2,178 victims.

Meanwhile, Ponzi mastermind Madoff languishes in a federal prison in Butner, N.C. He pleaded guilty in March 2009 to running a massive scam and was sentenced to 150 years.

His brother Peter recently began a 10-year sentence for helping to cover up the scheme.