If colonialism has made those countries poor, then they should have gotten rich once they were no longer part of a colonial empire. At the same time countries which had large colonial empires should have gotten poor, when loosing their empire.

What we mostly see is that this is not the case. Portugal got rich after its empire collapsed. Spain was about as rich as its former colonies for a long time. France and the UK did not collapse after loosing their colonies. There are rich countries, which never did have many colonies or only small ones for a limited time, like Germany, Scandinavia or Switzerland. You also have Oman, which did not get rich despite having had colonies. We also have Africa, which only has Botswana as a country genuinly benefiting from no longer being a colony. However that was after diamonds were found inside Botswana shortly after independence. Funnily enough Botswana also asked to be a colony. Everybody else more or less failed to get rich.

That is not to say that colonial empires should not pay for crimes they comitted or return stolen artifacts. The benefits of colonialism were mostly going to a small elite in the colonial countries and cost the states a lot of resources, which in many cases would have been better spend on other projects.