I never use abrasives on the rims - I reckon the brake pads will remove aluminium quickly enough as it is.
If there are black deposits or traces of oil on the rims, I use a rag with some solvent (it used to be trichorethane, but is now something else less toxic, sold as "trichorethane replacement"), then wipe again with a clean dry rag. That generally gets rid of the deposits, and sometimes cures brake squeal.
If the brake pads are very dirty, worn in an odd shape, or have embedded grit/metal particles I file them smooth. If still on the bike (possible with cantilever and V-brakes) I do this with a file, if removed from the bike (calliper brakes) I use a clean part of the concrete floor in my garage.
