Current is drawn from a power supply. The 650 mA current rating of your 12V charger is the maximum current it is designed to be able to supply. If the circuit or device that you are powering only draws 45~100 mA, then only 45~100 mA will flow. This is way under the limit for that power supply, so it will not be damaged.
If the power supply is poorly regulated, and your circuit is not drawing much current from it, you may find that the output voltage is higher than 12V. It might be 15V or even as high as 18V. This might possibly damage the thing you are powering.
Give us some more information and we may be able to be more specific. Do you have a manufacturer, model number, or photograph of the power supply? The device you want to power?
You need a power supply rated at 12V and about 30mA or more. You would be best advised (by Kris above, incidentally) to measure the voltage in case it is substantially higher than 12V when unloaded. a 20mA load on a 650mA power supply may be insufficient to bring the voltage into range. If so, get a better power supply.