Could you mount the IC upside down? You might be able to bend the pins around so the opposite side is facing outwards.
Do you intend to use a USB joystick interface permanently? Can you give the part number for the USB joystick interface? If it works with old joysticks (from the 1980s and 1990s) then it is not a voltage input circuit; it responds to resistance.
Disregarding the previous points, and assuming that a simple op-amp-based inverter will work, there are still issues. The OP-07 is a nice op-amp but you will need to provide positive and negative supply rails of at least +7V and -2V because its operating input and output voltages don't extend all the way to its negative and positive supply rails. This will require two voltage converters, either inductor-based or switched capacitor. Actually the MAX232 family and its equivalents will produce suitable voltage rails. A single-supply op-amp will not need a negative supply rail but will need a positive rail of about +7V.
If you want to power the circuit from 0V and +5V you will need an op-amp with rail-to-rail input and output capability such as the Maxim MAX9914/9915, see
http://datasheets.maxim-ic.com/en/ds/MAX9914-MAX9917.pdf - it's a pretty impressive device.
Please also include any other information that might be relevant.