@nrgmonkey, I hope you're still paying attention to this thread after 2 months. I don't know how I missed seeing this on Aug 28th or shortly thereafter, and I apologize for not responding sooner.
You said, "...
when the alarm is set on instant, and that door is opened, nothing happens." When you say it like that, it sounds like the system is behaving normally, i.e., operating as it's designed to.
One of the most common novice errors is to arm the system in Instant, thinking the Exit Delay has been eliminated. But arming in Instant eliminates the ENTRY Delay, not the Exit Delay. So if you Arm in Instant and immediately open a Delay door, it won't trip the alarm: It's designed that way, and it prevents a lot of User-error False Alarms, which was one of the alarm industry's biggest problems at the time that protocol was instigated.
The Entry Delay on your model Control Panel (the Vista-20P) is programmed at the time of installation, and is typically set at 30 seconds; but keep in mind that alarm systems are all, by their nature, custom installations and yours might be more. IIRC, 30 seconds is the minimum ENTRY Delay. Your system may have a longer Entry Delay and you may need to time it to be sure.
The Exit Delay time is set in firmware to be 15 seconds longer than the Entry Delay. That can't be changed in system programming. So if your Entry Delay is 30 seconds, then your Exit Delay is 45 seconds. So if you "tested" your door-in-question within 45 seconds (or longer, depending on Entry Delay set in programming), then opening that door should not cause an alarm. If you then moved on to test another door, but the Exit Delay ran out while you were fiddling with the door-in-question, the other doors would probably cause an alarm. All of which is normal operation for the system.
Personally I recommend Users test a door by setting the Chime function (CODE + 6 to toggle on/off) and opening/closing. Testing by generating an alarm leads to errors in testing like the one I described. If the door causes a Chime, then it will cause an alarm when system is Armed. But that's just my recommendation. After 2 months you may have already discovered that the door is working normally, but please post back and let us know the outcome.
I don't like to bad-mouth devices, but the switch you show in your picture is really old---it was pretty much obsolete long before 2004. I strongly suspect your Vista-20P Panel replaced an earlier panel of an older system, and the original switches were kept. Those switches were not hermetically sealed, unlike the replacement you bought, and some of those old switches caused a lot of false alarms after a few years in use. I'd recommend replacing all switches of that vintage in your house.
Another thing: DON'T ever paint, shellac or varnish etc over those plunger-type switches: Any coating will usually cause them to stick in the Closed position and the Control Panel won't see the door open. If any of your plunger-type switches have been painted, I recommend replacing them.
@lockman: You raise an interesting question about burglary sensors in alarm systems in North America: The alarm industry's jargon (Alarm-Speak), uses different terminology than standard Electronics-Speak: In the alarm industry, "Normal" status of switches refers to the state of the switch when a door or window is secure (closed). In that state, the switch is also closed, and is referred to as a Normally Closed switch, and those switches are called and labeled "Normally Closed" (NC), although standard Electronic nomenclature is Normally Open (NO) for that same switch. I happened to be in the industry in the mid-to-late 70s when different manufacturers were still going back-and-forth on whether to use standard electronic nomenclature, but by the early 80s, everybody had deferred to the ignorance of most alarm technicians, and called standard NO (per standard electronic nomemclature) switches "NC" based on the condition of a "happy" protective loop, i.e., all doors/windows closed.
None of which pertains to the OP's original question, but just for future reference.