If you are getting clean glitch-free audio at 512 samples then changing to 256 would be the equivalent of the pipes sounding about a meter closer to the listener. In other words, any audible latency would be coming from some other part of the chain between the key press to the audio being heard.
I've heard audible latency when using Reaper without arming the main HW track for recording. Never figured out why. I've heard audible latency with daisy-chained keyboard encoders. Also, excess MIDI messages such as might be generated by a dirty potentiometer or intermittent contacts (keys, pistons, stops) could overwhelm a conventional 5-pin MIDI circuit and cause buffering delays. Anything overloading the CPU as monitored from the HW Audio Meter?
All latency from encoder debouncing, excess buffering, MIDI regeneration (MIDI thru), PC delays, sample size, sample set delays, and speaker distance will add together to an overall latency. Somewhere around 30 to 50ms would be my guess as to what becomes audible. I've seen debouncing alone in some Arduino and Teensy sketches that border on audible with no other delays! They do have improved debouncing librarys now I think.