The console port is using a simple design of RS 232 standard (https://en.wikipedia.org/wiki/RS-232) , using RX, TX and GND only.
The purpose of the console port is for troubleshooting purposes only, so during reboot of an AP it is assumed that a laptop is connected or the connection is properly terminated in some way, such as being connected into a terminal server.
Leaving long UTP cables un-terminated to the DB9 side will transform the cable into an antenna that will capture anything and provide feedback to the AP - including the signals transmitted by AP during the boot sequence (TX will induce into the RX)
The longer the serial cable is the more energy will be collected - this is why there is a greater chance that an unterminated short cable would work properly while a longer one would most likely not work.
The AP will enter in Boot mode if receives ANY character.