In v320 stacking, If one physical link of a logical link goes down then the switch should declare that stacking link as down, so this is an expected behavior.
When removing one physical link of each logical link, then both stacking links are declared down and you will lose connectivity between the two slots.
This causes the backup-slot to take over mastership as the switch cannot operate without a master.
Upon reconnection of the links you will allow commuincation again, as with both physical links up the logical link is declared active, and the switches will notice that both slots are master. As two masters in a stack is an invalid state, one of the units has to reboot to clear this situation.
After the reboot you should have a regular master-backup scenario again.