System Preferences→Startup Disk saves your preferred boot volume in the computer's PRAM (aka NVRAM).

If it's not remembering your setting, either the PRAM is bad (possibly a weak battery?) or the external drive is taking so long to spin up that the boot code is giving up and settling for the next best option.

If the volume remembered in PRAM does not become available in a reasonable time, or there is no PRAM preference, it takes the first bootable partition on the first drive. (First by device number, not necessarily the first to spin up. Generally, that means the internal drive.)

From a cold start, the internal drive also has to spin up, and that may be slowing down the process long enough for the external to be seen in time.

Try resetting whichever of PRAM, NVRAM, or SMC is relevant to your machine, then once again select the boot volume in System Preferences→Startup Disk. If your machine has a PRAM/NVRAM battery, check it.

Note that option-boot does not alter the PRAM setting. It's used to temporarily override PRAM without affecting subsequent boots.

To learn which volume is currently selected as the boot volume, run the command:

bless --info --getboot

The output will be something like /dev/disk0s4. To turn that into a meaningful name, feed it into diskutil, as in:

diskutil info $(bless --info --getboot)