Annoying or not, I think this is the way it's supposed to work. When you are in "Find mode", all instances of your search are found and highlighted. You can then use the Find window to move through all of the occurrences to see and/or find the one you want.

At least in Safari 12.x, if you click anywhere on the web page you get out of Find mode and then command-down arrow works.

On a Mac since 1984.
Currently: 24" M1 iMac, M2 Pro Mac mini with 27" BenQ monitor, M2 Macbook Air, MacOS 14.x; iPhones, iPods (yes, still) and iPads.