Annoying or not, I think this is the way it's supposed to work. When you are in "Find mode", all instances of your search are found and highlighted. You can then use the Find window to move through all of the occurrences to see and/or find the one you want.
At least in Safari 12.x, if you click anywhere on the web page you get out of Find mode and then command-down arrow works.
On a Mac since 1984.
Currently: 27" iMacs, Macbook Air, macOS 10.14.x,; iPhones, iPods and iPads galore!