This is just an off the wall thought, but assuming the currently installed MacOS is Catalina and your Mac has a T2 chip, Catalina installs some deep protections that prevent booting other versions of MacOS and the installers do boot a stripped subset of their OS. Apple Support document 208198
describes these protections and how to disable them. NOTE:
This is not
As to the drive format, if I recall correctly, the High Sierra installer optionally reformatted the boot drive to APFS and the Mojave, installers automatically reformat the install drive to APFS if it is an SSD
but APFS was an option for HDs. (I don't know about Catalina, it may require APFS on SSDs, HDs, and Fusion drives because of its new multi-volume architecture. I also seem to recall there was an issue with Case Sensitive
MacOS Extended at one point but I don't recall exactly what the issue was/is.
I don't know what you are experimenting with or what Mac you have, but another possible workaround might be to boot from the Recovery Drive using Shift-Option-⌘-R which will install "Reinstall the macOS that came with your Mac, or the closest version still available