This is just an off the wall thought, but assuming the currently installed MacOS is Catalina and your Mac has a T2 chip, Catalina installs some deep protections that prevent booting other versions of MacOS and the installers do boot a stripped subset of their OS.
Apple Support document 208198 describes these protections and how to disable them.
NOTE: This is
not SIP.
As to the drive format, if I recall correctly, the High Sierra installer optionally reformatted the boot drive to APFS and the Mojave, installers automatically reformat the install drive to APFS
if it is an SSD but APFS was an option for HDs. (I don't know about Catalina, it may require APFS on SSDs, HDs, and Fusion drives because of its new multi-volume architecture. I also seem to recall there was an issue with
Case Sensitive MacOS Extended at one point but I don't recall exactly what the issue was/is.
I don't know what you are experimenting with or what Mac you have, but another possible workaround might be to boot from the Recovery Drive using Shift-Option-⌘-R which will install "
Reinstall the macOS that came with your Mac, or the closest version still available.
