I was curious and had a 128GB Lexar Flash Drive sitting around from a failed experiment and found
- if the flash drive were formatted Mac OS Extended (HFS+) with a GUID partition table I could remove it from the Time Machine exclusion list.
- the same flash drive formatted ExFAT with a Master Boot Record partition table could NOT be removed from the Time Machine exclusion list.
Since MBR and ExFAT are the
de-facto standards for high capacity thumb drives and SD cards and my results verified what you are seeing, I didn't attempt to test all the possible permutations of partition maps and format options, but I would venture the key is whether or not the drive is formatted Mac OS Extended. Given Time Machine's reliance, or should I say dependence, on the HFS+ volume structures that would not be surprising.
In the name of full disclosure, I should have said I am running Yosemite (OS X 10.10.2) on a Late 2012 Mac mini (Macmini6,2).