Q: Is there a way to tell if engine oil was placed in the transmission by mistake?

Hello. Technically, yes, it can be proven by chemical analysis. This would most likely have to be a forensic investigation so obviously you would have to send a sample of the fluid to a laboratory that has a capability in oil analysis but specifically forensic chemical analysis. So, it is "possible" but a little involved.

As the engine and transmission are laid out it would seem to me unlikely that engine oil could be introduced into the transmission filler. If you are concerned that the transmission fluid might be contaminated though, you could try draining the fluid as thoroughly as you can and refilling it. Repeated drains and re-fills would reduce the level of contaminants to a de minimus level.

With regard to the overheating issue, there are quite a few causes: low coolant, defective head gasket, plugged radiator, faulty radiator fan(s), engine timing off, bad thermostat, and so forth. If you’d like, a technician from YourMechanic could diagnose the overheating issues you are having with the car and give you an estimate of what it will cost to make it functional and reliable for you. Hope this helps.