The word มด in Thai, so far as most people know (including Thais), refers to the little insect with a sweet tooth that always ruins picnics. Or, to put it a bit more scientifically, insects of the family Formicidae. Ants!
And yet, there's this word มด that we find most commonly in the words แม่มด and พ่อมด, or "witch" and "wizard." They appear to literally mean "mother ant" and "father ant," but intuition leads us to reject this as a viable interpretation.
Aj. Chamnong Thongprasert (จำนงค์ ทองประเสริฐ), in an installment of his newspaper column ภาษาไทยไขขาน (Thai Unlocked) from the mid-80s (available on the Royal Institute website), discusses one possible origin.
The late Banchob Bandhumedha (บรรจบ พันธุเมธา) told him that Khmer has the word
One of my problems with Aj. Chamnong's theory is that he's making a leap between แม่มด and หมอผี on semantic grounds, instead of etymological grounds. We can translate หมอผี as "witch doctor," meaning a person who heals diseases by manipulating spirits. Nowadays, with the advent of modern medicine, it's often used to mean an exorcist, someone who casts out spirits, the logic being that in both cases the person has the power to control the spirits inhabiting a human. But if แม่มด means แม่มตะ, then the grammar seems off. In แม่+X constructions, X is usually a verb indicating what the female person does (e.g. แม่ค้า, a woman who sells, hence a female shopkeeper) or a noun indicating the person's domain (e.g. แม่บ้าน or แม่ครัว). มตะ just means dead, or to die, so in this case it means what, a female dead person? I'm not buying it.
So for me, Aj. Chamnong's explanation doesn't satisfactorily explain why the phrase is แม่มด and not something more like, say, หมอมด. I did some more looking around. It turns out that Khmer also has the phrase មេមត /mee mʊət/, which corresponds exactly to แม่มด, which he doesn't mention (or doesn't know). Looking on SEAlang's Khmer dictionary, it gives this definition of
This would seem to discredit the theory of มด being related to Pali, because this word is apparently unrelated to death and dying.
SEAlang's definition for មេមត់ /mee mʊət/:
"medium, one who can contact the spirit world; witch"
Now, I have to make the disclaimer that I don't know Khmer, but this would appear to be a good source for the Thai phrase. It would mean someone with secret understanding, who knows things ahead of time, i.e. can commune with spirits. Note that medium is a different thing than a หมอผี, too, who heals by controlling the spirits.
I think the case is good that มด comes from Khmer, because it is rarely seen outside the fixed phrases แม่มด and พ่อมด. One exception is มดหมอ, an elaborate version of หมอ. But มดหมอ doesn't mean witch doctor, it just means doctor. As is common with such elaborate nouns, we see มด and หมอ used as an "elaborate pair" in phrases like หามดหาหมอ "see the doctor," which is just an elaborate version of หาหมอ. But outside of this association with witches and doctors, มด is meaningless in Thai, but not so for Khmer. Loanwords tend to have narrower meanings than their source language, which appears to be the case here. There are several other phrases in the SEAlang Khmer dictionary with មត់.
But whether it is a native Khmer word or not, someone else with more knowledge on that subject will have to help me out on. Given Aj. Chamnong's explanation, though, I find the connection to Pali misguided. I welcome further evidence or discussion.