May i suggest you write a 'survival guide to family dinners' 


Good idea, I will think on that! 
I'm going to have to agree with that and vote "invasion".
Going to someone else's house is stressful, too, but not quite as bad, because they're not in -my- space.
I deal with it by avoidance, mostly. I had one sleepover, ever, and it was back back when I as in 5th grade. Never repeated it for a reason. A couple weeks ago a coworker brought up the idea of her coming over for something, and I internally panicked. Thankfully, she changed her mind, and never did come over.
That too. That's one reason why I'm not living in the apartments on-campus anymore. Because there, if someone "official" wanted in, they'd knock, and then just come in (they had a key card that worked for all the rooms).
Once I stalled at that door for a long time, even to the point of holding it shut. What's even more disturbing is knowing that they could come and go when I wasn't home, too. Many times I'd come home and something would be wrong, sometimes I couldn't tell what, but other times it was a light turned on that wasn't on when I left, or something moved, or a mark on the wall or floor that wasn't there before. Sometimes they'd leave notes explaining why they were here (pest control, room check ,etc) but more often than not, there was no note, just a feeling of wrongness and feeling out of control.
On more than one occasion I swore that I'd strip the next time they came in, 'cause my body's not exactly the type you'd want to see naked unexpectedly, lol
Unfortunately, I never got the chance to. 
It's great for us to be able to live in an home of silence and safety, but that's no way for you to have to live all the time. I hope you find a way to work around it somehow so he can tolerate it. If I was NT I would not want to give up the connection you get from being able to have the people you care about in your home.
The other thing, with him thinking he wants socializing and then freaking out, I do it and my husband also does. It's like this stupid denial delusion that your problems won't be there, like you get nostalgic for something you never even had. I would ignore this as wishful thinking, even if you want to respect and believe him. My husband and I are pretty brutal around this and burst any suspect "I want to see them, really" balloons as soon as we see them in each other. It's more humane and it gives the person more time to mentally prepare for just how bad it's going to be.
We are getting to a better place with that. I think now that I am more aware of the issues, I can be more proactive in watching the situations. He is more aware too, and that makes us a better team. 
Same thing on job interviews or meetings: Hate 'em, great at 'em. There's something about the closed-ended structure of these situations that works for me. I know how long it's going to last, so I know how long the performance will take.
Crap, that just reminded me of something. I was at a friends house when I was 16-years-old, and her uncle was there. He was a consultant for corporations that depened on his intuition. My knowledge of how his career worked is vague, but he told me straight-faced that he had this gift and was well paid for it. He came into the house, my friend introduced us, then I remained quiet while the family chatted. After 5 minutes he looked right at me and said, "You should head straight for Central Casting. You've been acting from the moment I laid eyes on you." The only time I had spoken was to say, "Nice to meet you." I just realized he had picked up on me performing my way through an uncomfortable setting. Weird.
(Shhh....I know the 5th doesn't apply to Australian law)
We are doing better now Johanna. I think it is the barbed wire and electric fences, all with signs that say "STAY OUT". KIDDING! 
But I think we have come to a better understanding of it. And we will have many less visitors. The end conclusion has to be worth the effort, you know what I mean?
Well, sometimes I think an island would be better for me. I need very little social interaction. By the time I am done at work for the day, in fact, I would prefer to be all alone with my family.
The problem was more that he was suggesting the visits then freaking out! Total mixed messages. I have established a little more control with things.
Honestly, there are no better people in the world to spend my time with than my husband and daughter.
I have, they are my favorite people EVER! We have a pretty good groove going now, about guests and friends anyway.
Actually, I have posted a lot about these kinds of things. You might try and look at my "threads" rather than just my postings. I have been sharing a lot about this journey we have been on. 

Oh, I could tell you some stories about some rather similar things. Send me a PM!
STOP!
Before you open the door, ask yourself:
Is my name <Roommate's name here>?
Did I knock?
Is the building on fire?
Are aliens invading?
If >2 of these things are true, do not open the door.
(Note that if a random person enters without knocking I will be forced to assume that the building is on fire and aliens are invading.)
I always get those mixed up.