When I get criticism about milsim, its usually "you take the game too seriously" or "you're too hardcore". These criticisms usually occur when we win a game. When we lose a game there are no criticisms, just gloating. Generally the anti-milsim sentiment keeps these two camps separated. As zapplez said, it would be a tough sell.
...plus I really don't know how people would feel about being cycled into a field where a bunch of milsimers have had time to consolidate a position and strategy. Actually I do know, and it would kinda look like uh, rape. Thats not being elitist, its just stating a fact - we practice at that kind of stuff, it would be unfair. When you march milsimers into positions prepared by other milsimers, its usually rape, except milsimers tend to appreciate a good ambush whereas non-milsimers are usually freaked out by it.
Might work better if you reversed it though, put the skirmishers out and have the milsimers cycle patrols in... that would give the skirmishers a chance to prepare positions and take static prepared positions advantage away from milsimers. This more resembles my 'Op Holy Vengence' game two years ago, Iraqi's versus SAS. That worked pretty good but it wasn't milsim by any real stretch. It was more a scavenger hunt chess game.
Either way, milsim appears to be a dirty word these days hence why the two camps generally don't mix.