So I found the problem to all of this! The culprit was the anti-reversal latch!
Since I was a new to opening the gearbox, I pulled up a video on youtube on its disassembly. I followed the video and when it came time to re-assembly I was pretty ok with putting it back together. There weren't many reassembly videos from open gearbox to closed gearbox that I remember where they explained how to fix this.
The problem was that when reassembling the gears, when I was putting the anti-reversal latch back in I wasn't compressing the spring to ensure it was past 45 degrees, rather it was sitting more at 20 degress, thus not even sitting properly within the gearbox. This was also causing a gearbox seizure even in semi.
I corrected this, and the first time putting it back together it went in fine with NO resistance on the selector switch.
I now have a working M4.