I have doubts that the filter will fare well against any sort of BBs at any velocity / distance. But I keep mine on all the time, simply because at least the impact energy will be dissipated by the filter and not directly on the front element of my $1500 lens.
Over the years I've been shot at way more as a camera man than a player. So far my PowerShot S50 has taken some hits (with BB dents to the body). Luckily never on the lens. But with my DSLR it's a bit different, the lens is just so much bigger of a target. I had a close call before (someone thought my lens was the scope of a sniper rifle). But the BB missed and hit me in the goggles instead.
Best bet is to act defensively when you're on the field with the camera, be ready to dodge, roll, prone, etc. That is, if you decide to take the camera directly into action and not standing on the sidelines.
I think I'm also lucky in the fact that local players have come to know me as the camera whore. I'm always on the field taking pictures. Make sure other players know you'll be on the field with a camera, that'll definitely work to your advantage.
Edit: You might not want to use your polarizing filter, just go with a basic UV filter, since they cost way less. Or you can custom make a polycarb / mesh lens protector - but that would definitely degrade your picture quality.