If there is a noticeable build up then the callouses are too big. Lots of good suggestions about how to trim them down. I use a single edged razor blade, carefully, as needed.
If the callouses are toward your palm rather than on your fingers the I'd say you should work on your grip. Let the handle rest at the first knuckle of your finger. You'll still get callouses but you are less likely to rip your palm.
I've also found that gripping a cold water bottle immediately before skiing can alleviate some hand pain, though not a substitute for managing your callouses wherever they are.