You should be able to "back off" the grips a mm or two without affecting the ability to put the stock or aftermarket bar ends on.
- Woodstock