I never switch off, never have and never will. As long as you don't remove it while the warning message about switching off is on screen, you're fine. In the teaching studio, I would insert and remove sticks from PSR , Genos and Tyros keyboards multiple times a day, often multiple times in a 1/2 hour lesson. None of my students do so, regardless of the make or model of keyboard, piano, synth or organ. Not one has ever had a problem due to not switching off.
I often don't worry about 'ejecting' my USB sticks, DS cards, phones and tablet on my PC either. Either I've been exceedingly lucky since USB sticks first arrived on the scene or it's just Microsoft covering their ...... !
Just follow the simple rule, don't remove while you're writing to the stick.