I have replaced three joysticks on various Korg arrangers, including PA4X. Why? Because if you carry it in a bag and happen to bang the bag against something you can break the joystick mechanism because it protrudes above the keyboard. It took me three times to figure out what was breaking it.
The solution is to put foam rubber around and over the joystick, cutting a hole in the foam to fit the stick. Or carry it in a hard shell case.
It isn't too hard to do on Korg except for taking out about a hundred screws. The only Yamahas I've had with joysticks are the SX900 and Genos, and I didn't keep them long enough to do any damage, so I don't know if this is even relevant.
Anyway, if it's under warranty, send it back for sure!