As a current UX VR Designer at Samsung I say yes, you should definitely pursue learning how to design for Virtual Reality. Currently there are few designers in the field and there is a lot of room for impact. From the 7+ months I have been in the field here are some suggestions:

1) Start testing out VR applications (note that the type of app you design for Gear VR is different from the Rift or Vive that have positional tracking). Some good GearVR apps: Jaunt, Oculus Apps (Video, Arcade, Social Trivia, Photos, etc.), Land's End, Samsung Internet, YouVisit VR (their old UI was better), Within (formerly Vrse) Good Vive Apps: Tilt Brush, Pool Nation VR, Job Simulator. For the Rift I recommend watching the demo reel to see the full capacity of the HMD.

2) Learn Unity. Unity seems to be the standard for VR. You don't need to learn how to script, but know how to place objects, create UI elements, and deploy to your HMD. Unity has the roll-a-ball tutorial on their website which is a good start. You can also find some good classes on Udemy.

3) Learn Maya. This is a more long term one. But to create a great VR experience you need to be able to build environments and objects.

If you're interested in more the Oculus design team has some good videos on YouTube from Oculus Connect and so does the Google Daydream team from Google IO 2016.

You can also consider going to the Oculus Connect conference in October which is targeted to designers and developers.