If you experience this issue there is a possibility your controller has thumbstick drift. You may need to clean the thumbstick area with electronic contact cleaner or blow out any debris with compressed air.
You may also adjust the thumbstick center dead zone, as well as the thumbstick range from the 'Experimental Features' section within the settings panel of your device. We also have a move and rotate dead zone adjustment in the 'Controls' section in the Pause Menu that you can also try.
If the solutions above do not work it would be best to troubleshoot your issue and contact Meta Support directly. They may have options for your situation and may be able to fix or replace your controller.