YouTube or any flash video player website not opening full seems to be recurring bug in the Firefox browser. This has been addressed in many bug reports of the Firefox in the past. It has multiple set of solutions, so depending on the one that is applicable for your build version of Firefox, one of the below fix may work.
1. Page Permissions : In case of Firefox core, it requires website to manually configure before it can use fullscreen settings. You can configure permissions by following these steps.
a. In your address bar, right click on the "youtube.com" domain.
b. From menu, select "page info".
c. Select "permissions" tab and find "fullscreen" property and set it to allow or default value.
This should resolve the issue in this case.
2. Disable Flash hardware Acceleration
1. Go to the Adobe Flash Player Page here : http://www.adobe.com/products/flash/about/
2. Right click on the flash player on the page.
3. Click on the settings icon or link on the dialog box.
4. Uncheck the checkbox from "Enable hardware acceleration. ".
This should solve the issue if it is related to the hardware acceleration for flash content.
There are few other fixes but most of them are patched before Firefox Build version 48.
If none of the above methods work then do try the HTML5 player of youtube. You can check that here : http://www.youtube.com/html5
See if that also leads to solution to the full screen issue. If it does then I suggest switching browser to chrome or opera.
