OBS Settings From Fresh Install
#1 - Create Scene in OBS
Create a "Source" in OBS, choose "Display Capture" to record whole screen/desktop or "Game Capture" to only record the TO.
#2 - Set Video Resolution
Go to Settings -> Video:
Set Base (Canvas) Resolution to 1920x1080 (Or 2560x1440 for better quality / have fast pc)
Set Output (Scaled) Resolution to 1920x1080 (Or 2560x1440 for better quality / have fast pc)
Set Common FPS Values to 60
#3 - Microphone Settings
Go to Settings -> Audio:Disable Mic/Auxiliary Audio if want to disable Microphone during recording. This might depend what type of "Scene" you use to record the game if its recorded or not)
#4 - Audio/Video Quality
Go to Settings -> Output:
Set Audio Bitrate to 256 Kbps
Set Video Bitrate to higher value. This is linked to both the "Video Encoder" you use and the "Encoder Preset" to determine final video size / quality.
#5 - Hotkeys
Go to Settings -> Hotkeys:Set a "Start Recording" and "Stop Recording" hotkey, i used Numpad Plus/Minus in the past... make sure its not used ingame by TO for something.
#6 - Fix Scene Size
Make sure Video Canvas is filled when using lower game resolution than the recording resolution.
Left click on the canvas (recording window in OBS), then right click on the "TO Scene" and go to Transform -> Stretch To Screen.
#7 - Fix Video Brightness non-v469 TO
When not using v469c TO you must add a "Color Correction Filter" to your "Scene" with around 0.45-0.50 Gamma as OBS cant record TO's "Brightness".
Video Bitrate, Encoder Preset & Video Quality
CPU Encoder tends to give better quality at lower bitrate/filesize if your CPU is fast enough and heavier preset is used. With an AMD/Nvidia gpu u can also try its video encoder, it will have different "Encoder Presets/Qualities".
Youtube will heavily processes an uploaded video, the quality can be allot worse than viewing with local video player. For maximum quality you'll have to test heavier presets and see if your pc can still handle it can give a smooth video file.
On a slower system its also possible to just record 1024x768 in OBS and later on upscale that with shotcut/vidcoder/handbrake to 1080p/1440p, Youtube processes videos in 1440p in much higher bitrate than 1080p.
When doing test recording make sure to rapidly move mouse to give allot of screen movement. Sometimes recording can look smooth during simple test recording but end up stuttery on real game with rapid mouse movements.
Better Upscaling in OBS
To squeeze out even more quality when recording in lower game resolution to higher video resolution use upscale filters. Click on the Video Canvas and try the "Scale Filtering" settings for allot better upscaling in OBS. (But can be heavy on performance)
TO Sound Settings
The audio can sound glitchy if you have "Surround Sound" or "3D Surround" enabled in the ingame TO audio settings, restart TO for changes of these settings to take effect.
TO Video Settings
For maximum video quality you could set Anti Aliasing to 8x and make sure Anisotropic Filtering is set to 16x, read how in the AA & AF guide.
Or go even further and force 8x Super Sampling from your graphics driver settings...