The easiest way to improve quality is to increase the x264 preset slider(in the video tab, same place as the x264 tune option mentionned) to very slow instead of leaving it to medium, the filesize will be a lot lower and the for roughly the same quality(since you set constant quality).
After that, you can the lower the RF setting is set to. The lower that number is, the better it'll look(but the bigger it'll be). I personally like to use 18 there.
I'd also check constant framerate instead of variable since I don't like having variable framerate because it makes some aspect of playback harder if you care about trying to sync it to refresh rates, but that won't change much.
What I would suggest if you're not married to handbrake and are using windows is to use MeguIVit. It was made for exactly your purpose in mind and to be very easy to use.
I repacked it since the original version had issues with windows versions newer than XP and stopped working entirely after a windows patch on some systems. You can find it there:
https://www.akiba-online.com/thread...d-encoder-v1-0-1-1.73866/page-51#post-3167139