There are three different factors, some of you are confusing the issues. The file extension is the most obvious factor you can see, it reflects the "container" format. Unless you are married to certain proprietary player (especially older brand name hardware) you can do what I do: ignore the container format. Earlier I found that some 1080p anime MKV has problem playing on my Android (ARM) set-top box, but recently I have been using a mini-PC with a more powerful CPU (still the low-end for a PC) it's no longer an issue.
The second factor is most important, the codec. Especially the video codec. These are the mpeg4, xvid, vidx, qt, h.264 etc. A good codec can deliver excellent quality video in a tiny file size. But it may be more demanding of the playback hardware (and certainly for the encoding hardware). However, I still try not to obsess over the codec as I am just a humble consumer, so for movies I go with the most reputed release groups such as yify, for anime, HorribleSubs etc. And for AV... sadly there's literally no choice except sometimes able to avoid the yellow watermarks. If you care the screen caps often list the tech details such as resolution, FR, codec and bitrates in the header.
The third factor is the parameters chosen during encoding. The most obvious choice is the resolution (480p, 720p, 1080p, and infinite number of smaller or in-between sizes) and bitrate (determine the file size). But there are other factors which I am no expert myself. I just think of it this way: having chosen the container, codec and target size it's still easy to mess it up with bad parameters. A knowledgeable person can make a good file with a certain target filesize. A very skillful, knowledgeable and careful person can optimize the parameters to the amount of colors, details, action, types of scenes etc to squeeze that last extra bits of quality to make it like 10-20% better for the very sharp-eyed viewers for a certain target file size.
Mostly I found that 1080p for action/scifi movies are worth it. But often not the case for anime (720p is fine) and AV (over 480p is often wasted). The (container) file format is very insignificant as a factor.
A happy exception is HD uncensored vids, some of them have such amazing quality (e.g. 1Pondo-092415-159 - Aoi Shirosaki) it really blows even the best SD censored vid out of the water.