Had to delete my old post (to avoid the double post) so I could mark this as unread. My old post read like this:
The interface should have two buttons that say A and B. They'll be among the play, pause, fast forward buttons. You use these to mark the beginning and the end of the section you want to keep.
Skip to the frame you want the section to begin at and mark it with the A button, then skip to the place where you'd like your clip to end and mark it with the B button. Then encode.
You can also use A and B to mark sections you want to cut out (in case there's something in the middle of the sections you'd like to keep that you could do without). In this case, mark the section you want to cut out with the A and B markers. Then go to the EDIT menu and select DELETE. That should take the section you selected out.
The Avidemux website does have a pretty decent wiki in case some things aren't clear.
Scratch what I said about their Wiki. After re-reading through it, it leaves a lot to be desired. And even their forum isn't very helpful. I'll admit here that I only use the software for simple trims.
As far as the file you have, when you save the vid in Avidemux its been my experience that you actually have to type in the file extension when saving the vid. So, If i wanted to save an AVI file as "RioHamasaki" I'd have to call it "RioHamasaki.avi". Make sure the format you're saving it to is correct with the format specified on the menus on the left of the GUI.
I'm sorry if the prog is giving you trouble. Its admittedly not the most user friendly program out there, but it does work for the purposes you're looking for. I'll start giving my responses a little more thought when recommending prog from now on, instead of just throwing out GNU progs that are sometimes more trouble than their worth.
As long as you're not looking to change the video or audio format, you should be able to just trim the parts you want and save it. But name the file with the same extension as the source file. So, if your working with an avi file, save it as one, too.
The other menus (video and audio with configure options) on the left are for if someone wants to re-encode the video or audio into a different format than the source. So if you're just looking to cut parts out, you shouldn't have to change anything there.
As far as virtualdub being any better, you'd have to play around with it to see.