Mostly automatic encoding/preparing for posting process

[SamKook]

I started to encode a lot more videos than before lately which started to take too much of my time since I didn't want to compromise on the quality of my post.
To remediate this problem, I wrote a bunch of .bat files to automate as much of my posting process as possible while keeping the result exactly as I wanted it.

Since it might encourage other people to make better quality encodes, I decided to share it and explain enough of it so it can be modified if others needs are different than mine.
Installing all of the necessary software might be a little intimidating since there are many and some of them can be a bit hard to setup properly.
Once it's done though, it can be used by pretty much anyone with very little learning needed(which I'll explain here).

A word of warning before you start to use it, it demands a pretty good computer at the default settings so if you planned on using it as is, make sure your computer is up to the task.
I have an i7-2600k @ 4.7ghz(hyperthreading on) and it takes me around 18-19 hours to encode the 10-bit mkv and 8-9 for the avi.

I'll list all the software used first and explain how to find and properly install some of them after for those that need it.
For those of you with a 64bit OS, you can use a 64bit version of the software that have "(64bit)" after them(and only those, unless you know exactly what you're doing).
For those that only have a 32bit OS, just get the 32bit version and it should work without any problems(I haven't tested it though).


*Required Software*
-Avisynth(with a bunch of extra filters)
-x264(64bit)
-Winrar(64bit)
-MediaInfo(64bit)
-avs2pipemod
-neroAACcodec
-mkvtoolnix
-VirtualDub
-DVDDecrypter
-MPEG4ModifierCL
-DGmpgDec

**Optional(but recommended) software**
-par2cmdline(64bit)
-timer(by Greg Wittmeyer)
-Gimp
-md5deep(64bit)
-PSTools
-movie thumbnailer
-pure-sfv
-putty
-FlashFXP(not free) or WinSCP(free)
-Daemon Tools lite
-cURL
-yEncBin Poster

**Installing the software**

The most important(and hardest) software to install is avisynth and its filters. It's the heart of the encoding process and there isn't a 1 click solution to install everything that is needed.
-The first thing to do is install the official version 2.5.8 which can be found here(get the .exe): http://sourceforge.net/projects/avisynth2/files/AviSynth 2.5/AviSynth 2.5.8/
-Next, you need to install the unofficial version 2.6 MT(Multithreaded) by SEt which can be found here(as well as the instructions on how to install it): http://forum.doom9.org/showthread.php?t=148782
-For the filters, everything goes into the plugins folder where you installed avisynth 2.5.8(unless stated otherwise) which is most likely C:\Program Files (x86)\AviSynth 2.5\plugins (remove the (x86) on 32bit OS) and they all need to be the 32bit(x86) versions or they won't work.
The source for the plugins is often included in the downloads, but you don't actually need it so there's no need to extract it.
-First, install QTGMC(an HQ deinterlacer since most JAV is interlaced) and its modded plugins package which can both be found here: http://forum.doom9.org/showthread.php?t=156028
Put QTGMC .avsi in the avisynth plugins folder and for the plugins pack, read the "Where_to_put_these.txt" file inside the first 2 folders to know where to put the files.
-Next you need the SmoothAdjust plugin which can be found here: http://forum.doom9.org/showthread.php?t=154971
Read the INSTALLATION part of the readme to know which version of the dll is best suited for your computer and put it in the avisynth folder.
-You also need the autocrop plugin which can be found here: http://forum.doom9.org/showthread.php?t=87602
-Next comes the NicAudio plugin which can be found here: http://nicaudio.codeplex.com/
-And finally you need DGDecode dll included in DGmpgDec which can be found at the bottom of that page: http://hank315.nl/
Once you've put the dll in the avisynth plugins folder, you also need to extract everything in the archive(including the dll you just extracted) somewhere to install DGIndex which is also needed.

If you've managed to do all this, the hard part is over and the rest of the installation will be either extracting archives or running installers.
-x264: http://x264.nl/
I recommend(and am using) the 10bit-depth version since it create a smaller and slightly better looking encode, but it takes longer to do it and QTGMC already slow things down quite a bit.
-Winrar: http://www.rarlab.com/download.htm
-MediaInfo(CLI Interface): http://mediainfo.sourceforge.net/en/Download/Windows
-avs2pipemod(There might be a newer version later in the thread): http://forum.doom9.org/showthread.php?p=1565165&postcount=116
-Nero AAC Codec: http://www.videohelp.com/tools/Nero-AAC-Codec
-mkvtoolnix: http://www.bunkus.org/videotools/mkvtoolnix/downloads.html#windows
-VirtualDub: http://virtualdub.sourceforge.net/
-DVDDecrypter: http://www.videohelp.com/tools/DVD-Decrypter
-MPEG4ModifierCL: http://forum.doom9.org/showthread.php?t=117553

The following software are ones I use but aren't essential(it will require modifying the .bat files if you choose to not install them).
-par2cmdline(creates recovery files): http://chuchusoft.com/par2_tbb/download.html
-Timer(Calculate duration of an interval): http://www.gammadyne.com/cmdline.htm#timer
-Gimp(Image editing software): http://www.gimp.org/downloads/
-md5deep(calculate hashes): http://md5deep.sourceforge.net/#download
-movie thumbnailer(create thumbnails image): http://moviethumbnail.sourceforge.net/
-pure-sfv(create .sfv files): http://sourceforge.net/projects/pure-sfv/files/
-FlashFXP(best ftp software): http://www.flashfxp.com/download/
or WinSCP(free ftp software): http://winscp.net/eng/download.php
-Daemon Tools lite(mount .iso files in virtual cd-roms): http://www.disc-tools.com/download/daemon
-cURL(Mimics a browser behavior): http://curl.haxx.se/download.html
-yEncBin Poster 1.0.343(usenet uploader): If you use the latest beta(1.0.457 Beta2), you'll need to press an ok button 5 times for every upload. I don't have a download website to recommend to get it so just google it.

You most likely won't need the next two software and they are extremely easy to remove from the bat files(and I'll explain how).
-PSTools(execute software on remote computers): http://technet.microsoft.com/en-us/sysinternals/bb896649.aspx
-putty(connect to remote computers using ssh): http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

And finally, if you're using windows XP, you will probably need to install the "Windows Server 2003 Resource Kit Tools" which can be found here: http://www.microsoft.com/en-us/download/details.aspx?id=17657
You don't need it if you have windows vista or 7.


*Preparations*
First, download the archive containing all the required files at the end of this post.
Create a folder on your hard drive where you'll be working and extract the content there.
There is 1 config file and some small files which you need to fill(only once) before starting in the _Templates folder you extracted the files to.

You need to enter the comment in "comment_enc.txt" and "comment_up.txt"(for uploaded files that weren't encoded) you want to put in the rar files containing the videos you will be creating or simply replace Your_Name_Here with your name.

Edit the "hosts.txt" file to only include the ddl hosts you're going to upload to and you can add a message before or after that will be included in every post.

**Config file explained (_Templates\Bat\_0_config.txt)**
The format of the file is always the name of the bat file variable, a ; that act as a separator and the value of the variable each on their own line.
It is very important that it stays like this or weird things might happen.

The first line(the variable "ttl") is probably the only one that you will need to enter a value for every movie.
It's the title that will show up in the mkv informations when you play the movie.
I usually enter the english title if I have it followed by the DVD code in brackets. For example: ttl;English JAV title [ABC-123]

Next("lang") is the mkv language code information for the audio which will most likely always be japanese(jpn).

For the next 22 lines or so, you need to set the path to all the software you installed.
I left mine there since they may be the same for you and to make it easy to know which variable is for which software.
Pay close attention to the .exe name since it might not always be the one you expect to use. For example, winrar uses rar.exe instead of winrar.exe .
The only case where your .exe name might be different is when I added the version number to it. For example I used x264_r2208.exe instead of x264.exe.

"vset1"(1st pass) and "vset2"(2nd pass) are the saved processing settings I use to encode to avi.
If you don't like my default settings, you can create you own in virtualdub(google it) and put the name of your files there instead.

"tmpltdir" is the path to the _Templates folder that you extracted.

"ftpuppath" is the network path to copy the final files when they are ready to be uploaded.
You can omit it unless you plan to use a different computer to upload than the one you use for encoding like I do.

"ftpuploc" is the local path on the remote computer where the files were copied with "ftpuppath".
Only useful if you also set "ftpuppath".

"psupip" is the ip of the remote computer previously specified.

"psupuser" is the username of the remote computer previously specified.

"psuppass" is the password of the remote computer previously specified.

"isodir" is the path where the DVD images will be extracted.
You need to create that folder manually if it doesn't already exist(like all folders in this config file).

"dtdrv" is the device letter where the DVD is mounted(if using Daemon tools) or inserted(if you have the physical DVD).

"srvpath" is the http address where the files you uploaded are located.
This is only useful if you have access to a web server you can use to remote upload your files to other filehosts.

"ftpadr" is the ftp address to upload your files to(and is also the one used by putty). By default, there is a template on how to enter the ftp infos in the file.
This can either be to your web server if you have one or directly to a filehost that support ftp upload.
It isn't actually used if you use FlashFXP as your ftp software.

"sitename" is the name you assigned to your ftp infos in FlashFXP site manager.
You need to add your ftp infos directly in flashFXP instead of in "ftpadr" if you're using it. If not, you can omit it.

"spoof" is the browser infos used to automatically upload to ddl site.
You don't need to touch this and it isn't actually used right now since I'm having trouble with that part of my script these days and isn't included yet.

"HFuser" and "HFpass" are your hotfile username and password(assuming you're uploading to it).

"ttrange" are the default frames(at 60fps) used to create the test encodes which equals about 14:40-15, 29:40-30 and 44:40-45 for a total of 1 min.
No need to change this unless you're source is too short which is unlikely if it's a DVD.

"qtgmc" are the default deinterlacer settings.
Since I use pretty extreme settings which can make the encode take a lot of time, those can be modified for faster encodes(consult the documentation that came with qtgmc before changing them).

"x264param" are the default encoding settings for the mkv.
Again, I use settings that might slow the encoding quite a bit(but not as much as qtgmc which is usually so slow that these doesn't matter much for speed) so these can be changed to your liking(google "x264 documentation" before).

"x264threads" is the number of threads used for the mkv encode.
Set this to the highest number that doesn't make the encode crash. 0 is for letting the computer decide, but it will very likely crash if you use my other default settings.
For me, this is: My CPU number of logical cores(4 physical + 4 logical when hyperthreading is enabled) - 1

"xvidthreads" is the number of threads used for the avi encode.
Set this to the highest number that doesn't make the encode crash. 0 for auto(same warning as above).
I set this to 1 less than for the mkv since the avi encode doesn't go through avs2pipemod which helps prevent crashes due to memory buffer overflow.


*How to use the bat files*
The first thing to do is to create a folder in you work folder named after the DVD code and copy all the files from _Templates\Bat\ in it.
For example, my work folder is D:\_1enc_\ and if I wanted to encode the DVD IESP-515, I would create the folder D:\_1enc_\IESP-515\ and copy the config and all the bat files in it.

All the bat files are numerated in the order they need to be executed to make things less confusing.

"_0a_Create_infos_files.bat" will copy a template of the .nfo file(which is used to store information about the movie. Open it with notepad or any text reader).
It will also add the work path and the DVD code(which is used for many things) at the end of the config file.

"_0b_MountISO.bat" will mount a .iso if you drag and drop one on it or simply open daemon tools if you simply double-click it.
It's not that useful if you associated the iso files to daemon tolls since you can simply double-click on the iso and it will mount itself. In that case it can be skipped.

"_1_Create_EZ_avs.bat" does many things to prepare the required files:
-Extract the DVD to the ISO folder.
-Index it with DGindex so it can be used in avisynth.
-Create a .avs script to help configure the required settings for the encode.
-Copy the chapter file.

Now there's real work to be done.
You need to open the newly created file that ends with "_EZ.avs" with VirtualDub and it should load the video with an histogram on the right.
If it doesn't, there's most likely a problem with your avisynth install or you're missing a filter. Read the error message carefully and google the error to try and fix it if you have no idea how.
If it loads properly, open the ..._EZ.avs file with notepad this time and be ready to edit the script which goes as follow:
The first line, as well as all other line starting by # is a comment and is ignored.
Next you have SetMTMode which enables multithreading.
The next 3 lines load the video, audio and merge them together.
Then the multithreading setting is changed to 2.
The video is deinterlaced as fast as possible so that it's not extremely slow when viewing it.

And now to the important stuff you need to change:
Autocrop will try to remove the black borders on the side of the video by itself.
If you're not satisfied with it, you need to remove the line starting with autocrop, remove the # before Crop(0, 0, -0, -0) and replace the 0s with how many pixels you want to remove on each side manually(starting on the left and going clockwise).
You can also use the values from autocrop as a reference(see #3 on the picture).
[FULLATTACH]621587[/FULLATTACH]
If you're satisfied with what autocrop did, just leave it alone and go to the next setting.

Seek to a couple of places in the video and look at the white bars in the top histogram on the right(#4 on the picture).
If everything stays between the brown sides, go to the next setting.
If there's yellow lines in the brown sections, remove the # before SmoothCurve(Ycurve="0-2;255-235") and change the 2 and the 235 until there's no or very little yellow.
This will make adjustments to the brightness of the video, especially to fix the spot that are too bright.
The defaults are the most common values, but won't work 100% of the time.
Don't forget to save the .avs every time you edit it and press f2 in VirtualDub to see the changes you made.

Now you need to trim the beginning and the end of the video to remove any undesired section like the manufacturer logo or previews of other movies.
First, remove the # before "Trim(0,)" .
Seek to where you want to cut the video at the beginning(use the arrows, page up and page down for more precision) and write the frame number(#1 in the picture) instead of the 0 after Trim.
Do the same for the end(write it after the ",").

Now that everything is set(you can close the .avs opened with notepad), you need to create the chapters.
Open the file ending with "_Chapters.txt".
The first chapter always start at 00:00:00.000(HH:MM:SS.milliseconds) and you write the desired name after "CHAPTER01NAME=".
Seek to where the next chapter begins and write the timestamp(#2 in the picture) in the chapter file as well as the desired name.
Repeat until you're at the end of the video and delete all the unused chapters.
A completed chapter file will look something like this:

CHAPTER01=00:00:00.000
CHAPTER01NAME=Chapter 01
CHAPTER02=00:05:47.581
CHAPTER02NAME=Chapter 02
CHAPTER03=00:11:40.600
CHAPTER03NAME=Chapter 03
CHAPTER04=00:21:31.390
CHAPTER04NAME=Chapter 04
CHAPTER05=01:04:27.864
CHAPTER05NAME=Chapter 05
CHAPTER06=01:13:40.883
CHAPTER06NAME=Chapter 06

Once it's done, you need to enter some infos in the file ending with _.nfo.
The first 2 lines are the English title(If available or make sense if you translate the japanese one with google translate) and the japanese one
Then replace "http://" with the official address of the DVD if you can find it or delete it if you can't.
Set the "Runtime" to the official length on the cover or website if it differs from the default.
Add the "Studio"(company) name.
Replace the "-" with the "Label" of the DVD if there is one(there often isn't).
Choose a "Genre" between the 2(delete the other) and you can add more categories if you want(separate them with a comma followed by an empty space: ", ". This is very important for the tag creation).
Write the "Actresses" names if you know them or leave Unknown if you dont. Use the "Eng name(Jpn name)" format for example: Rui Saotome(早乙女ルイ) . Separated them the same way as the Genre if there's more than one.
And finally enter the "Release Date" in the YYYY/MM/DD format.
The rest will all be added automatically once the videos are encoded.

Now you only need to download the DVD cover and save it with the DVD code followed by .jpg to have all the required files.

"_2_Create_10b_avs.bat" will create the final .avs with the proper deinterlacing settings and adjust the aspect ratio of the movie to compensate for the cropping, calculate the resolution for the avi encode so that it's mod16 compliant and add both to the config file.

Now you need to set the deinterlacer settings.
Open the file named with the DVD code followed by .avs with notepad.
The most important setting to change is "EZKeepGrain" and you sometime need to change the "Sharpness".
To decide what number to use, run "__Test_Enc.bat", write something to identify the test(I like def for the first test) and press enter.
It will create a sample encode to show you what the result will be like.
You can try to play with both settings to see what the result looks like and choose the one you like best.
I usually set EZKeepGrain to 0.4(which is the value I end up using the most) for the second test and name it 4-3(after the 2 settings which help remembering which one looks best).
There is no magic settings since the source are all different, but those are a good place to start. You can read qtgmc manual if you want to play with the settings more.

"__Test_Enc_XviD.bat" will do the same as "__Test_Enc.bat" but will create an avi instead of an mkv.
Since the avi isn't as good looking as the mkv, I don't recommend it to choose the settings since it makes it harder to judge if they help or not.
It is only included since I needed it once in a while to test stuff and can be used to make sure encoding avis is set up right before the final encode.

Once you've chosen the best settings, it's time to run "_3_Final_Enc.bat.
It will encode the video, then the audio and merge both with the chapters, title and aspect ratio in the mkv.
Next it will call "_4_Prepare Info and Post files.bat" which will:
-Create a .nfo file using the one you half filled and complete it.
-Create a .txt file based on the .nfo to copy in your post(it will add a redirection for the website and a template of you ddl hosts after).
-Create a folder.jpg file by cropping the front of the DVD cover.
-Create a thumbnail file.
-Create a .md5 file to verify if the video becomes corrupt at some point.
-Create rar archives separated at 400MiB with your personal encoding comment from the _Template folder.
-Create a .sfv file to verify if the rar becomes corrupt at some point.
-Call "_4a_Tags.bat" which will create a "_tags.txt" file containing a list of tags to add to your post.

Next it will call "_3_Final_Enc_XviD.bat" which will encode the avi and then set the proper aspect ratio.
In case the encode would crash during the avi creation(it sometimes happen and I don't know why), you can simply run "_3_Final_Enc_XviD.bat" manually and it will encode the avi and prepare the files and infos.
Once the avi is done, it will also call "_4_Prepare Info and Post files.bat" which will do the same thing it did to the mkv, but for the avi(there will only be 1 tag file though).

If you have the option of uploading the files using ftp, run "_5_FTP_upload_Flashfxp.bat" and it will upload all the .rar files as well as the .sfv to it.
I configured this one for my very special needs so it might take a little editing before you can use it.
-If you want to use it to upload from another computer, you need to change the following with your remote computer info(in the line before last) before it will work: "\\IP_of_remote_computer -i -u "remote_computer_name\username" -p password"

-If you want to upload from the same computer you're running the bat file instead of another computer on your network like I do:
Replace

FOR /F "tokens=2 delims=;" %%G IN ('findstr ftpuppath; "%wrkdir%_0_config.txt"') DO SET ftpuppath=%%G

on line 7 with

SET ftpuppath=%wrkdir%

And replace

FOR /F "tokens=2 delims=;" %%G IN ('findstr ftpuploc; "%wrkdir%_0_config.txt"') DO SET ftpuploc=%%G

on line 8 with

SET ftpuploc=%wrkdir%

Then go to the end of the file and:
-You can remove the "ROBOCOPY" line since it only copies the files to the remote computer.
-Replace

START "FlashFXP upload" "%psexec%" \\%psupip% -id -u "%psupuser%" -p %psuppass% "%flashfxp%" -c2 -logfile="%ftpuploc%%basefile%.log" -get "%ftpuploc%%ftp_queue%"

with

START "%flashfxp%" -c2 -logfile="%basefile%.log" -get "%ftp_queue%"

And if you don't want to upload the .sfv file, removing all 3

FOR /F "delims=" %%H IN ('dir /b !basefile!.sfv') DO ECHO 04-%sitename%%ftpuploc%%%H%%H%%~zH>>"%ftpuppath%%ftp_queue%"

should do it(or screw everything up since I made this bat file way too complex to easily modify)

"_5_FTP_upload_WinSCP.bat" does the same thing as the FlashFXP one except that it will use WinSCP as a ftp software instead. The possible changes are also the same as FlashFXP.

The "_5a_putty.bat" file will connect to the remote server you FTPed to in case you need to verify the integrity of the rar files with the .sfv before uploading them somewhere else(assuming you're using your own server).

The "_5b_HF.bat" file will read the prequeue.txt file(created during the ftp upload) and upload the archive files from the remote server to Hotfile using the remote upload feature and then add the links directly into the .txt file created earlier based on the nfo.

The "_5c_Usenet_upload.bat" file(you need to drag-and-drop a video file on it and choose if you want to queue it or post it immediately and also if you want to post it to the lesbian group or only the asian one.
Will:
-Create a Usenet folder where all the usenet related files will be moved for uploading.
-Create a new set of rar files optimized for usenet.
-Create a .sfv file for those rar files.
-Create a set of par2 files with 5% redundancy so it become possible to repair the rars if they become corrupt.
-Copy the thumbnail file and the nfo files for posting(you can also add a line to copy the cover if you also wish to upload it).
-Create a nzb file(used by usenet clients for downloading which you should post to the forum).
-Copy everything to the remote computer and start uploading the Usenet folder to usenet.

As with the ftp uploading, I do it on a remote computer so if you don't wish to do the same, remove near the end:

START "Usenet upload" "%psexec%" \\%psupip% -id -u "%psupuser%" -p %psuppass%

twice and replace

"%ftpuploc%%avs%"

after it by

"%avs%"

and remove these 3 lines:

ROBOCOPY %wrkdir%%newsd% %ftpuppath%%avs% /R:0 /W:0

REM Check if a .nzb was created at an earlier time for this movie
IF EXIST "%ftpuppath%%avs%\%~n1.nzb" (DEL "%ftpuppath%%avs%\%~n1.nzb")

For the required configuration instructions or if you wish to modify some of the posting options, please refer to the configuration section in my "auto usenet posting script" tutorial which is based on this script. You can find it here: http://www.akiba-online.com/forum/showthread.php?t=1199077

Finally, "_6_Cleanup.bat" will remove all files other than the ones that were added inside the rars and the bat files.
Only use it after you're done uploading and posting and only if you're too lazy to delete them manually.
It won't remove the extracted iso though so you'll have to do that manually either way or modify the script to make it delete it.


*Known bugs*
-The avi encode sometimes crash(weird memory problem apparently) when lauched from the mkv encode batch file.
-Timer doesn't handle multiple encodes at the same time properly since it follow the last in first out format instead of remembering who told it to start to calculate the elapsed time.
-Movie thumbnailer creates color smudges with 10-bit videos, but it's hard to notice most of the time unless you look for it.


*Download*
v1.2: View attachment Auto_Enc-Post_v1.2.rar


*Troubleshooting*
-If you get a missing msvcr71.dll error when opening virtualdub, you need to install the .NET framework v1.1
If you have windows 7 64bit, it won't install so you need to get it manually from inside the .NET framework 1.1 SDK which you can find here:
http://www.microsoft.com/en-us/download/details.aspx?id=16217
You need to open it with winrar(or your favorite archiver).
Then open the netfxsd1.cab file also with winrar
Extract the file named FL_msvcr71_dll_____X86.3643236F_FC70_11D3_A536_0090278A1BB8
Rename it to msvcr71.dll
And finally, move it to the syswow64 folder(located in the windows folders on the partition windows is installed on)

-If you get a green image when the video is paused in virtualdub, do the following:
In virtualdub, go to Options->Preferences...->Display->Uncheck "Use DirectX for display panes" and click save


*Final word*
If you discover mistakes or something unclear in the post(it's so long that I don't feel like proof reading after 5 hours of writing) or in the process, have question about how to make it work(at least make a minimum of effort on your own first) or on how to configure something, just let me know and I'll be happy to help/fix.


*Version History*
v1.2:
-Chapters for the mkv test file to separate the 3 sections
-A lot of verification has been implemented to prevent mistakes
-Can now queue usenet uploads
-Can Choose if movie should be uploaded to a.b.m.e.a or a.b.m.e.a and a.b.m.e.l with usenet
-Can now choose to upload with FlashFXP or WinSCP
-Added bat file with informations about resolutions and aspect ratios in case a manual change is needed in the config file
-Separated the nfo creation to permit information guathering before ripping the DVD
-Added scripts for posting/reposting a movie from scratch
-Added proper * alignment for all resolutions
-Added support for company V&R in the tags creation
-Cleanup will now also remove the temporary usenet folder as well as all the bat files
-The remote computer connection settings are now in the config file
-Many small fix/tweaks/cleanup I forgot about

v1.1:
-Added uploading to Hotfile(using cURL)
-Added uploading to usenet(using yEncBin Poster)
-Fixed drag-and-drop for some scripts
-A few other fixes, tweaks and cleanup I can't quite remember or not worth mentioning.

v1:
-Initial Release.

 

[savajex]

Hi, I'm back to pester you with more questions :notagain:

I'm following your guide, except I'm using 8bit instead of 10bit, and I've made all the adjustments. Everything seems to work fine until I try to run _Test_enc.bat. I end up with a sample MKV with a fps that is doubled. :destroy: 60000/1001 instead of 30000/1001. And I can't figure out why.

I examined _Test_enc.bat to the best of my abilities and this is what I found (though I don't know if it'll help): I figured the fps was being written incorrectly somewhere so I looked at the d2v first. Says 30000/1001. Then I looked at the tmp.avs and everything looked fine. Then I got to a2pInfo.txt and saw that the fps was 60000/1001.

Why?? :cry: One other note, the batch printed "The system cannot find the path specified" but I'm not sure which path. Either way, it still made a 1min MKV but with double fps. I don't know if that's relevant.

I'd be very grateful for any help. I seem to be forever doomed with getting the right fps...

 

[SamKook]

I end up with a sample MKV with a fps that is doubled. 60000/1001 instead of 30000/1001. And I can't figure out why.

That's actually perfectly normal and it's the intended default behavior(I probably should have mentioned that).

An interlaced video is showing 2 half frames for every frame of a progressive video so to keep the original motion, the deinterlacer recreates the missing half of each of those frames so it ends up with double the framerate.

If you want it to stay at 30000/1001, you need to add SelectEven() right after the QTGMC line in the DVD_code.avs file. This will tell it to drop half the frames and that's actually what it will do for the Xvid version if you make one.

I prefer to keep 60fps since the video has a more fluid look and it's not that much bigger(around 15% I think) since many frames are similar to another one so it compress really well.

One other note, the batch printed "The system cannot find the path specified" but I'm not sure which path.

It's probably the wrkdir path which is added to the config file after you run the _1 bat file(I think, I changed it in the version I currently use) since it will use the current directory by default 99% of the time. Before running the _Test_enc.bat file, you need to have run up to the _2 bat file.

If you've already done that, then it's probably trying to load a software you chose not to use. After writing all this, I took a look at your screenshot for a second time and I'm almost certain it's the timer since this is where the start time of the encode should be displayed. It won't affect the encode itself, it simply won't calculate the time it took to encode.


Don't be afraid to pester me more, I'm glad to see people are actually using it and I'll be looking forward to more high quality encodes posted on the forum.

 

[savajex]

Ohh... ...so I'm not completely dense? :relief:

The deinterlacing and double framerate make complete sense now. Thank you for explaining so patiently and clearly. Every time I look at these video technical things, they just go straight over my head because there's just so much info.

And you're probably right about the timer. I'm not using it.

Thanks again! :bow-pray:

 

[Kevin_san]

Allow me to add a specialized procedure for optimized DVD transcoding:

http://pastebin.com/KGMXT8EC
This is a usable script file for Avisynth, with a settings package for Avidemux.
It contains short and long descriptions.

There is a number of ISO DVD rips around, that are hard to transcode into h264. Default procedures give poor, blurred results.
I guess they have been posted for that reason also, as the mp4 attempts were perhaps not worth uploading.
Also, there are many mp4/avi uploads of DVDs that look only ok at given file size, but not brilliant. Still thanks.
If small size is the purpose, then we can't do much about, but at file sizes of 12-20 MB per minute, we can achieve way better quality,
using the suggested procedure and filtering.

I developed a set of customized convolutionary matrix filters, to add preprocessing that slams the encoder with optimized detail resolution.
It is packaged and described in the above pastebin link.
Mark and copy the RAW Paste Data, and save it with .avs extension. Open it with your script editor, or the operating system's low level editor.
From there, save the bottom section away as a .py file for the Avidemux.

Default settings already should look great with many difficult DVDs.
Still, the different matrix specimen should be tested to further optimize, as the particular problems in the source material may be quite different.
Addressed problem are:
- deinterlacing artifacts
- smearing of edges, with blurring of a wider radius like 3-5 pixels
- some ringing, with lucky choice of matrix parameters
- blurring by the encoder that thinks the details are noise and not relevant, because the interlacing and poor mpeg2 quality makes the detail unrecognizable by the motion detection
- blurring and ringing by the video player software
Many processed videos look remarkably better than what vlc player makes from directly opening the iso file.

Particular specialties are: Asymmetric convolution and upsampling.

 

[SamKook]

I'm a bit late in reading this since I lost the thread notification in the last update, but that's not compatible with the script I made in this thread(which are a bit dated now, I haven't posted the new version because the interest is pretty much non-existent as far as I know and it's not quite user-friendly enough yet) so I'm not sure why you posted it here(I don't even use avidemux) unless it was to provide some alternative to it. If you only provided the avs file template, it would be very easy to use it with what I made.

I had a quick look and I noticed you're using yadiff as the deinterlacer instead of qtgmc which is much better at the job so I'm not sure why that is. I strongly suggest you look into qtgmc if you don't know about it, you won't regret it.

Also, you seem to be using the color space for HDTV by default instead of the one for DVD so unless there's some place that ask you to change it(which I don't think is the case from your explanation), you might want to change that since you mentioned this was for DVD.

Also, you have some hardcoded path in there(for example: F:\programs\AviSynth+\dgdecode.dll) so it might be hard for someone to use, especially since the description at the beginning talks about putting that in the C drive.

 

[Kevin_san]

Thanks for reading,
yes it's an alternative, probably I should have posted it in a new thread.
I use AviDemux for ease of tuning. Soon, I should work out an identical Avisynth template where I can enter the settings that I found by adjusting in AviDemux.


It is all a basic idea about using convolution to mitigate some notorious issues, where people often use nonlinear algorithms, but as I said, to me the linear approach looks more organic.

I know about qtgmc but I am not willing to spend the processing time for it. Natually, everyone can replace the yadif easily. But the fine-tuning might have to be redone because of the different behavior in the sharpening kernels.
The resolution upsampling is essential, and also helps with deinterlacing-artifacts. And so to say, it makes the pixels fully "round", while preserving a maximum of sharpness that is possible with a DVD source.

I used it for the type of ISO files posted here, and I found out that the color space usually fails. It may be something with the automatic conversions in avidemux.
Anyway, my use of the color matrix together with the color filters in avidemux gives me the most natural and true look, compared via VLC. Only for some videos, it is different.
I tell this after trying a lot of matrix pairings, of course including what the documentation says. Maybe for real movie DVDs, things come out differently.

I commented, that the hardcoded paths need to be corrected. I have installed different versions of Avisynth and have to make sure it loads the correct DLL. For beginners, perhaps not so trivial and obvious.
It is all meant to be customized by the user, so they would insert their own path definitions anyway, starting with the video itself.

Thus, I was hoping for some discussion about achieved quality.

However, I can share an updated set of kernels:
#GeneralConvolution(0, "3 -4 8 -10 112 0 0 -0 -4 -15 0 0 -6 -0 15 0 0 0 0 -5 0 0 0 0 0") #C3.1
#GeneralConvolution(0, "3 -3 9 -13 112 0 0 -0 -4 -15 0 0 -6 -0 15 0 0 0 0 -5 0 0 0 0 0") #C3.2
#GeneralConvolution(0, "3 -3 9 -13 140 0 0 -0 -4 -15 0 0 -6 -0 15 0 0 0 0 -5 0 0 0 0 0") #C3.2a
#GeneralConvolution(0, "3 -3 10 -15 112 0 0 -0 -4 -12 0 0 -6 -0 14 0 0 0 0 -4 0 0 0 0 0") #C3.3
#GeneralConvolution(0, "4 -5 14 -15 112 0 0 -0 -4 -12 0 0 -6 -0 14 0 0 0 0 -4 0 0 0 0 0") #C3.3a
#GeneralConvolution(0, "2 -5 12 -20 112 0 0 -0 -4 -10 0 0 -6 -0 12 0 0 0 0 -4 0 0 0 0 0") #C3.4

#GeneralConvolution(0, "3 -5 17 -21 112 0 0 -0 -4 -13 0 0 -6 -0 15 0 0 0 0 -4 0 0 0 0 0") #C4.0
#GeneralConvolution(0, "3 -8 7 -11 112 0 0 -0 -6 -12 0 0 -3 2 14 0 0 0 0 -4 0 0 0 0 2") #C4.1 sharper

C3.2a and C4.0 are recommended for testing first. C4.0 looks especially natural.


After the convolution has been tuned for minimal artifacts, a nonlinear anti-ringing filter might become useful, without losing detail.

So, if you have the time and interest, I would be very glad to hear about results.
I can post the recent full scripts also, as they become more tuned over the time, e.g. about borders and cropping.

 

[SamKook]

I'll give it a try at some point, but it will probably take at least a week or two until I can get to it, I have a bit too much on my plate right now.

 

[Kevin_san]

Thx, I'm looking forward. I'll give you a more recent example of settings soon.

Addendum,
AviDemux gives me the benefit that >90% of a 4 core CPU will be used. The tablet encodes 4-7 fps, the old i7 notebook would at least double that.
For improvements, I think the way to go is, to work out the whole formula with use of bit-depth upsampling and dithering, in a non-logarithmic code space of at least 10 bits per color plane. This would become visible in softness of the color spectrum. (Less "banding" etc.) There are tool families for this available in the avisynth forums.
AFAIK, AviDemux can be scripted and used with command lines.

 

[Kevin_san]

here is a recent package of script settings


MyProject.avs

## check all hardcoded paths and replace with your project and configuration!

#-- define input video, here we use MPEG2 sources
LoadPlugin("F:\programs\AviSynth+\dgdecode.dll") ## Version 1.5.8 + corresponding DGIndex.exe
MPEG2Source ("F:\work\myproject.d2v",cpu=0,info=2)


#-- optionally, correct erraneous field-shift for fake-interlaced sources:
# (that's a quick and dirty method when the default would combine fields of two different frames.)
#SeparateFields
#Loop(30,4,5) # this is just the header picture, in case we use it. This example found it at position 4/5.
#Loop(0,0,0)  # delete first field, when field order is incorrect in that it spills over to the next frame.
#Weave


#-- Deinterlace Yadif
#-- delete this, when source is not actually interlaced => check the "combing" and the frame-by-frame view
Load_Stdcall_plugin("F:\programs\AviSynth+\plugins\yadif.dll")
Yadif(mode=1) # double frame rate mode

AssumeFPS(60000,1001) # if we have double-fps yadif, try for some videos. there may be false duration of the video, or jerky playback, set or remove this option according to results.
# we assume that the DVD audio was mastered with the standard video frame rate for NTSC.


rff=60000./1001. # numeric representation of 60fps
###rff=29.97*2.  # alternative hardcoding for 60fps
#--- show some still pictures: (correct the audio intro length for this!)
#tpos=1*60+45.537  # hard coded [pics start time] = minutes*60 + ss.ddd #(seconds.decimals)
#fpos=int(tpos*rff)  # convert to frame position
#tsl=int(1.*rff/2.)+1  # display duration for one pic
#n=fpos+1  # starting frame no. of pic series
#Loop(tsl,n+4,n+5)  # now looping frames for picture gallery: every pic is listed with an entry here
#Loop(tsl,n+2,n+3)  # (show some pictures in this DVD menu for half a second each)
#Loop(tsl,n,n+1)  # ! use the loop statements in decreasing order of frame position!




AddBorders(4,0,4,4,$7F7F7F)
ConvertToRGB32(matrix="PC.709",interlaced=false,chromaresample="lanczos4")
# this turns out the best setting for color conversion, in context with the AviDemus filters.
# facit: default for this type of work: PC.709 = HDTV in spite of converting DVD source.
# though most DVDs = Rec601 or should be!
# Rec.709 should always be used when your source is HDTV (but fits some DVDs also)
# "Rec601" : Use 601 coefficients, scaled to TV range [16,235].
# "PC.601" : Use 601 coefficients, keep full range [0,255].


# --------------------------- convolution kernel library, select one only! ----------------------------------------------------
# -- some symmetric kernels, caution - they need different border and crop settings from the asymmetric
#GeneralConvolution(0, "-1 0 3 0 -1  0 -1 -3 -1 0  3 -6 48 -6 3  0 -1 -3 -1 0  -1 0 3 0 -1")
#GeneralConvolution(0,  "0 0 3 0 -0  0 -1 -3 -1 0  4 -4 44 -4 4  0 -1 -3 -1 0  0 0 3 0  0")
#GeneralConvolution(0, "-1 0 3 0 -1  0 -1 -3 -1 0  3 -5 56 -5 3  0 -1 -3 -1 0  -1 0 3 0 -1")  # 'Crystal Outline'
#GeneralConvolution(0, "-1 0 3 0 -1  0 -1 -3 -1 0  3 -5 48 -5 3  0 -1 -3 -1 0  -1 0 3 0 -1")
#GeneralConvolution(0, "-2 0 3 0 -2  0 -1 -3 -1 0  2 -6 56 -6 2  0 -1 -3 -1 0  -1 0 3 0 -1")  # C2 entry level
#GeneralConvolution(0, "-3 0 4 0 -3  0 -1 -3 -1 0  12 -12 56 -12 12  0 -1 -3 -1 0  -3 0 4 0 -3")
#GeneralConvolution(0, "-2 0 3 0 -2  0 -1 -3 -1 0  11 -12 56 -12 11  0 -1 -3 -1 0  -1 0 3 0 -1")

# -- select one of these asymmetric kernels, ordered by increasing radius and impact:
#  C3.1 will do for most good quality DVDs with sufficient sharpness
#  you can select the entry by removing the comment character (#)

#GeneralConvolution(0, "2 -5 15 -20  96 0  0 -0 -6 -12  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0") #X7

#GeneralConvolution(0, "3 -4  8 -10 112 0  0 -0 -4 -15  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0") #C3.1
#GeneralConvolution(0, "3 -3  9 -13 112 0  0 -0 -4 -15  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0") #C3.2
GeneralConvolution(0, "3 -3  9 -13 140 0  0 -0 -4 -15  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0") #C3.2a
#GeneralConvolution(0, "3 -3 11 -13 173 0  0 -0 -4 -15  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0") #C3.2c
#GeneralConvolution(0, "3 -3 10 -15 112 0  0 -0 -4 -12  0  0 -6 -0  14  0  0  0  0 -4  0  0  0  0  0") #C3.3
#GeneralConvolution(0, "4 -5 14 -15 112 0  0 -0 -4 -12  0  0 -6 -0  14  0  0  0  0 -4  0  0  0  0  0") #C3.31
#GeneralConvolution(0, "2 -5 12 -20 112 0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0") #C3.4

#GeneralConvolution(0, "3 -5 17 -21 112 0  0 -0 -4 -13  0  0 -6 -0  15  0  0  0  0 -4  0  0  0  0  0") #C4.0 'natural look'
#  adapt this (15) for deinterlacing-artifact optimization
#GeneralConvolution(0, "3 -8  7 -11 112 0  0 -0 -6 -12  0  0 -3  2  14  0  0  0  0 -4  0  0  0  0  2") #C4.1 'naturalextrasharp'

# C4.0 has strong halo softening and anti-ringing compensation, but only for DVDs that have correct bandwidth limiting.
# on pixel-sharp and high-contrast vertical lines it will still show a ringing pattern.
# use C4.0 for sources that would look awkwardly blurred after the x264,
# when they won't allow the stronger sharpening by the other kernels because of noise, grainy lines, and sharp halos/outlines
# it is a good choice for most videos, but other kernels can deliver a sharper picture when the source has enough detail and little noise.
# C3.2b/c is also a good choice, try together with a triple convolutional sharpener in avidemux on upsampled level (~1600x960)


#these are softer and have a wide radius:
#GeneralConvolution(0, "5 -13 17 -21 100  0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0") #a2
#GeneralConvolution(0, "5 -13 18 -21 103  0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0") #a2'
#GeneralConvolution(0, "5 -13 15 -20 120  0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0") #a3



#-------------- test bed for convolution tuning ---------
tpos=2.*60.+15.000  # test seq starting time = minutes*60 + seconds.milliseconds
### rff=29.97*2  # frame rate, defined already above
fpos=int(tpos*rff)
tsl=int(40.*rff)  # test slice length in seconds
# basically, you select a time slice and repeat it with a number of kernels to test with.
# switch it off with the comment brackets  /* ....... */
/*
#Trim(fpos, -tsl) + blankclip(last,5) \
#+Trim(fpos, -tsl).GeneralConvolution(0, "2 -6 15 -20 96  0  0 -0 -4 -12  0  0 -6 -0  15  0  0  0  0 -4  0  0  0  0  0").subtitle("x8") + blankclip(last,15)\
Trim(fpos, -tsl).GeneralConvolution(0, "3 -4  8 -10 112 0  0 -0 -4 -15  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0").subtitle("C3.1") + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, "3 -3  9 -13 112 0  0 -0 -4 -15  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0").subtitle("C3.2") + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, "3 -3 10 -15 112 0  0 -0 -4 -12  0  0 -6 -0  14  0  0  0  0 -4  0  0  0  0  0").subtitle("C3.3") + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, "2 -5 12 -20 112 0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0").subtitle("C3.4") + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, "3 -5 17 -21 112 0  0 -0 -4 -13  0  0 -6 -0  15  0  0  0  0 -4  0  0  0  0  0").subtitle("C4.0") + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, "5 -13 18 -21 103  0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0").subtitle("a2'") + blankclip(last,15)
*/


Crop(0,2,-2,-2)

ConvertToYV12(matrix="PC.709",interlaced=false,chromaresample="lanczos4")
# see above for the heuristic choice of color matrix

MyProject.py - to load with AviDemux:

#PY  <- Needed to identify #
#--automatically built--

adm = Avidemux()
adm.loadVideo("::ADM_AVS_PROXY::")
adm.clearSegments()
adm.addSegment(0, 0, 9000000000)
adm.markerA = 0
adm.markerB = 5000000000
adm.videoCodec("x264", "useAdvancedConfiguration=True", "general.params=AQ=20", "general.threads=0", "general.preset=ultrafast", "general.tuning=film", "general.profile=baseline", "general.fast_decode=False", "general.zero_latency=False"
, "general.fast_first_pass=True", "level=-1", "vui.sar_height=27", "vui.sar_width=32", "MaxRefFrames=5", "MinIdr=25", "MaxIdr=250", "i_scenecut_threshold=40", "intra_refresh=False", "MaxBFrame=5", "i_bframe_adaptive=2"
, "i_bframe_bias=0", "i_bframe_pyramid=2", "b_deblocking_filter=True", "i_deblocking_filter_alphac0=-1", "i_deblocking_filter_beta=0", "cabac=True", "interlaced=False", "constrained_intra=False", "tff=True"
, "fake_interlaced=False", "analyze.b_8x8=True", "analyze.b_i4x4=True", "analyze.b_i8x8=True", "analyze.b_p8x8=False", "analyze.b_p16x16=True", "analyze.b_b16x16=True", "analyze.weighted_pred=2", "analyze.weighted_bipred=True"
, "analyze.direct_mv_pred=3", "analyze.chroma_offset=1", "analyze.me_method=2", "analyze.me_range=17", "analyze.mv_range=-1", "analyze.mv_range_thread=-1", "analyze.subpel_refine=9", "analyze.chroma_me=True"
, "analyze.mixed_references=True", "analyze.trellis=2", "analyze.psy_rd=1.100000", "analyze.psy_trellis=0.000000", "analyze.fast_pskip=True", "analyze.dct_decimate=True", "analyze.noise_reduction=0", "analyze.psy=True"
, "analyze.intra_luma=0", "analyze.inter_luma=0", "ratecontrol.rc_method=0", "ratecontrol.qp_constant=0", "ratecontrol.qp_min=4", "ratecontrol.qp_max=58", "ratecontrol.qp_step=8", "ratecontrol.bitrate=0"
, "ratecontrol.rate_tolerance=1.000000", "ratecontrol.vbv_max_bitrate=0", "ratecontrol.vbv_buffer_size=0", "ratecontrol.vbv_buffer_init=0", "ratecontrol.ip_factor=1.400000", "ratecontrol.pb_factor=1.400000"
, "ratecontrol.aq_mode=1", "ratecontrol.aq_strength=1.400000", "ratecontrol.mb_tree=True", "ratecontrol.lookahead=40")
adm.addVideoFilter("fluxsmooth", "temporal_threshold=15", "spatial_threshold=0")
adm.addVideoFilter("addBorder", "left=0", "right=2", "top=2", "bottom=2")
adm.addVideoFilter("swscale", "width=1600", "height=968", "algo=2", "sourceAR=2", "targetAR=2")
adm.addVideoFilter("chromashift", "u=-1", "v=-1")
adm.addVideoFilter("Sharpen", "luma=True", "chroma=True")
adm.addVideoFilter("Sharpen", "luma=False", "chroma=True")
adm.addVideoFilter("Sharpen", "luma=True", "chroma=True")
adm.addVideoFilter("contrast", "coef=0.950000", "offset=40", "doLuma=False", "doChromaU=False", "doChromaV=True")
adm.addVideoFilter("colorYuv", "y_gain=0.000000", "y_bright=0.000000", "y_gamma=3.000000", "y_contrast=10.500000", "u_gain=0.000000", "u_bright=0.000000", "u_gamma=0.000000", "u_contrast=12.000000", "v_gain=0.000000", "v_bright=0.000000"
, "v_gamma=0.000000", "v_contrast=0.000000", "matrix=0", "levels=0", "opt=False", "colorbars=0", "analyze=1", "autowhite=False", "autogain=False")
adm.addVideoFilter("swscale", "width=972", "height=646", "algo=2", "sourceAR=1", "targetAR=1")
adm.addVideoFilter("rotate", "angle=90")
adm.addVideoFilter("chromashift", "u=-1", "v=-1")
adm.addVideoFilter("rotate", "angle=270")
adm.addVideoFilter("crop", "top=4", "bottom=2", "left=8", "right=12")
adm.audioClearTracks()
adm.setContainer("MP4V2", "optimize=0", "add_itunes_metadata=1")

In AviDemux, you need to open the [File][connect avsproxy] first.
Seek to the end of the video, and remember the time stamp there. Leave the position there.
Then [File][Project Script][Run Project (tinypy)] and open this template file.
Caution, we have a hack here in that you inherit an incorrect (hopefully bigger) length of the video. Use the A and B markers to select the portion to encode. In a simple case, you just press <B> and set the end of the video by this. (Because the time pointer is already at the end of the video.)

Tuning:
This time the 1600 upsampled domain uses 2 full sharpeners and 1 chroma sharpener additionally. This is for getting the last yota of sharpness, without creating too much of ringing.
Some videos need only 1 full sharpener at 1600.
Some need it in the 960 domain.

Re-check chroma shift settings as the last step, after setting final sharpener and border crop.

addendum
adm.addVideoFilter("contrast", "coef=0.950000", "offset=40", "doLuma=False", "doChromaU=False", "doChromaV=True")
- chroma V reduction might be set to 0.97 or even 1.0 (neutral) if you feel the skin can become reddish. (note, that the contrast filter user interface does not allow frac numbers with two digits after the decimal sign. it steps integers/10, so this has to be edited and loaded with a script.)
But then also,
adm.addVideoFilter("colorYuv", "y_gain=0.000000", "y_bright=0.000000", "y_gamma=3.000000", "y_contrast=10.500000", "u_gain=0.000000", "u_bright=0.000000", "u_gamma=0.000000", "u_contrast=12.000000", "v_gain=0.000000", "v_bright=0.000000"
, "v_gamma=0.000000", "v_contrast=0.000000", "matrix=0", "levels=0", "opt=False", "colorbars=0", "analyze=1", "autowhite=False", "autogain=False")
- the chroma amplification (here: 12.0 - there is only one working parameter for both axis together) might be reduced, to 10.5 or 11.0

this setting was used with S.B.V.D._.0.1.4.4. it has a lot of jpeg noise and is blurry. it came out with 2GB but beautiful and better than the raw DVD playback.
did a similar older (pnpk) it looks absolutely amazing. eyes, teeth and skin much more sexy. ~90m of main video sans audio use up 1.2GB this time, compression rate is 21 (was 20 above).


---------------------------------------------------------------------------------------------------------------------------------
Another case study: Progressive (non-interlaced) DVD that looks very blurred after transcode

LoadPlugin("F:\programs\AviSynth+\dgdecode.dll")
MPEG2Source ("F:\fun\MyProject.d2v",cpu=0,info=2)

Loop(0,0,2) # delete 3 black frames

#-- Deinterlace -- delete this, when source is not actually interlaced; test the "combing"
#Load_Stdcall_plugin("F:\programs\AviSynth+\plugins\yadif.dll")
#Yadif(mode=1)
#progressive frame DVD: no yadif here. caution - some DVDs have interlaced sections among progressive ones.



/* ============================== segments ==============================================
mecklogo:
0 - 0:18.451
menu:
0:18.451 - 2:23.905
stillpics:
2:23.905 - 2:24.005
introshow:
1:21:57.620 - 1:23:38.450
video+credits:
2:24.005 - 1:21:57.620
special:
1:23:38.450 - 01:26:50.503E
-- this list is for cutting video (see below) and audio (outside this scope)
===================================================================================== */

global rff=30000./1001. # make sure frame rate is correct

vv=last
pics = cut(vv,0,18.451, 0,18.483) + cut(vv,2,23.905, 2,24.010)
pics = Loop(pics,30,3,3)
pics = Loop(pics,30,2,2)
pics = Loop(pics,30,1,1)
pics = Loop(pics,30,0,0) # ... thus, ~4 seconds to add to audio!

# -- actual cut and splice command:
last = cut(vv, 0,0., 0,18.451) + pics + cut(vv, 81,57.730, 83,38.450) + cut(vv, 2,24.005, 81,57.620)\
  + cut(vv, 83,38.450, 86,50.503) + cut(vv, 0,18.451, 2,23.906)

# leave this out, when you won't edit the audio accordingly.
# --- end of segment handling ---




AddBorders(4,0,4,8,$7F7F7F)
ConvertToRGB32(matrix="PC.709",interlaced=false,chromaresample="lanczos4")
# default for this type of work (encoding and fine-filtering in AviDemux): PC.709 = HDTV
# this turns out the best result.
#  although officially most DVD = Rec601 !
#  Rec.709 should be used always, when your source is HDTV (but fits some DVDs also).
#  "Rec601" : Use 601 coefficients, scaled to TV range [16,235].
#  "PC.601" : Use 601 coefficients, keep full range [0,255].



# ------------------------- convolution and library of kernels ----------------------------------------------------
# -- symmetric kernels
C20 = "-2 0 3 0 -2  0 -1 -3 -1 0  2 -6 56 -6 2  0 -1 -3 -1 0  -1 0 3 0 -1"  #C2 entry level
C21 = "-1 0 3 0 -1  0 -1 -3 -1 0  3 -5 56 -5 3  0 -1 -3 -1 0  -1 0 3 0 -1"  #Crystal Outline
C22 = "-1 0 3 0 -1  0 -1 -3 -1 0  3 -6 48 -6 3  0 -1 -3 -1 0  -1 0 3 0 -1"  #
C23 = "-1 0 3 0 -1  0 -1 -3 -1 0  3 -5 48 -5 3  0 -1 -3 -1 0  -1 0 3 0 -1"  #
C24 = " 0 0 3 0 -0  0 -1 -3 -1 0  4 -4 44 -4 4  0 -1 -3 -1 0  0 0 3 0  0"  #base for C3
# -- for using symmetric kernels, borders and cropping need to be adjusted.

# -- asymmetric kernels with better compensation
# select one of these kernels, ordered by increasing radius and impact:
# C31 will do for most good quality DVDs with sufficient sharpness.
# C40 has strong halo softening and anti-ringing compensation, but only for DVDs that have correct bandwidth limiting.
# On pixel-sharp and high-contrast vertical lines it will still show a ringing pattern.
# Use C40 for sources that would look blurred after the reencoding, when they won't allow
# the stronger sharpening by the other kernels because of noise, grainy lines, and sharp halos/outlines.
# Most kernels are optimized for 960*640 upsampled format.

C000 = "0 0 0 0 100  0 0 0 0 0  0 0 0 0 0  0 0 0 0 0  0 0 0 0 0" #asymmetric dummy kernel (to keep the shifting)

C31  = "3 -4  8 -10 112 0  0 -0 -4 -15  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0" #C3.1 tight
C32  = "3 -3  9 -13 112 0  0 -0 -4 -15  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0" #C3.2 tight sharper
C32a = "3 -3  9 -13 140 0  0 -0 -4 -15  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0" #C3.2a tight sharper, less intensity
C32c = "3 -3 11 -13 173 0  0 -0 -4 -10  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0" #C3.2c for progressive DVDs! ###ok
C33  = "3 -3 10 -15 112 0  0 -0 -4 -12  0  0 -6 -0  14  0  0  0  0 -4  0  0  0  0  0" #C3.3 wide
C331 = "4 -5 14 -15 112 0  0 -0 -4 -12  0  0 -6 -0  14  0  0  0  0 -4  0  0  0  0  0" #C3.31 wide attenuated
C34  = "2 -5 12 -20 112 0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0" #C3.4 wide sharper ##

C40  = "3 -5 17 -21 112 0  0 -0 -4 -13  0  0 -6 -0  15  0  0  0  0 -4  0  0  0  0  0" #C4.0 natural look ##ok
C40a = "3 -5 21 -23 112 0  0 -0 -4 -12  0  0 -8 -0  14  0  0  0  0 -4  0  0  0  0  0" #C4.0a nat sharper
C40b ="-2 -3 19 -21 112 0  0 -0 -4 -13  0  0 -6 -0  15  0  0  0  0 -4  0  0  0  0  0" #C4.0b (experimental,+antiringing)
C40c = "3 -3 23 -23 112 0  0 -0 -4 -10  0  0 -7 -0  15  0  0  0  0 -5  0  0  0  0  0" #C4.0c 720x480 test
C40d = "3 -5 23 -25 112 0  0 -0 -4 -10  0  0 -7 -0  15  0  0  0  0 -4  0  0  0  0  0" #C4.0d 720x480 test
C40e = "3 -5 23 -25 140 0  0 -0 -4 -10  0  0 -6 -0  15  0  0  0  0 -4  0  0  0  0  0" #C4.0e 720x480 !### ok
#  -10..-15  13..15  # depends on interlacing artifacts
C41  = "3 -8  7 -11 112 0  0 -0 -6 -12  0  0 -3  2  14  0  0  0  0 -4  0  0  0  0  2" #C4.1 naturalsharp extreme for LQ DVD

CX7  = "2 -5 15 -20  96 0  0 -0 -6 -12  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0" #X7  wide strong

#these are softer and have a wide radius:
AC1 = "5 -13 17 -21 100  0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0"
AC2 = "5 -13 18 -21 103  0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0"
AC3 = "5 -13 15 -20 120  0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0"

GeneralConvolution(0, C32c) # now do it - select kernel to use, specify its variable name here


#-------------------- convolution tuning -------------------
#check eyes, teeth, keeping of skin texture, carpets, edges of doors, plant leaves, fonts.
#sometimes fonts and artgraphic can't have the same optimal kernel as the actual video, as of postproduction reasons.
#I'd go for organic looking actual video.
#some kernels may come so well adjusted to the source, that even halo and ringing will be lower while result looks sharper.

#rff=29.97*2 # factor *2 for 60fps  #see above for rff
#test slice definition:
# -- test slice start time = minutes*60. + ss.ddd (seconds.milliseconds)
tpos=2.*60.+15.
fpos=int(tpos*rff)+1
# -- length of test slice in secs:
tsl=int(40.*rff)+1
# -- for production, we comment off the following test section with /* .... */ markers.
/*
# -- list the kernels you like to test here, name them in subtitles:
Trim(fpos, -tsl).GeneralConvolution(0, C000).subtitle("raw" ) + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C31 ).subtitle("C31" ) + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C32a).subtitle("C32a") + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C34 ).subtitle("C34" ) + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C40 ).subtitle("C40" ) + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C40a).subtitle("C40a") + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C40e).subtitle("C40e") + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C41 ).subtitle("C41" ) + blankclip(last,15)
*/

Crop(0,0,-2,-2)

ConvertToYV12(matrix="PC.709",interlaced=false,chromaresample="lanczos4")

#AssumeFPS(30000,1001)



#--------------------------- helper funcs ---------------------------------------
function cut(clip In, int Mins, float secs, int Mins2, float secs2) {
# -- theclip,  minutes1, seconds1,  minutes2, seconds2 (from-to)
tpos = 60.*Mins + secs
te =  60.*Mins2 + secs2
Outclp=Trim(In,int(rff*tpos)+1,-int(rff*(te-tpos))-1)
return Outclp}

==> updated kernel library above!!


MyDemux.py

#PY  <- Needed to identify #
#--automatically built--

adm = Avidemux()
adm.loadVideo("::ADM_AVS_PROXY::")
adm.clearSegments()
adm.addSegment(0, 0, 5500000000)
adm.markerA = 0
adm.markerB = 5088719360
adm.videoCodec("x264", "useAdvancedConfiguration=True", "general.params=AQ=21", "general.threads=0", "general.preset=ultrafast", "general.tuning=film", "general.profile=baseline", "general.fast_decode=False", "general.zero_latency=False"
, "general.fast_first_pass=True", "level=-1", "vui.sar_height=27", "vui.sar_width=32", "MaxRefFrames=5", "MinIdr=25", "MaxIdr=250", "i_scenecut_threshold=40", "intra_refresh=False", "MaxBFrame=5", "i_bframe_adaptive=2"
, "i_bframe_bias=0", "i_bframe_pyramid=2", "b_deblocking_filter=True", "i_deblocking_filter_alphac0=-1", "i_deblocking_filter_beta=0", "cabac=True", "interlaced=False", "constrained_intra=False", "tff=True"
, "fake_interlaced=False", "analyze.b_8x8=True", "analyze.b_i4x4=True", "analyze.b_i8x8=True", "analyze.b_p8x8=False", "analyze.b_p16x16=True", "analyze.b_b16x16=True", "analyze.weighted_pred=2", "analyze.weighted_bipred=True"
, "analyze.direct_mv_pred=3", "analyze.chroma_offset=1", "analyze.me_method=2", "analyze.me_range=17", "analyze.mv_range=-1", "analyze.mv_range_thread=-1", "analyze.subpel_refine=9", "analyze.chroma_me=True"
, "analyze.mixed_references=True", "analyze.trellis=2", "analyze.psy_rd=1.100000", "analyze.psy_trellis=0.000000", "analyze.fast_pskip=True", "analyze.dct_decimate=True", "analyze.noise_reduction=0", "analyze.psy=True"
, "analyze.intra_luma=0", "analyze.inter_luma=0", "ratecontrol.rc_method=0", "ratecontrol.qp_constant=0", "ratecontrol.qp_min=4", "ratecontrol.qp_max=58", "ratecontrol.qp_step=8", "ratecontrol.bitrate=0"
, "ratecontrol.rate_tolerance=1.000000", "ratecontrol.vbv_max_bitrate=0", "ratecontrol.vbv_buffer_size=0", "ratecontrol.vbv_buffer_init=0", "ratecontrol.ip_factor=1.400000", "ratecontrol.pb_factor=1.400000"
, "ratecontrol.aq_mode=1", "ratecontrol.aq_strength=1.400000", "ratecontrol.mb_tree=True", "ratecontrol.lookahead=40")
adm.addVideoFilter("fluxsmooth", "temporal_threshold=29", "spatial_threshold=5")
adm.addVideoFilter("addBorder", "left=0", "right=2", "top=2", "bottom=2")
adm.addVideoFilter("swscale", "width=1600", "height=968", "algo=2", "sourceAR=2", "targetAR=2")
adm.addVideoFilter("chromashift", "u=-1", "v=-1")
adm.addVideoFilter("Sharpen", "luma=True", "chroma=True")
adm.addVideoFilter("Sharpen", "luma=True", "chroma=True")
adm.addVideoFilter("Sharpen", "luma=False", "chroma=True")
adm.addVideoFilter("contrast", "coef=0.950000", "offset=40", "doLuma=False", "doChromaU=False", "doChromaV=True")
adm.addVideoFilter("colorYuv", "y_gain=0.000000", "y_bright=0.000000", "y_gamma=3.000000", "y_contrast=10.500000", "u_gain=0.000000", "u_bright=0.000000", "u_gamma=0.000000", "u_contrast=12.000000", "v_gain=0.000000", "v_bright=0.000000"
, "v_gamma=0.000000", "v_contrast=0.000000", "matrix=0", "levels=0", "opt=False", "colorbars=0", "analyze=1", "autowhite=False", "autogain=False")
adm.addVideoFilter("swscale", "width=972", "height=652", "algo=2", "sourceAR=1", "targetAR=1")
adm.addVideoFilter("rotate", "angle=90")
adm.addVideoFilter("chromashift", "u=-1", "v=-1")
adm.addVideoFilter("rotate", "angle=270")
adm.addVideoFilter("crop", "top=6", "bottom=6", "left=4", "right=8")
adm.audioClearTracks()
adm.setContainer("AVI", "odmlType=1")

Here, fluxsmooth denoiser uses stronger settings. This is because of the MPEG2 artifacts, as the source DVD compression became quite lossy already with the rich detail in the video.

This was tested with S.B.K.D.0.1.3.2.
size: 1.65GB, duration 1:27, 960x640, color dynamics nicely tamed, color subsampling artifacts reduced, way sharper, progressive 30fps, encoding at 7-8fps on i7/2.6GHz
the left one is the new w/kernel C3.2c

 

[Kevin_san]

Make DGIndex create a WAV audio file (normally track 80), and insert some silence or music with an audio editor, with the correct length, when you create a slide show from the DVD special content. (some have, some not).
Naturally, the length of the audio should exactly fit the video length.
Use AviDemux in a second step, to add the audio to the video, and optionally use the audio delay setting (positive and negative is possible). Set audio compression to ac3.
For many videos, before the start of each chapter (at least: the video, and then the bonus scene making of etc, but sometimes every scene in the menu), the audio file needs an insert of 250ms of silence. (This is an outcome of what DGIndex is producing.)

----
GeneralConvolution(0, "3 -5 21 -23 112 0 0 -0 -4 -12 0 0 -8 -0 14 0 0 0 0 -4 0 0 0 0 0") # C4.0a
# new kernel, sharper than C4.0 - for a more modern DVD, that has very fine detail.
# for this one, disk space should be compared closer to 1280x720/60fps resolutions, than with 720x480, because of the fine interpolation of structures up to the pixel, and the technical 960x640 resolution. the source will likely show some ringing already, and in the result, the ringing artifacts have very fine lines with weak contrast, so to me this is tolerable, for what the whole thing is.

----
(probably for imperfections in AviDemux, when I put together a video from several segments, often it says "no sufficient timing info" or something.
for now, what I do is, create each segment in an AVI container, put them together into an AVI, during this mux in the ready edited audio, then throw it into MKVtoolnix for creating an MKV format (maybe check some enforcing timing option there), and only then, go back with that output to AviDemux, and have it make an MP4.
this is only anecdotally, and has nothing to do with usage of convolution.)

 

[Kevin_san]

(script update and new kernel library in posting #10. thx)

 

[SamKook]

I've been messing around with it for a few hours(sorry it took me so long to get to it), but I'm having trouble making a fair comparison with my usual process since your x264 settings are quite a bit lower(and avidemux is a pain to deal with to change them when you're used to do it by command line) than mine so everything always looks worse. I also forgot how much shimmer yadif produced in some cases which makes some scenes look horrible so I'll have to replace that for future testing. The only problem is that since I'm not very familiar whith your whole process, I don't know if making those changes will affect the efficiency of your processing.

Avidemux also adds 2 or 4 extra frames for some reason which is a bit worrisome to me. The frame rate is set as variable instead of constant as it should be, not sure why. My guess would be that because avidemux is not loading the avs script directly, it likely relies on directshow to load the output of avisynth proxy and that would destroy the frame accuracy DGindex is known for.

One other detail is bugging me. Why do you make the final size 952x640 instead of 720x480? I understand the first upsampling for processing purpose, but it's much better to let the video player upscale the video during playback(I don't use madvr for nothing) than to do it in encoding.

It would also be a good idea to set the proper aspect ratio directly in the stream rather than doing it later in the container since not everything supports it properly when it's in the container. The first thing that comes to mind are a lot of the thumbnail maker software.
I'm not 100% sure how to do it in avidemux since there's nothing called sample aspect ratio(the -SAR option in command line), the only way I found to make it work is to use the Predefined Aspect Ratio under the Pixel aspect ratio section in the output 1 tab of the x264 settings. For some reason, the Custom option doesn't actually set it right which is a problem if you don't have a standard AR.

 

[Kevin_san]

Thanks for taking a serious look at that, and for the detailed comment!

My goal was the cheapest possible process that works with templates.
Cheap also means, neither the tablet nor the business notebook (i7) have a usable GPU.
AviDemux has one advantage, it has implicit threading, so nothing to think about getting threading issues in the avs system work and updating all these libraries,
and this way AviDemux eats roughly 70-75% CPU, while Avisynth takes the other 20-25%.
Also, the tweaking of borders and chroma shift is still quickly possible but less nerdy than editing and reloading scripts.

The other goal is to have a set of kernels and surrounding processes, that come down to transcode something like ~5 typical notorious DVD camera and mastering types, such as shared on these blogs and forums.
A poor DVD is a poor DVD and it is moot to invest CPU time for the better algorithms. Only some of the typical flaws can be improved reasonably.
Many so-to-say good DVDs are being proliferated mostly in mp4 format only and these look good enough for most lurkers. But there are really poor examples, and I realized that they come much more often as full ISOs, probably because a default transcode looks so poor it is pointless to share. Or, some are just difficult with their complex spectrum.
So, convolution is the most powerful tool for these problems, as it is linear and it can at least find and gather some of the smeared detail, and it can even improve the "beauty" of the spectrum, i.e. reduce the halo, and remove some wide-radius blur.
Some mp4s of such difficult material end up shared with still ~2GB and still a bit blurred and still 30fps though it was interlaced and can deliver 60fps. Many others are huge because erraneously set to 60fps but the interlace was fake so there is no content for the double framerate anyway.
Thus I hope to inspire a few other users who want to improve a bit, but don't want to study the deeper issues of programming and DSP.

x264
You may try any bunch of settings and parameters, to me only two issues seemed important:
- motion search: the subpixel refinement and enhanced hexagon (this or better)
because of the strong sharpening and "organic" look
- intra luma/chroma quantisation deadzone
because I felt that skin texture comes back much better this way. I switch off this optimization because it can flatten the texture too much.
So there is no problem to choose even more precise settings. I just want to stay within 20MB disk space per minute. Often 10 only.

yadif
the issue is that yadif is very neutral to the spectrum. All other complex processes will mess with the spectrum, and have their own sharpening kernels, and nonlinear corrections.
Thus the kernels must be retuned for a complex "intelligent" deinterlacer.
But OTOH I can tweak the vertical component of my kernels to reduce some deinterlacing artifacts. This depends strongly on the material.

avidemux
yes, lots of issues, but I don't care that much because the material is not worth it. The frame rate chaos (jerky playback) can be cured with repackaging in MKVtoolnix.
The aspect ratio and frame timing in the stream is a matter of the encoder. Avidemux should pass the correct parameters to its internal x264 but I don't know.

aspect and upscaling
I found that VLC and Windows Media Player look very different when playing back the transcoded material, and also it depends too much on the screen. (drivers, GPU etc.)
The reason is the limited real-time capacity in upscaling, or in other words the precision of the low-pass "reconstruction filter".
My devices are just not doing 720*480 sharp enough! Windows player is sharper though.
But with 960*640, there is headroom enough, and no detail to ruin at the top of THAT spectrum. The difficult work has been done mostly at encoding/processing time.
So I have the benefit that the video looks very much the same on all devices and players, besides the obvious differences in brilliance and pixel count.
Further, with many videos it gives a nice subtle texture that competes the other artifacts (like deinterlacing) and looks just better to me, like a physical projection screen for good old analog home movie. Some kind of glittering.

The other issue with the upscaling and downscaling is the choice of the filter (e.g. lanczos vs. bilinear). With the bilinear filter and a different kernel I can improve halo issues a lot!! Still fully linear.
I will share the details of the anti-halo (anti-oversharpening) soon. Some basics are probably visible above already.

 

[Kevin_san]

How to perhaps benefit from my idea:

-Use the test routines to compare kernels (this would encode the same short segment repeatedly with a number of kernels, and show the kernel name as subtitle).
Cut a test segment that has different patterns like extra sharp edges, blurred patterns (like wooden textures), and scenes that show eyes, hands, jewels etc.
Sometimes gets best result with combining one asymmetric and one symmetric kernel.
Don't look for perfection, just that the video becomes clean and crisp enough to watch, and looks still natural, when it wasn't before.

-Find a better deinterlacer that is spectrally as neutral as possible, or experiment with explicitly replacing kernels within the deinterlacer (but has to be at least 5*5 matrix else it is pointless).
Should be multi-threaded.

-Dump AviDemux completely, but then you need to deal with thread optimization and thread-able filter DLLs and all those exotic Avisynth versions and their bugfixes. Else, processing time will e.g. quadruple.

-Do everything in 10-bit format or higher. Non-logarithmic value scales are even better. Convolution in logarithmic 8-bit is definitely incorrect, seen mathematically. Thus, some extra artifacts when contrast is high. But this is a problem practically with all resizers, unless they upsample to non-logarithmic hi-res.
Some complex deinterlaceing scripts are already available updated for this. But I haven't seen a n*n convolutional HD processor yet. There should be a GPU "shader" possible to create.

-Use the scientific "test impulse" procedures to mathematically compute convolutional kernels for certain exact identification-models. My kernels are heuristic multi-issue animals.
It would be interesting to create a quasistatic process identification for the usual deinterlacing artifacts, including yadif. Sure the big cats with the expensive video studio software have done this.
(the theories can be found in audio-acoustic technology like how to harvest hall-impulses. then proceed to 2-dimensional stuff like x-ray tomography. but that's all rocket science to most of us.)
-research suggestion:
A 3D modeling of a convolution should be entirely possible, for yadif, and for a general deinterlacing process. 2 coordinates are x/y, the third would be frame number, when we can model the temporal smoothing as a set of convolutional taps or coefficients. This makes sense because of the upper/lower frame quasistationary pattern. We would end up with at least four 2D kernels that are applied alternating. Frame n * kernel A and at the same moment add previous frame (n-1) * kernel B. Then frame (n+1) * kernel A1 plus frame n * kernel B1, etc.

(options are ordered by ascending difficulty )

 

[SamKook]

For using cheap settings, it certainly does a good job, the picture looks pretty good compared to what most fast encoders are doing.

The only problem is that a person with a good computer like myself doesn't get all of the benefit with your current process since with your default settings, the whole thing only uses 25% of my CPU. That means either the avisynth part is a bottleneck or the processing in avidemux isn't multithreaded(but the encoding itself is since x264 takes care of that). If I change the encoder settings to something close to what I'm usually using, I can reach roughly 75%.
The same principle is true for playback. Upscaling it has benefits if your going to play it back on a slow device since it will use the lowest quality scaling possible, but if you have a decently fast device, you could get a much better gain than what you're currently doing. madvr is a renderer for directshow players like mpchc and you can choose your scaling algorithm and add different processing to it. That way, someone with a crazy good GPU could use very demanding algorithms in real time. That's why I think that if you're going to release something to the public, it's a good idea to at least set the defaults to something that is the most flexible which would be leaving the final resolution as-is(for tools or encodes you release to the public, if you're encoding it for yourself, do whatever works best for you).
I personally built myself a "cheap" 700$ HTPC instead of using a 35-45$ raspberrypi just so I could use jinc scaling with a de-halo made specifically for it which I really like for playback after trying a different bunch.

One of the big benefit of using overall better settings(better meaning closer to the higher quality presets) is that you'll get a decent reduction in size for pretty much the same visual quality. I did a test a while ago encoding something using all of the presets and using the same CRF value and the results were quite surprising: https://www.akiba-online.com/threads/wmv-to-mkv-re-encoding-help.1082131/#post-2090532
I also tend to be sensitive to compression artifacts so I pretty much never use a CRF higher than 18 since that's my sweet spot to stop noticing a higher value looks worse. Of course better settings mean much more CPU power/time which isn't a concern of mine, but is for most people.

Yadif is better than a lot of deinterlacer, but QTGMC(I forgot what's the name of the deinterlacer it uses with the higher settings since some of the poor settings use yadif since it's much faster) does such a good job getting rid of most problems introduced by deinterlacing that I'm unable to accept anything less now.

The issues with avidemux are the same that most encoders so it's not that big of a deal, but I spent so much time trying to get the most reliable solution I could that I'm being a bit too much demanding nowadays.
Most software will use a 1:1 pixel aspect ratio by default which means the output will have the same AR as the input. The problem here is that DVDs always change the AR from 720x480(3:2) to either 853x480(16:9) or 640x480(4:3), but since you're using an avisynth script instead of the DVD files as a source, the encoder is unable to know that it's actually a DVD so it can't read what the proper AR is so it won't be able to correct it by itself which is why you have to set it manually.
It's not that big of a deal since most players nowadays support the AR flag in the container, but for the few cases it won't, it's better to do it during the encoding process since it has to be done eventually anyway so might as well do it when it can fix more potential problems.

BTW, the nnedi3 avisynth filter does wonders for upscaling(only by a power of 2 though) for doing extra processing in case you didn't know about it, but it's pretty demanding, like most good things are.


The thing with video encoding is that not everyone is sensible to the same problems so it's impossible to make something that is perfect for everyone. Personally, I can't stand interlacing and compression artifacts, but I can barely notice halos(so that's not something I can comment much about).

Edit: I haven't read your latest post yet as I post this.

 

[SamKook]

My problem is that I'm not familiar enough with convolution and some of the specifics of your other processing to mess too much with your script(for now at least, I'll certainly be looking up what exactly convolution does in the near future). I tried getting rid of avidemux and porting your processing to avisynth, but I couldn't get everything transferred and you're doing weird cropping and border adding(for example, you end up with only a partially cropped border on the left and right when you exit the avisynth part of your processing and the video is stretched by 2 extra pixels vertically if I remember right) so I'm not exactly sure if that's necessary for some of the steps or not. I wanted to keep only the convolution part to test it out, but that stopped me.

I can easily deal with the thread problems in the current version of my script since I can choose to split the video in multiple single threaded lossless encodes for the processing part(got the idea from vitreous, the guy who made QTGMC and did the MeguIVit upgrade) and then do the final lossy encoding at full speed without having to worry about anything slowing it down. The time it takes is roughly the same as when I used to deal with mutithreading avisynth, but it's much more stable without MT and since I usually do 3 different encodes, I save a lot of time that way since I only need to do the heavy processing once.

 

[Kevin_san]

* AviDemus 2.5.8, while doing some filtering before feeding its x264, it uses up 3 kernels here. So this is something useful. I can shift some filtering between the systems to balance thread usage.

* Cheap playback is an issue here, but the tablet can play 1920x720, 60fps smoothly. VLC seems not to expose a parameter for the scaling-filter quality. Though it has the usual postprocessing flags. It might be true that it switches to a sharper mode on a better CPU/GPU system.

* So far, I haven't tried madvr. It would be interesting to enforce a HQ screen scaler, to see how things look, even when it becomes jerky.

10 bit process:
I see some good synergy as you explained your work more in-depth. Definitely, our eyes are a bit different. For my part, I dislike blur and halos/ringing more than some 8-bit noise or minor MPEG artifacts.
I checked a certain 10bit mkv with 4 slim office ladies right now. ^_^
Very excellent with its smoothness and dark shades, but not extremely crispy here. My player system still shows some minor pixelisation, and when slanted horizontal lines, they may have some blurry stairs (watched with the looking glass) that are also traveling along the object.
That process is obviously complex, because all kinds of artifacts are blurred away with the right amount of blending, or are cured with the motion estimation, denoising, and nnedi neural decision matrix. But this also means, that with further sharpening, many artifacts would return. On microscopic view, for a sharp edge, the room or libration zone between some pixel n and pixel n+1 (of original 720px domain) is full with noise, which is not seen from a normal distance. This does not happen with my more primitive process.
So, for me the yadif I use, actually can add some bobbing, plus the notorious "running ants", when a line or gap is 1px wide and has a certain slant or tilt.
The other typical deinterlacer artifacts with my script can come out somewhat worse than the above mkv example, which was probably made with QTGMC and nnedi3. That added flaw by yadif does not matter much to me. Because the DVD is poor anyway, else I would not process it. More processing complexity to me would be worth while for a few videos only.

My point about yadif is, that the convolution and the side-effects of the 4/3 factor in result size (720 * (4/3) = 960) with many videos are strongly reducing those yadif artifacts, so it starts looking more like good old valve TV with its typical soft interlacing features (think of a good B&O), but more crispy. It needs tuning though, or testing by quickly switching through the library of kernels and compare the critical edges.


But what remains, for the synergy, is that I could find some wide-radius blur in that mkv, and a chroma shift!
Also, sometimes a halo and even ringing can show up. But these have been somewhat blurred off by complex algorithms. I look at it with a magnifying glass.

* With the right convo-kernel (before the more complex processing happens), and correct chroma reshift, that video I checked, probably could improve remarkably.
The wide-range blur will go away, colors will be crispy, and with luck, the halos and ringing patterns will be cancelled out. With time, I have learnt to adjust coefficients by hand against ringing. It's not very intuitive though but has its logic.

What should be tested, is to apply the convolution on the half-frames before deinterlacing. It will have some disadvantage in the vertical sharpness, but QTGMC can handle that as far as possible, without artifacts returning. We can decrease the vertical coefficients to apply some wide-range sharpening only (if there is required at all).


For the chroma shift, you might adapt the following function for 10 bit processing:

function CShft(clip In, float X, float Y) {
w = In.Width()
h = In.Height()
Tmp = In.BlackmanResize(w, h, -X, -Y, w-X, h-Y, 5)
In = MergeChroma(In, Tmp)
return In}

use it like:
CShft(last,-0.66,0.0)

The Blackmanfilter with 5 taps worked great for me, it also sharpens the color plane (but will increase subsampling errors for some videos, then use bicubic resizer).

x264 parameters:
Did you verify that the speed presets have any intrinsic functionality at all?
My theory about that is, they are just collections of the other values, so I can select any preset, but when I change every exposed particular parameter after, there will be no impact of that preset at all.
There are only two caveats regarding my way of processing:
a) low bitrate will simplify down some of the artifacts, thus the quality does not degrade as much as expected.
b) I especially want to switch the intra luma&chroma quantisation deadzones to 0, as this seems to improve skin texture. That mkv still has a little wax-paint feel. Else probably would eat up more data space.
So, you are totally free to choose the best encoding quality you like. Deadzones zeroing even improves quality.

Which parameters would you recommend me to change in my template, for better quality and smaller file-size? I mean, beyond the obvious primary quality value, that I set to 21, and you seem to prefer 18 or lower.

mpeg artifacts:
How do you deal with massive MPEG-2 artifacts in a heavily compressed DVD? This is the reason I don't care for encoding with higher settings. The quality loss of the encode around 21 is insignificant to the already existing artifacts, and my cheap way for that is temporal smoother as seen in the template, and the intrinsic functionality of the MPEG-4 encoder with its moving-block recognition and lazy-change policy. It will just leave out many artifacts because they are technically complex and would take too much bandwidth.
My goal, as said, is to process material with already poor quality, to make it look somewhat more pleasant, and reduce file size at least down to 30%.
But I am very sure the convolution here is the game changer, and it has still more power, if used in a HQ process.

my script and templates:
1. You may completely ignore all the sloppy border and crop settings. First, the border policy comes from the required chroma shift, plus the asymmetric convolution that shifts the picture.
It somehow works and I forget it, otherwise I tweak the final crop and the resizing factor, to maintain the geometry of the video, and to avoid black margins (but some videos lack width to fulfill 720 x 480, so we might end up with (960-w) x 640, upscaled to 4/3 mode, eg. 940x640. The 4-pixel-modulo restriction can play us here, and we result with green borders or losing 1 column or row of content when we crop them.
The elegant way would be to use some analysis on the source geometry (including to detect black margins and create auto-crop parameters), and compute automated border and crop parameters for the whole process already, by some code that calls a margin analyzer after reading the source video.
2. The convolution is a spectral thing, like a 5-band audio equalizer, and like those FFT tools. Imagine, that the screen is full with wide and narrow waves, that add together for the actual picture. Naturally, the kernel must harmonize with all parts of the process that change the spectrum. But this is for my predefined library. Of course, any other process too can be improved with a set of kernels that are a bit different from mine.
So, the resizing filters are an intrinsic part, means the choice of Lanczos (or Bilinear in a special dehalo application). The principle is, like in audio studio mixing, to combine a set of equalizers to put together a quite complex spectral change.
3. You may choose between
a) chroma shift in AviDemux (only integer pixels, but we have these upsampling factors to produce some distinct subpx shifts),
or b) in Avisynth to use a "CShft" function like what I have shown above. I made me one that sharpens chroma a little bit. Due to the 8bit data, it will increase noise.

What seems not fully clear, is the built-in convolutional sharpener in AviDemux. Without AviDemux, then it must be replaced with some corresponding kernel already in my script. Though I don't yet know its coefficients.
It is a very simplicist tactics, to add just 1 or 2 fixed one-trick sharpeners, to keep up with the blur that comes with the resizing. But it is taken into account with the overall process and harmonizes with the customized kernels.
Probably it will work quite well, when you replace the AviDemux sharpener with one of the symmetric kernels in my library. The chroma sharpening (by switching off luma in that filter) can often be ignored as the Blackman shifter will sharpen chroma anyway as it is applied on chroma only.
Upsampling is essential for chroma sharpening, else the chroma subsampling in the common data format will ruin all attempts anyway. This is one of the reasons, that I prefer 960x640. With some luck, colors look better. Because I have adjusted this myself to my own taste - which a HQ playback device won't probably do. madvr might be able to apply my choices in real-time, but I still have to tell it so, and it must recall different settings for different videos. But preprocessed means ready to watch.





You might just build your own solution, the main part being the kernel library and the _testing_ (skipping through kernels) with every video to be processed. It is meant to make aesthetic choices.

1. read video source file

2. a) yadif and then RGB + borders + asymmetric kernel (generalconvolution needs RGB)
or b) RGB + borders + asymmetric kernel (with less vertical processing) and then QTGMC

3. switch back to YUV or native format of the environment. (don't know what QTGMC prefers.)

4. you may do a CShift() at this position, using my function.

5. fluxsmooth or other temporal smoother/denoiser (obsolete with QTGMC that has denoiser settings)

6. upsize to 1600x968, this is a very heuristic choice but has some symmetry and analogy within the process.

7. use cheap integer-format chroma shifters here, they need not do any filtering at all. You can easily compute the resulting shift from the resizing parameters. It need not be too exact though, because there is a notorious fluctuation in chroma shift anyway.
8. use more sharpening here, and you will result in sharpening radius < 1 ! this is part of the game and makes sense only with upscaled result file. It has impact on halos. I will show a procedure for oversharpened videos soon. These will get blurred with radius = 1 but resharpened with radius < 1.
(We cannot beat Nyquist but edges will be drawn more exactly, mapped to the 4/3 scale, and everything looks rounder and smoother and still crispy.)
Choose a symmetric kernel for this type of sharpening.
9. You may do extra chroma sharpening here in the upsampled domain.
10. If required, extra colorspace and contrast corrections. It will mitigate rounding errors this way. What I have specified in the template, is corresponding with my use of the color space conversion.
What you told about aspect ratio is probably true for color space also. The AviDemux process is guaranteed to mess it up, but my corrections look pleasant enough. Yet to see if the 10bit color space convertors can live without extra corrections. A number of videos will come out wrong anyway probably by incorrect mastering.

My chroma shift settings worked for many videos, very few required extra tweaking. But the resulting chroma shift is depending on many other filters in the process that contribute to chroma errors because they are not commercial and don't do their own intrinsic and transparent upsampling.
My eyes are sensitive to chroma shift. Would be one of the simplest corrections to make.

11. Downsample to 960+(cropX1,2) x 640+(cropY1,2)

12. rotate + chromashift + rotate, using a cheap chromashifter for vertical errors. The CShft function with float-number parameters can make this obsolete.

13. crop this and that until there are no black and no green borders and the geometry comes close to the original.

----

 

[Kevin_san]

Example for oversharpened source:

(in short: use a kernel from the C50 group, and then do only one upsize to 960/640 with a bilinear filter, and then apply another simple sharpener like shown in the template.)

avs-script to use:

## check all hardcoded paths and replace with your project and configuration!

#-- define input video, here we use MPEG2 sources
LoadPlugin("F:\programs\AviSynth+\dgdecode.dll") ## Version 1.5.8 + corresponding DGIndex.exe
MPEG2Source ("F:\work\myproject.d2v",cpu=0,info=2)

#-- optionally, correct erraneous field-shift for fake-interlaced sources:
# (that's a quick and dirty method when the default would combine fields of two different frames.)
#SeparateFields
#Loop(0,0,0) #delete one extra field
#Weave

#-- Deinterlace Yadif -- delete this, when source is not actually interlaced i.e. shows combing without
Load_Stdcall_plugin("F:\programs\AviSynth+\plugins\yadif.dll")  # check path!
Yadif(mode=1)  # try to add order=1 or 0 when field order comes wrong

#AssumeFPS(30000,1001) # if we don't deinterlace
AssumeFPS(60000,1001) # if we use double-fps yadif, try for some videos. there may be false duration of the video, or jerky playback, set or remove this option according to results.
# MKVtoolnix can correct timing errors.
# we assume that the DVD audio was mastered with the standard video frame rate for NTSC.

## you may also add resampleFps: 59940/1000 in Avidemux



#---- define frame rate for usage in the script
global rff=60000./1001. # numeric representation of 60fps
o=0
/*
#---- we can activate a slide show here
vv=last
pics = cut(vv,0,18.451, 0,18.483) + cut(vv,2,23.905, 2,24.010) # example to cut out two slide shows from the source
pics = Loop(pics,30,3,3) # catch some pics and repeat these frames
pics = Loop(pics,30,2,2)
pics = Loop(pics,30,1,1)
pics = Loop(pics,30,0,0)
# example for cutting out scenes and inserting the slide show pics:
last = cut(vv, 0,0., 0,18.451) + pics + cut(vv, 81,57.730, 83,38.450) + cut(vv, 2,24.005, 81,57.620)\
  + cut(vv, 83,38.450, 86,50.503) + cut(vv, 0,18.451, 2,23.906)
*/


/*
#--- show some still pictures: (correct the audio intro length for this!), simpler example
tpos=0. # *60+10.176  # pics start time = minutes*60 + ss.ddd #(seconds)

fpos=int(tpos*rff)
tsl=int(1.*rff/2.)  # duration of one pic
n=fpos+0  # starting frame no. of pic series
Loop(tsl,n+20,n+21)
Loop(tsl,n+18,n+19)
Loop(tsl,n+16,n+17)
Loop(tsl,n+14,n+15)
Loop(tsl,n+12,n+13)
Loop(tsl,n+10,n+11)
Loop(tsl,n+8,n+9)
Loop(tsl,n+6,n+7)
Loop(tsl,n+4,n+5)
Loop(tsl,n+2,n+3)  # now show some DVD header pictures for half a second each
Loop(tsl,n+0,n+1)  # .. use the loop statements in decreasing order of frame position!
*/




AddBorders(4,0,4,8,$7F7F7F)

ConvertToRGB32(matrix="PC.709",interlaced=false,chromaresample="lanczos4")
# this turns out as the best setting for color conversion, in context with the AviDemus filters.
# facit: default for this type of work (encode and fine-filtering in AviDemux): PC.709 = HDTV in spite of converting DVD source.
# although most DVDs = Rec601 or should be!
# Rec.709 should always be used when your source is HDTV (but fits some DVDs also)
# "Rec601" : Use 601 coefficients, scaled to TV range [16,235].
# "PC.601" : Use 601 coefficients, keep full range [0,255].




# ------------------------- convolution and library of kernels ----------------------------------------------------
# -- symmetric kernels
C20 = "-2 0 3 0 -2  0 -1 -3 -1 0  2 -6 56 -6 2  0 -1 -3 -1 0  -1 0 3 0 -1"  #C2 entry level
C21 = "-1 0 3 0 -1  0 -1 -3 -1 0  3 -5 56 -5 3  0 -1 -3 -1 0  -1 0 3 0 -1"  #RioNatsume Crystal Outline
C22 = "-1 0 3 0 -1  0 -1 -3 -1 0  3 -6 48 -6 3  0 -1 -3 -1 0  -1 0 3 0 -1"  #ChiharuN(prog), Seina126
C23 = "-1 0 3 0 -1  0 -1 -3 -1 0  3 -5 48 -5 3  0 -1 -3 -1 0  -1 0 3 0 -1"  #SaraM
C24 = " 0 0 3 0 -0  0 -1 -3 -1 0  4 -4 44 -4 4  0 -1 -3 -1 0  0 0 3 0  0"  #Narumi 1800 very perfect ; c3 base

# -- asymmetric kernels
# -- select one of these kernels, ordered by increasing radius and impact:
#  C3.1 will do for most good quality DVDs with sufficient sharpness
C000 = "0 0 0 0 100  0 0 0 0 0  0 0 0 0 0  0 0 0 0 0  0 0 0 0 0" #dummy for asymmetric (to keep the shifting)

CX7  = "2 -5 15 -20  96 0  0 -0 -6 -12  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0" #X7

C31  = "3 -4  8 -10 112 0  0 -0 -4 -15  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0" #C3.1
C32  = "3 -3  9 -13 112 0  0 -0 -4 -15  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0" #C3.2
C32a = "3 -3  9 -13 140 0  0 -0 -4 -15  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0" #C3.2a
C32c = "3 -3 11 -13 173 0  0 -0 -4 -10  0  0 -6 -0  15  0  0  0  0 -5  0  0  0  0  0" #C3.2c
C33  = "3 -3 10 -15 112 0  0 -0 -4 -12  0  0 -6 -0  14  0  0  0  0 -4  0  0  0  0  0" #C3.3
C331 = "4 -5 14 -15 112 0  0 -0 -4 -12  0  0 -6 -0  14  0  0  0  0 -4  0  0  0  0  0" #C3.31
C34  = "2 -5 12 -20 112 0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0" #C3.4

# C40 group has strong halo softening and anti-ringing compensation, but only for DVDs that have correct bandwidth limiting.
# on pixel-sharp and high-contrast vertical lines it will still show a ringing pattern.
# Use C40 for sources that would look awkwardly blurred after the reencoding, when they won't allow
# the stronger sharpening by the other kernels because of noise, grainy lines, and sharp halos/outlines.

C40  = "3 -5 17 -21 112 0  0 -0 -4 -13  0  0 -6 -0  15  0  0  0  0 -4  0  0  0  0  0" #C4.0
C40a = "3 -5 21 -23 112 0  0 -0 -4 -12  0  0 -8 -0  14  0  0  0  0 -4  0  0  0  0  0" #C4.0a
C40b ="-2 -3 19 -21 112 0  0 -0 -4 -13  0  0 -6 -0  15  0  0  0  0 -4  0  0  0  0  0" #C4.0b natural+antiringing
C40c = "3 -3 23 -23 112 0  0 -0 -4 -10  0  0 -7 -0  15  0  0  0  0 -5  0  0  0  0  0" #C4.0c
C40d = "3 -5 23 -25 112 0  0 -0 -4 -10  0  0 -7 -0  15  0  0  0  0 -4  0  0  0  0  0" #C4.0d
C40e = "3 -5 23 -25 140 0  0 -0 -4 -10  0  0 -6 -0  15  0  0  0  0 -4  0  0  0  0  0" #C4.0e 720x480
#  -10..-15  14..15  # depends on interlacing
C41  = "3 -8  7 -11 112 0  0 -0 -6 -12  0  0 -3  2  14  0  0  0  0 -4  0  0  0  0  2" #C4.1 naturalsharp strong

# C50 group is for oversharpened sources that show a lot of halo already.
# This group should be used with the alternate AviDemux template that does only one upsize to 960x640 using _bilinear_ filter,
# and then inserts multiple (normally 2) sharpeners.
# The other kernels would worsen the artifacts in this case, here we hope that halos at least won't grow or even shrink a bit.

C50  = "4 -4  6  -3 121 0  0 -0 -2 -25  0  0 -4 -0  30  0  0  0  0 -7  0  0  0  0  5" #C5.0
C50b = "3 -5  3  -4 121 0  0 -0 -3 -15  0  0 -4 -0  15  0  0  0  0 -4  0  0  0  0  0" #C5.0b
C50c = "5 -3  11 -9 121 0  0 -0 -3 -21  0  0 -4 -0  23  0  0  0  0 -9  0  0  0  0  3" #C5.0c
C50c1= "7 -3  13 -9 121 0  0 -0 -3 -21  0  0 -4 -0  25  0  0  0  0 -9  0  0  0  0  3" #
C50d=  "5 -3  13 -9 121 0  0 -0 -3 -25  0  0 -4 -0  27  0  0  0  0 -4  0  0  0  0  2" #


#these are softer and have a wide radius:
AC1 = "5 -13 17 -21 100  0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0"
AC2 = "5 -13 18 -21 103  0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0"
AC3 = "5 -13 15 -20 120  0  0 -0 -4 -10  0  0 -6 -0  12  0  0  0  0 -4  0  0  0  0  0"

GeneralConvolution(0, C50) # select kernel to use, specify its variable name here



#-------------------- convolution tuning test routines -------------------
# check eyes, teeth, retaining of skin texture, carpets patterns, edges of doors, plant leaves, fonts.
# sometimes fonts and artgraphic can't have the same optimal kernel as the actual video, as of postproduction reasons.
# I'd go for an organic looking actual video.
# some kernels may come so well adjusted to the source, that even halo and ringing will be lower while result looks sharper.

#rff=29.97*2 # factor *2 for 60fps  #see above
#---test slice start time = minutes*60. + ss.ddd (seconds.milliseconds)
tpos=61.*60.+40.  # set starting position
fpos=int(tpos*rff)+1
# -- set length of test slice in secs:
tsl=int(20.*rff)+1
# -- for production, comment away the following test section with /* .... */
/*
# -- list the kernels to be tested here, name them in subtitles:
Trim(fpos, -tsl).GeneralConvolution(0, C21 ).subtitle("c21" ) + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C31 ).subtitle("C31" ) + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C32a).subtitle("C32a") + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C34 ).subtitle("C34" ) + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C40 ).subtitle("C40" ) + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C40a).subtitle("C40a") + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C32 ).subtitle("C32") + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C50 ).subtitle("C50") + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C50b ).subtitle("C50b") + blankclip(last,15)\
+Trim(fpos, -tsl).GeneralConvolution(0, C50c ).subtitle("C50c" ) + blankclip(last,15)
*/

Crop(0,0,-2,-2)

ConvertToYV12(matrix="PC.709",interlaced=false,chromaresample="lanczos4")

#CShft(last,-0.66,0.0)  # we can do a subpixel chroma shift correction here



#--------------------------- helper funcs ---------------------------------------
function CShft(clip In, float X, float Y) {
w = In.Width()
h = In.Height()
Tmp = In.BlackmanResize(w, h, -X, -Y, w-X, h-Y, 5)
In = MergeChroma(In, Tmp)
return In}

function cut(clip In, int Mins, float secs, int Mins2, float secs2) {
tpos = 60.*Mins + secs
te =  60.*Mins2 + secs2
Outclp=Trim(In,int(rff*tpos)+1,-int(rff*(te-tpos))-1)
return Outclp}

#==========================================================================================

template for AviDemux:

#PY  <- Needed to identify #
#--automatically built--

adm = Avidemux()
adm.loadVideo("::ADM_AVS_PROXY::")
adm.clearSegments()
adm.addSegment(0, 0, 9000000000)
adm.markerA = 0
adm.markerB = 1000000000
adm.videoCodec("x264", "useAdvancedConfiguration=True", "general.params=AQ=21", "general.threads=0", "general.preset=ultrafast", "general.tuning=film", "general.profile=baseline", "general.fast_decode=False", "general.zero_latency=False"
, "general.fast_first_pass=True", "level=-1", "vui.sar_height=27", "vui.sar_width=32", "MaxRefFrames=5", "MinIdr=25", "MaxIdr=250", "i_scenecut_threshold=40", "intra_refresh=False", "MaxBFrame=5", "i_bframe_adaptive=2"
, "i_bframe_bias=0", "i_bframe_pyramid=2", "b_deblocking_filter=True", "i_deblocking_filter_alphac0=-1", "i_deblocking_filter_beta=0", "cabac=True", "interlaced=False", "constrained_intra=False", "tff=True"
, "fake_interlaced=False", "analyze.b_8x8=True", "analyze.b_i4x4=True", "analyze.b_i8x8=True", "analyze.b_p8x8=False", "analyze.b_p16x16=True", "analyze.b_b16x16=True", "analyze.weighted_pred=2", "analyze.weighted_bipred=True"
, "analyze.direct_mv_pred=3", "analyze.chroma_offset=1", "analyze.me_method=2", "analyze.me_range=17", "analyze.mv_range=-1", "analyze.mv_range_thread=-1", "analyze.subpel_refine=9", "analyze.chroma_me=True"
, "analyze.mixed_references=True", "analyze.trellis=2", "analyze.psy_rd=1.100000", "analyze.psy_trellis=0.000000", "analyze.fast_pskip=True", "analyze.dct_decimate=True", "analyze.noise_reduction=0", "analyze.psy=True"
, "analyze.intra_luma=0", "analyze.inter_luma=0", "ratecontrol.rc_method=0", "ratecontrol.qp_constant=0", "ratecontrol.qp_min=4", "ratecontrol.qp_max=58", "ratecontrol.qp_step=8", "ratecontrol.bitrate=0"
, "ratecontrol.rate_tolerance=1.000000", "ratecontrol.vbv_max_bitrate=0", "ratecontrol.vbv_buffer_size=0", "ratecontrol.vbv_buffer_init=0", "ratecontrol.ip_factor=1.400000", "ratecontrol.pb_factor=1.400000"
, "ratecontrol.aq_mode=1", "ratecontrol.aq_strength=1.400000", "ratecontrol.mb_tree=True", "ratecontrol.lookahead=40")
adm.addVideoFilter("crop", "top=2", "bottom=4", "left=2", "right=4")
adm.addVideoFilter("swscale", "width=960", "height=640", "algo=0", "sourceAR=1", "targetAR=1")
adm.addVideoFilter("Sharpen", "luma=True", "chroma=True")
adm.addVideoFilter("Sharpen", "luma=True", "chroma=True")
adm.addVideoFilter("contrast", "coef=0.970000", "offset=40", "doLuma=False", "doChromaU=False", "doChromaV=True")
adm.addVideoFilter("colorYuv", "y_gain=0.000000", "y_bright=0.000000", "y_gamma=2.500000", "y_contrast=11.000000", "u_gain=0.000000", "u_bright=0.000000", "u_gamma=0.000000", "u_contrast=11.300000", "v_gain=0.000000", "v_bright=0.000000"
, "v_gamma=0.000000", "v_contrast=0.000000", "matrix=0", "levels=0", "opt=False", "colorbars=0", "analyze=1", "autowhite=False", "autogain=False")
adm.addVideoFilter("rotate", "angle=90")
adm.addVideoFilter("chromashift", "u=-1", "v=-1")
adm.addVideoFilter("rotate", "angle=270")
adm.audioClearTracks()
adm.setContainer("AVI", "odmlType=1")

The resizer should use the bilinear filter, and then apply one or two built-in convo sharpeners. This way, the halos get blurred off, but details that have different frequency, will be restored.

Caution: some videos don't require the vertical chroma shift.

----
addendum to previous examples:
You may use two kernels, one asymmetric and one symmetric, when the video was extra blurred.
Something like
GeneralConvolution(0, C40a)
GeneralConvolution(0, C21)

 

[SamKook]

Sorry again for the very delayed response(it's crazy how fast time flies by), I really don't have as much time as I'd like to spend on encoding stuff.

I wanted to wait until I had more time test stuff and answer more in depth but I don't know when that'll be so here's some quick answers for some of what you said.

Instead of blending the pictures, QTGMC interpolate the missing half to keep the temporal resolution(I don't remember what yadiff does for this, but it's not interpolating it for sure) which makes things more fluid since that isn't lost, but interpolation isn't perfect so things don't always line up perfectly as a result.

You're also right in saying that my encodes could be sharper. I usually stick with my default settings instead of spending a lot of time fine tuning them properly and since many of the source I encode often have disappointing quality(quite a common thing sadly for JAV, IV are much better), I prefer to have a little more blur than having potentially more artifacts shown.

For heavily compressed DVDs, I'll usually use lower settings(21 is my usual goto for low quality sources or re-encodes of already encoded DVDs/digital releases). I'd have to introduce some blur to hide the compression artifacts and keep the higher setting so that would be kind of pointless since the quality would be reduced either way.

As for the x264 preset, you are correct in assuming all they are is a collection of other settings since that's exactly what they are and modifying a setting manually afterwards will indeed overwrite what the preset specified.
If you execute x264.exe with the --fullhelp parameter, you can see exactly which options they change:

      --preset <string>       Use a preset to select encoding settings [medium]
                                  Overridden by user settings.
                                  - ultrafast:
                                    --no-8x8dct --aq-mode 0 --b-adapt 0
                                    --bframes 0 --no-cabac --no-deblock
                                    --no-mbtree --me dia --no-mixed-refs
                                    --partitions none --rc-lookahead 0 --ref 1
                                    --scenecut 0 --subme 0 --trellis 0
                                    --no-weightb --weightp 0
                                  - superfast:
                                    --no-mbtree --me dia --no-mixed-refs
                                    --partitions i8x8,i4x4 --rc-lookahead 0
                                    --ref 1 --subme 1 --trellis 0 --weightp 1
                                  - veryfast:
                                    --no-mixed-refs --rc-lookahead 10
                                    --ref 1 --subme 2 --trellis 0 --weightp 1
                                  - faster:
                                    --no-mixed-refs --rc-lookahead 20
                                    --ref 2 --subme 4 --weightp 1
                                  - fast:
                                    --rc-lookahead 30 --ref 2 --subme 6
                                    --weightp 1
                                  - medium:
                                    Default settings apply.
                                  - slow:
                                    --b-adapt 2 --direct auto --me umh
                                    --rc-lookahead 50 --ref 5 --subme 8
                                  - slower:
                                    --b-adapt 2 --direct auto --me umh
                                    --partitions all --rc-lookahead 60
                                    --ref 8 --subme 9 --trellis 2
                                  - veryslow:
                                    --b-adapt 2 --bframes 8 --direct auto
                                    --me umh --merange 24 --partitions all
                                    --ref 16 --subme 10 --trellis 2
                                    --rc-lookahead 60
                                  - placebo:
                                    --bframes 16 --b-adapt 2 --direct auto
                                    --slow-firstpass --no-fast-pskip
                                    --me tesa --merange 24 --partitions all
                                    --rc-lookahead 60 --ref 16 --subme 11
                                    --trellis 2

It's been a quite a few years since I went through exactly what each parameter does so I'm not sure which have the most effect, but I'd have used at least all of the values for the preset. The --me and --merange have a fairly big effect if I remember right so I'd at least change those.