JavLuv JAV Browser

[TmpGuy]

I figured out the last major known issue blocking release. There are undoubtedly still lots of issues to find and fix, but I think it's good enough to at least replace the current broken functionality. You'll be seeing update notifications in JavLuv, so feel free to try it out and let me know how things work for you. If there are issues, I'll take a look at your logs to see what might be going on.

 

[lock_on]

First of all, thank you for the update, I know this is not an easy thing to do.
This is what I experience with the new update: First thing I do is opening javlibrary to check the captcha from browser and do the scanning in JavLuv, I scan around 60 movies and still only got around 30 metadatas. Then I tried to open javlibrary from browser, but it didn't ask captcha and I can open the website normally. I tried to regenerate the movies which don't have the metadatas with no result. By the way, I'm using vpn connection.
Can you tell me what is the difference between the old and new version of JavLuv regarding the javlibrary scraper? I think you had mentioned that we need to occasionally open javlibrary in browser to check the captcha, I have done it but nothing happened.

 

[TmpGuy]

First of all, thank you for the update, I know this is not an easy thing to do.
This is what I experience with the new update: First thing I do is opening javlibrary to check the captcha from browser and do the scanning in JavLuv, I scan around 60 movies and still only got around 30 metadatas. Then I tried to open javlibrary from browser, but it didn't ask captcha and I can open the website normally. I tried to regenerate the movies which don't have the metadatas with no result. By the way, I'm using vpn connection.
Can you tell me what is the difference between the old and new version of JavLuv regarding the javlibrary scraper? I think you had mentioned that we need to occasionally open javlibrary in browser to check the captcha, I have done it but nothing happened.

The difference is that the old scraper just retrieved static HTML. The new scraper uses an embedded web browser component, which can do things like decode Javascript and bypass landing pages, etc. In theory, it should be more robust, but there are also some additional complications to deal with. The code that parses the HTML code hasn't really changed much, except to fix up some errors I found.

Using advanced options, I'd recommend trying "Regenerate metadata" on a few movies that are failing. If they continue to fail, send me the logs, and I'll see what errors are being reported. If necessary, I'll add additional logging to try to isolate errors more precisely.

Also, yes, you'll eventually need to solve captchas on some sites like JavLibrary. I'm hoping to eventually detect and display those in JavLuv, but I figure I'll try to first get the basics working before I add new features.

 

[lock_on]

I wait a few minutes until javlibrary asked the captcha in the browser and retried scanning the 30 movies again. I got around 10 metadata, the other 20 movies still return nothing. Also, I noticed that the scanning takes a lot more data than before, I think it takes around 20MB for 1 metadata, is that normal?

I attach the log from my computer, maybe it can help you figuring something you need.

 

[TmpGuy]

I wait a few minutes until javlibrary asked the captcha in the browser and retried scanning the 30 movies again. I got around 10 metadata, the other 20 movies still return nothing. Also, I noticed that the scanning takes a lot more data than before, I think it takes around 20MB for 1 metadata, is that normal?

I attach the log from my computer, maybe it can help you figuring something you need.

I'll check out the logs and see if I can figure out what's going on.

And yeah, it's not unexpected that it chews up more bandwidth. JavLuv is loading entire sites now, images and movies included, instead of just the HTML. Depending on results, the program may be scraping up to four sites and combining the results. It's kind of ironic that the anti-scraping features are forcing everyone to use a lot more bandwidth. Then again, I kind of suspect that wasn't done because of programs like JavLuv, but to block competing websites that tend to scrape the entire site in bulk.

 

[lock_on]

Just now I tried reconnecting my vpn connection and I got another 8 metadatas. I don't know what's going on with this, I will try some other workaround and inform you if I found something useful as information.

I attach my latest log.

 

[Fetterbr]

Thanks for the update- I added some movies with different extension to the ones i already have. normally it would come and say i have different version and let me say which one I want but now it doesn't say anything.

 

[TmpGuy]

Thanks for the update- I added some movies with different extension to the ones i already have. normally it would come and say i have different version and let me say which one I want but now it doesn't say anything.

Note that you'll only get a notification if the movie is equal or better resolution. If the new movie is a lower resolution (even by a pixel or two), nothing will happen. You might want to double-check that. Also, you need to make sure this option is checked, of course:



I just tried importing a movie with better resolution, and it seemed to work fine for me, so not sure why it didn't work for you. That code really shouldn't have been changed at all.

 

[Fetterbr]

Note that you'll only get a notification if the movie is equal or better resolution. If the new movie is a lower resolution (even by a pixel or two), nothing will happen. You might want to double-check that. Also, you need to make sure this option is checked, of course:

View attachment 3522010

I just tried importing a movie with better resolution, and it seemed to work fine for me, so not sure why it didn't work for you. That code really shouldn't have been changed at all.

im will check again, now we talk about auto import. I am changing my movies extension to eigher mp4 or mkv to make the size smaller and to add the cover tag into the movie. When i am importing and the system asks me, the only thing i can see is the different file extention, is it posible to add the info like in win explorer that it lets you see if one of the files has a tag cover and the size would be realy helpful. thanks

 

[TmpGuy]

im will check again, now we talk about auto import. I am changing my movies extension to eigher mp4 or mkv to make the size smaller and to add the cover tag into the movie. When i am importing and the system asks me, the only thing i can see is the different file extention, is it posible to add the info like in win explorer that it lets you see if one of the files has a tag cover and the size would be realy helpful. thanks

I'd eventually like to add a custom dialog box that lays out more information in a better format. But it's not a huge priority at the moment.

If you're not changing resolution, only renaming the filename or extension, you could also use "Rescan files" to do this as well. If you add or delete files (for example, merging a two-part movie and replacing it with one file), it's useful for this as well.

 

[DScott]

I figured out the last major known issue blocking release. There are undoubtedly still lots of issues to find and fix, but I think it's good enough to at least replace the current broken functionality. You'll be seeing update notifications in JavLuv, so feel free to try it out and let me know how things work for you. If there are issues, I'll take a look at your logs to see what might be going on.

<----Bowing in gratitude..

 

[DScott]

Ok, General question for Tmpguy but if anyone knows the answer please let me know. I tried the new iteration of Javluv on a folder with about 2K files in it. At some point it failed/crashed. When I restarted there were several Hundred titles itemized but no Actress image data. So I carried on with the cataloging of that same file and it continued itimizing where it left off assuming that when it was complete it would dl all of the actress images including what was already cataloged pre-crash. When it was complete it , as it is supposed to , downloaded the Actress images. However it did not download any actress data for the files that were cataloged pre-crash. So, is this the normal response, IE I start cataloging, close the program, re-open and continue but only the actress data from the 2nd round will be catalogged? The catalogging process is incredibly slow, it took me about 2 days to dl the data for 2k files and that is very likely because of my brutally slow internet (8mbit/sec so about 1Meg/Sec). Now, the reason that I'm asking this is I am now in the process of catalogging about 17K files. I'm figuring that this is going to take at least 2 Weeks going steadily based on the speeds I saw with my previous scan. Ok, the question. If I break off a scan mid-way, am I going to miss out on all of the actress images for anything that was pre reboot? Thanks for the help.. and of course, Thank you so much Tmpguy for your hard work..

 

[TmpGuy]

So, the new scraper is unfortunately slower than the old one, because we're reloading the full site instead of only the HTML. I've also been more generous with timeouts before giving up on a site, because it's hard to tell if it's failed completely or just isn't finished loading yet. I can't assume everyone's connection is as fast as mine. Or a site just may be slow.

Unfortunately, because downloads happen in two phases, movies, then actresses, yes, if JavLuv is interrupted during scraping movies, it won't scrape actress data. You can partially mitigate this by trying to re-generate movie metadata. Or, you can individually update actress data when looding at actress data on movie details.

There's not a great solution here. I'm hoping to optimize the scraping operation and make it more reliable over the coming months as I get more feedback. My recommendation would be to try to scrape smaller sets of movies in the meantime in case this happens. I'll keep you guys posted on possible improvement. I'm going to add more logging and clean up a bit of parsing code to try to fix a few observed issues. Hopefully will have an update by next weekend.

 

[bestbuy]

So, the new scraper is unfortunately slower than the old one, because we're reloading the full site instead of only the HTML. I've also been more generous with timeouts before giving up on a site, because it's hard to tell if it's failed completely or just isn't finished loading yet. I can't assume everyone's connection is as fast as mine. Or a site just may be slow.

Unfortunately, because downloads happen in two phases, movies, then actresses, yes, if JavLuv is interrupted during scraping movies, it won't scrape actress data. You can partially mitigate this by trying to re-generate movie metadata. Or, you can individually update actress data when looding at actress data on movie details.

There's not a great solution here. I'm hoping to optimize the scraping operation and make it more reliable over the coming months as I get more feedback. My recommendation would be to try to scrape smaller sets of movies in the meantime in case this happens. I'll keep you guys posted on possible improvement. I'm going to add more logging and clean up a bit of parsing code to try to fix a few observed issues. Hopefully will have an update by next weekend.

While I noticed that the new scraper is slower it is more accurate and scrapes more metadata especially actress's. I really appreciate the current fix and looking forward to the updates.

 

[DScott]

So, the new scraper is unfortunately slower than the old one, because we're reloading the full site instead of only the HTML. I've also been more generous with timeouts before giving up on a site, because it's hard to tell if it's failed completely or just isn't finished loading yet. I can't assume everyone's connection is as fast as mine. Or a site just may be slow.

Unfortunately, because downloads happen in two phases, movies, then actresses, yes, if JavLuv is interrupted during scraping movies, it won't scrape actress data. You can partially mitigate this by trying to re-generate movie metadata. Or, you can individually update actress data when looding at actress data on movie details.

There's not a great solution here. I'm hoping to optimize the scraping operation and make it more reliable over the coming months as I get more feedback. My recommendation would be to try to scrape smaller sets of movies in the meantime in case this happens. I'll keep you guys posted on possible improvement. I'm going to add more logging and clean up a bit of parsing code to try to fix a few observed issues. Hopefully will have an update by next weekend.

Thanks TmpGuy. The problem is definately at my end with this terrible internet (absolutely no choice for better) In any case my speeds are terrible and additionally I am also mass downloading, one and a time with Internet Download Manager, Hundreds of files that I lost to my paltry connection is sharing my massive BW between Javluv, Internet Download Manager, and the occasional general internet traffic so it is not a surprise that my speeds are negatively exagerated. That said, thank you for the clarification. I had hoped to mass catalog a couple of big folders that exist on my 18TB drive in one big scan because the drive is seperated into (series folders) which account for perhaps 2 TB and the remaining 15 or so TB is is one hugh, clusterfuck of unsorted titles. Following your sage advice I will likely divide that folder into a series of smaller folders so if I have any issues at least I'll only have to deal with a much smaller effect. Thanks much for the response Tempguy. Oh, one last thing, with respect to the 'crash', I think that it was just a peculiar series of events that caused that. I have never seen this issue with Javluv and I have scanned around 30K files in one go so I don't think this was aproblem with the software// Cheers

 

[bestbuy]

Thank you TmpGuy for doing 3 releases 1.2.0, 12.1, 1.2.2 in 5 days that was a huge amount of work

 

[Fetterbr]

How does this Locked files work ?

 

[TmpGuy]

How does this Locked files work ?

I assume you're talking about the update notes?

If you happen to be watching a movie that is being moved / renamed, the operation would normally fail. I've added a specific pre-check to see if any movies are locked, meaning they're probably being played. I got tired of doing this to myself, so added this safety check.

 

[Moxy]

@TmpGuy Tested the new scraper with 22 new movies and all released within the last year. Im on 900mbps/900mbps and roughly get 112MB a second each way but its taken like 12 minutes to scrape 4. I use a VPN but ive tried it both ways.

Even though its taking ages, It seems to be scraping the correct info which is a Huge Win.

 

[TmpGuy]

@TmpGuy Tested the new scraper with 22 new movies and all released within the last year. Im on 900mbps/900mbps and roughly get 112MB a second each way but its taken like 12 minutes to scrape 4. I use a VPN but ive tried it both ways.

Even though its taking ages, It seems to be scraping the correct info which is a Huge Win.

Yep, the new scraper is agonizingly slow, but definitely seems more reliable. I'll be looking into some ways of improving the scraping speed next.