JAV database

[ding73ding]

I explored some options but gave up due to a few factors: lack of interest, lack of (my own) time and economy (most free hosts don't allow adult materials). Potentially the economy could work out as income (ad etc) should more than cover the cost (there are many many commercial hosts that are sex-themed) but for those I definitely don't have the time to manage.

I would throw my support behind you (or anyone) who can get a project off the ground. I can send you my code (which is probably broken by now) I can send your my database (which needs re-building bad).

It seems now there are a lot of websites, in addition to DMM/R18, that offers nearly complete JAV listing with detail info in Japanese and English. Actually R18 nowadays is a lot more complete than 1-2 years ago.

So while I don't want to discourage you, I'd suggest thinking hard about what and why you will DIY a JAV database project:

1. to really streamline the XBMC/Kodi/home theatre experience. Scenario: I'm watching the latest Naked Bus Tour Guide with actress X, I think hey I want to check out previous vids of this series in my collection, I check the info page of this vid, click on the suitable genre or series or studio, and Kodi shows me all the cover thumbnails that share that genre or series. 2 more clicks and I'm watching whichever vid that catch my attention. Another case, I'd like to browse actress by head-shots (with English and Japanese names), clicking on that actress bring up a list of her vids in my collection, presented as cover thumbs, click the cover to watch that vid. Kodi does it seamlessly for non-adult movies and tv shows, with scrapping from IMDB and other sites.

(why it's not such as needed anymore) compared to 5 years ago, good-enough JAVs are being released so fast that I can't keep up with downloading (the easy part) and watching (the harder part) even half of them. So yes often I have the urge to rewatch some all time favs and/or do a historic review, but the effort to make that possible, seems just tempting to work on the backlog of unwatched JAV. Another way to say it: JAV has become something to pick up, watch, and forgot on a daily or weekly basis, there's no more idea of collecting and enjoying and re-enjoying the good vids for months and years.

Also, even as powerful new hardware is making a serious home theatre so cheap and accessible now, Kodi (or anything comparable) isn't getting any more traction in either the mass or geek market.

2. With JAV (English) database already so common now, look for a different angle on it: in addition to a database centred around the vids/films/DVD, build something extra on top of it such as actress biography/filmography/gossip/career trend. Especially great if it can incorporate some social network functions and becomes a community.

 

[billy_z]

I explored some options but gave up due to a few factors: lack of interest, lack of (my own) time and economy (most free hosts don't allow adult materials). Potentially the economy could work out as income (ad etc) should more than cover the cost (there are many many commercial hosts that are sex-themed) but for those I definitely don't have the time to manage.

2. With JAV (English) database already so common now, look for a different angle on it: in addition to a database centred around the vids/films/DVD, build something extra on top of it such as actress biography/filmography/gossip/career trend. Especially great if it can incorporate some social network functions and becomes a community.

Appreciate your feedback. I agree with your opinions. R18 has improved much since 3 years ago.
I'll spend a little bit of time and see what I can get.

 

[LordPhantom]

Hi to everyone.
I have realized for fun a local web based db for my Jav collection, more of 6.000 file.
It run in a portable XAMPP server.
It take the new vids, controls if they already exist in the db, scrap information from JavLibrary and download the covers and put the files in the right locations.
With a normal browser I can see two kind of table (you can see in the pics) make search in the db on all items, launch the videos or enlarge the covers and see the records of single file.
It is fully functional but not ended.
It's developed in PERL with some cgi that do all work and the DB is a csv plain text file.
here the screenshots: https://www.akiba-online.com/media/albums/javdb-pics.418/
If somebody want to help for more develop (now the scraper work only for censored dvd because JavLibrary contain only that) is welcome.

 

[ding73ding]

Ciao, LP, eccelente!

For scraping uncensored vids, consider javbus.me It has become my one stop shop for researching both censored and uncensored vids.

My selfish concern is how to port the data to Kodi (XBMC), meaning I don't prefer to use a browser as my video launcher. (but I have to say, the new gen of low cost PC running Win10 maybe changing my calculations)

May I ask what's the format you use for the local DB?

I hesitate to offer my help... I've put down Perl for 12+ years already. But... adding uncensored scrapping, adding sorting (to search/browse), adding ranged search and advanced search (seems not yet available based on your screens) and most selfishly, adding bilingual to Title and Cast, all seems so tempting... I don't even have an updated book on Perl on my bookshelves.

Oops I was too excited to read, you clearly stated "csv plain text" dude you are my kind of geek.

 

[CodeGeek]

Hi to everyone.
I have realized for fun a local web based db for my Jav collection, more of 6.000 file.
It run in a portable XAMPP server.
It take the new vids, controls if they already exist in the db, scrap information from JavLibrary and download the covers and put the files in the right locations.
With a normal browser I can see two kind of table (you can see in the pics) make search in the db on all items, launch the videos or enlarge the covers and see the records of single file.
It is fully functional but not ended.
It's developed in PERL with some cgi that do all work and the DB is a csv plain text file.
here the screenshots: https://www.akiba-online.com/media/albums/javdb-pics.418/
If somebody want to help for more develop (now the scraper work only for censored dvd because JavLibrary contain only that) is welcome.

Looks interesting. Best wishes for the project.
I guess the competition is getting tougher. I should really work harder on my solution.

 

[Casshern2]

Wouldn't it be cool if there were options like this? I'm sure there are people who wouldn't mind a simple bare bones solution (something I'd have the time and patience to code) and people looking for a more polished or robust solution like the ones highlighted in this thread. I think it is great that these are in development.

 

[LordPhantom]

For scraping uncensored vids, consider javbus.me It has become my one stop shop for researching both censored and uncensored vids.

Good suggest JavBus.me, I have seen our pages, very simple integrate the research in.

My selfish concern is how to port the data to Kodi (XBMC), meaning I don't prefer to use a browser as my video launcher. (but I have to say, the new gen of low cost PC running Win10 maybe changing my calculations)

I use Kodi in the main TV, but I don't have tried to port the DB on, but I think is possible integrate it.
But the choice of a web server is for the simply build of a beautiful interface and the possibility to see it by wifi in all the home with a PC portable or by tablet and don't have the problem to mixing on the same device Cinema movies and Jav movies.
The video are visible in streaming, very simple.

I hesitate to offer my help... I've put down Perl for 12+ years already. But... adding uncensored scrapping, adding sorting (to search/browse), adding ranged search and advanced search (seems not yet available based on your screens) and most selfishly, adding bilingual to Title and Cast, all seems so tempting... I don't even have an updated book on Perl on my bookshelves.

No problem, also a brainstorming on the feature that I can integrate on is very useful.

Looks interesting. Best wishes for the project.
I guess the competition is getting tougher. I should really work harder on my solution.

Thanks! But I don't think is a competition, but different ways for coming at different solutions for the same question.

 

[CodeGeek]

Wouldn't it be cool if there were options like this? I'm sure there are people who wouldn't mind a simple bare bones solution (something I'd have the time and patience to code) and people looking for a more polished or robust solution like the ones highlighted in this thread. I think it is great that these are in development.

Sounds like some Linux based appliance. Or at least a modified Linux version which could run on something like that.
I have some Linux knowledge, but there a lot of people who know a lot more than me about it.
And I'm already busy with my project. But maybe some collaboration would be possible.

[...]
Thanks! But I don't think is a competition, but different ways for coming at different solutions for the same question.

Please, don't get me wrong. When I wrote "competition" I don't mean it in a serious way. I thought more about that competition who friends or acquaintances have when they meet for a cosy evening having some snacks and playing e.g. Mario Kart against each other.

 

[LordPhantom]

Sounds like some Linux based appliance. Or at least a modified Linux version which could run on something like that.
I have some Linux knowledge, but there a lot of people who know a lot more than me about it.
And I'm already busy with my project. But maybe some collaboration would be possible.

Is a server web based local appliance, that may run under many OS like a portable app.
It run on the computer and you can use them simply calling localhost by your preferred web browser.

Please, don't get me wrong. When I wrote "competition" I don't mean it in a serious way. I thought more about that competition who friends or acquaintances have when they meet for a cosy evening having some snacks and playing e.g. Mario Kart against each other.

Yes, I know that! I'm was joking.
In the next 2 weeks I try to make a demo downloadable, but I think for this is better that I open a new thread, I don't want make confusion in your thread.

 

[ding73ding]

Okay, I will bite, send me the project file. I will take a look. Why is it so big?? Do you mean the XAMPP download is 106 MB? I'm curious if it could run on the NAS where I store JAV, cinema movies, family photos. If it does (Linux version of XAMPP) and your implementation is pure Perl/CGI and HTML then it's very attractive.

Kodi already give me a beautiful interface (actually many interfaces) for each TV in the home and hides the adult stuffs from the kids by a password-protected profile. Kodi probably runs fine on tablets too, but I don't want the kids to stream movies on wifi, they would never get out of their bedrooms.

Of course Kodi is not perfect. It's not native to handle bilingual and it lacks some advanced search features (but the advanced search in my head is lacking in ANY program anyway). It might be possible to add the features in my mind to Kodi by its API (plug-in). But it's an effort to do that, so it's a question of effort/reward calculation.

With your project (BTW, maybe you should name it) you have to build the beautiful interface from scratch. It's doable for the first couple screens... but when you add more features and functionalists, will it get tiring to build each GUI/interface/screens?

So it's a question of learning the inner works Kodi plug-in to do the scrapping and/or porting DB to it, or like in your project, cooking your own GUI (by html) as well as scrapping and DB management.

One question is, how to handle multiple actresses in CSV? For many years I use Excel/CSV to track my collection but it's annoying for compilation and orgy JAVs. Say AVOP-119, sure it's rare, but 18 actresses, many are famous, so I prefer not to miss any one of them in a search. But CSV with fixed fields is not efficient for most AV which has only one actress. Actually how do you deal with the tag/genres in CSV?

So in my own (now in coma) project, I adopted the Kodi/imdb XML format which also has the benefit of letting me sort of handle bilingual:

<title>MXGS-837 Unicycle Policewoman. Patrol! Dispatched!</title>
<originialtitle>一輪車、婦警さん。 パトロールアッキー！出動します！</originialtitle>
<actor>
    <name>吉沢明歩</name>
    <role>Akiho Yoshizawa</role>
    <order>0</order>
</actor>

The downside is that now I can't use Excel to do simple DB management.

Not sure where my stream of consciousness is taking me... just some ideas for your consideration.

As for brainstroming your project... sure... Two screens that I might suggest look like this:

Actress lineup (mug shots)
Actress opus (DVD covers)

Of course, unlike the external website showing all vids for an actress, I want to see only those in my local collection.

About advanced searches, it would be nice to find, e.g.
all vids before July 2006 with genre (tag) "anal" and "huge cock"
all Shinoda vids without genre "shota"
all Anri Okita vids with at least one more actresses

Would be nice to save a search, not the search result, but the search rule.

There's also the idea of playlist, esp. nice to have the option to save an advanced search result (sorted with a user-selected rule) into a playlist.

And then it's tempting to allow editing a playlist, but now you get into the grunt work of cooking your own home theatre program when already everyone loves to hate Windows Media Player 10.

Beyond organizing, browsing and launching vids... once you have a scrapper for javbus and javlibrary, it's very interesting to do advanced search on them (not limited to my collection)... and once you find a cover you like, send the DVD code to kat.cr or somewhere (or simply take the torrent from javbus and/or direct download link from javlibrary) to get a torrent, and then...

Once you know how to code, it's very addictive.

 

[Casshern2]

This would only work if you're using actual queries to return records, but...You can surely have all 18 stars available to search for. You won’t want 18 fields (Actress1, Actress2, Actress3…) but you can use one field for the actress. This even works for titles with only two actresses. Use the Pipe! |

I’ve done this plenty of times. Using a Pipe character as an intra-field delimiter. So if I had a title with Rio Hamasaki and Miki Sato my ONE actress field would be: “Rio Hamasaki|Miki Sato”. Then when I want to find all Miki titles the queries would be: SELECT * FROM tblTitles WHERE tblTitles.Actress LIKE ‘%Miki Sato%’. That would return all titles that are just Miki titles plus any others that she happens to be in with other actresses, like the one with Rio Hamasaki.

If you already have a working database that might be a task to edit your fields, depending if you have multiple actress fields. This should work for your CSV backend.

 

[LordPhantom]

@ding73ding: Tomorrow I send you the links. Yes, in the zip there is XAMPP and the file of project, for that is 106 Mb.
So I am sure that all works. After I can send only the files for the new version in few kb.

@ding73ding & Casshern2: Tomorrow I reply to you, BUT in my thread.
I had many doubt to prosecute in this thread, and also if CodeGeek very nicely (thank you!) as tell me can I do, i think is not correct too many post on my project in his thread.

Now i go to sleep, see you tomorrow!

 

[ding73ding]

Well for years @CodeGeek has been most gracious letting this tiny tiny community of JAV DB cooks to share, brag and brainstorm (with no small amount of fantasizing too) in this thread. I think there's a huge huge advantage in discussing all the projects in this one unified thread. Which is why I chose to announce my project in his thread and kept all my discussion here. And also why this thread is worthy of stickyness. The mods here don't sticky or unsticky threads often, probably average fewer than once a year, so let's treasure this.

In fact, if I remember correctly, this thread only got sticky after I and @cyberzen (and perhaps other ppl) each bring his/her project to discuss here. So a few folks in addition to CG deserve credit for this (moderately) successful thread.

Anyway, with due respect to CG and any topic opener, topic opener doesn't equal topic owner. S/he has a say, perhaps more than a normal share of say, but s/he can't dictate terms in the thread. Just as in a bar or party, someone who open a (good) topic for conversation is appreciated and respected but is helpless if the community (rowdy mob) takes it out of his control.

Hello ding73ding,
my gosh, that's a long post.

Hello cyberzen,
thanks for taking your time and posting here.

 

[Casshern2]

After reading all the recent activity here I dusted off my efforts to get IIS working correctly on my PC so I can use Classic ASP (yes...haven't quite wanted to use ASP.NET yet). I was hoping I could use the aforementioned XAMPP Server, but I quickly saw it didn't support ASP. Thanks to CodeGeek and cyberzen and ding73ding and LordPhantom and everyone else in this thread for having the will, talent, and need to create something. All inspirations.

 

[ding73ding]

Use the Pipe! |

Quite quite! Now I remember using both pipes and commas (and earlier than that, tabs) for delimiters. I was educated in the Unix era and flat text data was our religion. Seems old school geeks are not quite a dying breed as Jedis.

But in past decade I'm sliding towards the dark side, picking up laser pistol (Word) and putting away my light saber (LaTeX). More and more I am using powerful apps instead of cooking my own code. So the wider world is moving towards more elaborate data formats XML, SQL etc. And considering I can't devote time to coding I am more accepting whatever standards and solutions adopted by the bigger community, even if none of them perfectly answer my needs. IMHO, imdb's XML format is the industry giant that we shouldn't ignore.

 

[CodeGeek]

Well for years @CodeGeek has been most gracious letting this tiny tiny community of JAV DB cooks to share, brag and brainstorm (with no small amount of fantasizing too) in this thread. I think there's a huge huge advantage in discussing all the projects in this one unified thread. Which is why I chose to announce my project in his thread and kept all my discussion here. And also why this thread is worthy of stickyness. The mods here don't sticky or unsticky threads often, probably average fewer than once a year, so let's treasure this.

In fact, if I remember correctly, this thread only got sticky after I and @cyberzen (and perhaps other ppl) each bring his/her project to discuss here. So a few folks in addition to CG deserve credit for this (moderately) successful thread.

Anyway, with due respect to CG and any topic opener, topic opener doesn't equal topic owner. S/he has a say, perhaps more than a normal share of say, but s/he can't dictate terms in the thread. Just as in a bar or party, someone who open a (good) topic for conversation is appreciated and respected but is helpless if the community (rowdy mob) takes it out of his control.

I have to admit - and I also wrote that to @LordPhantom - that I'm divided between both solutions: Keeping the discussion of single projects here and giving them there own thread. Each of the solutions has its own pros and cons and I can understand each position. So I let him/her make that decision. I also wrote him/her that I don't think that I "own" any thread here. It's interesting that you, @ding73ding, brought that up. Still it was very nice of him/her to ask me before doing so. I guess so far no one ever asked me here if he/she can post in one of the threads I opened.

I suggest the following: We keep the discussion here - to keep the thread and ideas alive and also to promote new projects. And if a project gets bigger that project should get its own thread (for discussion about that project, bug fixes, new features and so on) so it doesn't dominate the discussion here. What do you all think?

 

[LordPhantom]

I think that the best solution may be to put in this thread the note on the project related to new versions, new feature etc, and the big brainstorming or discussions that are of interest only for peoples that follow the specific project in the thread dedicated.
In this way the presence in this thread is not heavy and there is tracking of all projects.
In any way very very thanks to CodeGeek! A real (and rare) gentleman.

Here the last news on the project: https://www.akiba-online.com/threads/yet-another-db-jav-scraper.1614249/#post-3198524

 

[ding73ding]

At the moment there are some trick that require a big clean in the script, that I will do in next versions BUT that VERSION is fully working! Also the translation is not complete.
...
working or you want download it from the original site: http://portableapps.com/apps/development/xampp

Wonderful! I haven't tested it yet, but I took a look at the code, looks smart.Every program is always in need of a big clean up anyway, so releasing a fully functional version is a good achievement.

Translation... I am guessing it's not an urgent issue. Easy enough to guess what each link does. At some point, assuming your project is getting a bigger adoption (n > 5?) you might think about a multi-language pack (Euro-5 + Asian-3?). I can help, but only when the demand is clear.

When I search XAMPP, I found the current version is 5.x or 7.0.1 what version are you bundling? The version on your "original site" seems to be hosting 1.8.3 from 2013? Would JavDB works with a current version?

For me to adopt your JavDB, it has to run on Linux on my QNAP TS-209 NAS. Actually the NAS already has a web server and MySQL, PHP and Perl should be easy or even trivial, so it's tempting to just try JavDB on NAS, may need some tweaking of code (paths etc). That would save me from installing XAMPP which seems redundant on the NAS.

 

[LordPhantom]

Yes, JavDB virtually can run in any web server with perl installed, this is the only request, also every version of XAMPP can work.
Maybe required a little work for the paths of the scripts for point to the perl executable and the paths for videos and covers.
For the version that I linked is the portable version, instead you can use a standard version, now 5.6.5 with PHP 7.01 here: https://www.apachefriends.org/index.html
For the translation are very few frases to translate, a little work.
For the code the must important script is Scraper.cgi, write from scratch, the others are free script for managing the DB.

 

[Casshern2]

I know this is trivial compared to the things being developed and discussed here, but I thought I'd post an update on what I've had in mind for my use and finally got something going.

Volume Database

Unless I’m on a project for work, “robust” doesn’t compute. I finally got around to making my little asset control piece. And when I say it is bare bones…the bones are barely there. No web scrapers, no scripts, just something to help keep track of what I have. Wouldn’t expect much else after a few hours effort. To be sure, I was only thinking of an asset control platform, just something to allow simple searches for DVD Codes to check if I had something across any of my HDDs. Super simple Classic ASP with a CSV backend. Believe me, my whole process and setup for all of this is a poor man’s everything. I had a simple need…I created a simple solution for a simple man (that’s me!) with no regard for true scalability.

The CSV Files
I created an Excel Application in VBA that sits on the HDDs along with the files. For an initial catalog or when titles are added/deleted I run these on the appropriate HDD. They call MediaInfo to give me the container, FPS, length, etc.. I also grab the Volume name of the HDD and obviously the name of the file. This is done on the six of my current HDDs and the resulting files are saved as CSVs in a central location. I have a final Excel App that combines all of the separate CSV files into a master CSV file that the site runs off of.

The Site
The site is just a listing running on localhost, paged 30 titles at a time. A simple search page for searching specific codes or codes with the same prefix (JUFD, ASW, MOT, etc…) with the results displayed in the same manner as the primary listing for multiple records. Basically, if the search box contains a hyphen it queries what was entered (Title = ‘JUFD-215’). If the search box does not contain a hyphen it searches for anything with that prefix (Title LIKE ‘JUFD%’).

The Images
Because of my legendary foot traffic problem where my PC is (okay, it isn’t legendary…but it was once associated with Tom Cruise…and he’s pretty legendary) there are two flavors for this thing. Vanilla…and Plain Vanilla. I have some cover images as I find time to grab them. Just the thumbs, though, not the full images. I may do that perhaps, but it isn’t as important for asset control. This is the Vanilla version. The Plain Vanilla version has a generic image for every title. A simple click on the site logo toggles this for me. Thankfully because of it running locally the transition is lightning fast. You know…in case those feet come marching in.

Like I told a really good friend while discussing this, and I won’t drop names (@jugulear)…I wanted to be able to say I have a “thing” I use to keep track of my JAV and not just that I have an Excel or CSV file. Granted…that’s what this thing runs on, but it isn’t just that. I’m happy so far. BTW if anyone can come up with a spiffier name, I'm all ears. But it can't have "JAV" in the title...in case the foot traffic decides to Google what that is...

Tools:
IIS on Windows 7
HomeSite (OLD html editor)
Macromedia Fireworks (OLDER image program)
Microsoft Excel
MediaInfo

Vanilla


Plain Vanilla


Specific Search




Broad Search