Tool Daz Daz Content Manager [0.5] [rythamun]

[xxx3me]

The trick worked for me. DB open using Notepad++, all instances from 255 to 800 and it's scraping again. The jDownloader function is excellent, by the way.

There is a few things a possible (improbable?) new developer would need to see:

- Renderosity blocking scrapers
- Not important, but ZoneGfx seems to be useless, their download links are paid, I guess.

 

[PJWhoopie]

Maybe that is my problem (Renderosity Blocking Scrapers)... None of the images are showing up (from Renderosity) in DCM... And I get a shit ton of errors in the Work Que.

I wish there was a way to purge all in the work queue, cause I keep hoping that something will happen miraculously and it works again... until then I Scrape New Assets and then hand delete the errors.

I have been using DCM for some time now, and Never got ZoneGfx to work. The bread and butter for me were F95 and 3D-Load, but now that 3D has complicated the process there, F95 is the only thing that I can get to scrape. (And Daz + Renderotica)

Are you having any luck scraping 3D-Load?

I just saw your post about octoparse ... which sites do you scrape with that... care to elaborate a bit more on how you combine octoparse /DCM?

 

[realteamkiller]

Maybe that is my problem (Renderosity Blocking Scrapers)... None of the images are showing up (from Renderosity) in DCM... And I get a shit ton of errors in the Work Que.

I wish there was a way to purge all in the work queue, cause I keep hoping that something will happen miraculously and it works again... until then I Scrape New Assets and then hand delete the errors.

I have been using DCM for some time now, and Never got ZoneGfx to work. The bread and butter for me were F95 and 3D-Load, but now that 3D has complicated the process there, F95 is the only thing that I can get to scrape. (And Daz + Renderotica)

Are you having any luck scraping 3D-Load?

I just saw your post about octoparse ... which sites do you scrape with that... care to elaborate a bit more on how you combine octoparse /DCM?

I'm scraping Daz and Renderosity with octoparse. For the rest I still have to configure the scraper.
With this app you can also save the result als json.
I save the result as xlsx and import it into access (own db).
With octoparse you should be able to handle all websites.

You must be registered to see the links

 

[xxx3me]

Seems like Renderosity is not working anymore, it's returing an "Error in packaging process". Tried two packages and same result, example:

"Could not download

You must be registered to see the links

"

Following the link, it's a xml like this:

<Error>
<Code>MissingKey</Code>
<Message>Missing Key-Pair-Id query parameter or cookie value</Message>
</Error>

 

[PJWhoopie]

I think DCM is officially dead (at least for me) as daz's new site update seem to be incompatable with the scraping (Like Renderisity). Anyone have a work around or do we just go quietly into the night....?

 

[scarlettyoung]

If someone took up the project and either built off the existing code or wrote their own, I would totally pledge at a level of at least $10 a month.

 

[cooperdk]

If someone took up the project and either built off the existing code or wrote their own, I would totally pledge at a level of at least $10 a month.

I develop a porn organizer app on Python (YAPO e+) and can tell you that you'd probably need 50 x $10 donations to even get close to the amount of time required for development. I'm not talking commercially but it would really take time, especially as a one man project ... and I'm surprised that ryth did all this work and then scraps it :-/

 

[cooperdk]

I'm scraping Daz and Renderosity with octoparse. For the rest I still have to configure the scraper.
With this app you can also save the result als json.
I save the result as xlsx and import it into access (own db).
With octoparse you should be able to handle all websites.

You must be registered to see the links

Yeah, but that way, you won't combine source sites with download sites. You have to write a program or at least an SQL procedure to do the matching.

If anyone wants to collaborate on a similar project but in Python I would be able to do some work. But I have no experience with Java.

 

[cooperdk]

New renderosity items don't work. The file names of the images are longer than 255 characters.
With which free database app you can open the db file. Have already tried a few but they don't recognize the db file.


Problem solved.:

I have open the db file with notepad++ en change :

"MEDIA_SRC" VARCHAR(255) to "MEDIA_SRC" VARCHAR(800) (32 times)

I looked over the code and in the cconfig file application.yml it was revealed what DB is used.

datasource:
type: com.zaxxer.hikari.HikariDataSource
url: jdbc:h2:file:./dcm-database;DB_CLOSE_DELAY=-1;MVCC=true;query_timeout=10000
username: test
password:
hikari:
poolName: Hikari
auto-commit: false
maximum-pool-size: 500


For some reason, rythamun chose to use H2 as the database for this project (strange choice).

You will need a program like RazorSQL to edit the tables/data. RazorSQL is also based on Java and has a connector for H2.

Add a connection and you should be able to work in the database

 

[cooperdk]

PS:

RazorSQL is here in a cracked version:

You must be registered to see the links

I had a lot of empty content after trying to rescrape Renderosity. It does seems like it doesn't work at all anymore, even with the database changes.

To remove all asset IDs higher than a certain number (in my case it was 296595), execute this in RazorSQL - choose "execute SQL (all statements)" in the "SQL" menu:

DELETE FROM ASSET_ARTIST where ASSET_ID>296595;
DELETE FROM ASSET_CATEGORY where ASSET_ID>296595;
DELETE FROM ASSET_FIGURE where ASSET_ID>296595;
DELETE FROM ASSET_INCLUDED_ASSETS where ASSET_ID>296595;
DELETE FROM ASSET_LIST_ASSET where ASSET_ID>296595;
DELETE FROM ASSET_MEDIA_LIST where ASSET_ID>296595;
DELETE FROM ASSET_REQUIRED_ASSETS where ASSET_ID>296595;
DELETE FROM ASSET_SOFTWARE where ASSET_ID>296595;
DELETE FROM DOWNLOAD_PAGE_ASSET_DOWNLOAD_LINKS where DOWNLOAD_PAGE_ASSET_ID>296595;
DELETE FROM DOWNLOAD_PAGE_ASSET where ASSET_ID>296595;
DELETE FROM ASSET where ID>296595;

To remove the entire queue, execute:


To remove all queue items, run:

DELETE FROM QUEUE_ITEM;

This will reset your queue.

 

[PJWhoopie]

Well... I am having a hard enough time just learning DAZ/Blender... let alone a program language to fix this... but I can paypal/patreon anyone who fixes this so I can scrape/download Daz Store & Renderosity again (along with 3D-Load and add render-state.to )

I miss using this... sigh....

What is anyone using as a replacement?

 

[scarlettyoung]

Well... I am having a hard enough time just learning DAZ/Blender... let alone a program language to fix this... but I can paypal/patreon anyone who fixes this so I can scrape/download Daz Store & Renderosity again (along with 3D-Load and add render-state.to )

I miss using this... sigh....

What is anyone using as a replacement?

Good question. When I buy assets on the official Daz store it's added to my downloads and I can just get it directly from DAZ Install Manager, but anything I get from Renderotic comes in a non-IM format and this tool was the only way I have been able to import those into DAZ and keep track of if I have them installed or not. As I mentioned before, someone could start a Patreon and I would be the first to join.

 

[knightgames28]

never mind what t i do i get some missing files anyone have link to download as package ?

 

[PJWhoopie]

Anyone figure anything else out with this awesome program?

I think the only scrapeable sites now are Renderotica and F95. The others changed their format with site upgrades/updates and has broken the way DCM works...
Sigh...

 

[cooperdk]

I'm scraping Daz and Renderosity with octoparse. For the rest I still have to configure the scraper.
With this app you can also save the result als json.
I save the result as xlsx and import it into access (own db).
With octoparse you should be able to handle all websites.

You must be registered to see the links

In fact, Content Grabber 2 is better, and it offers much more plus it runs locally.

Anyone figure anything else out with this awesome program?

I think the only scrapeable sites now are Renderotica and F95. The others changed their format with site upgrades/updates and has broken the way DCM works...
Sigh...

Uhm, if there was interest, I could set up a Patreon and write a Python app to scrape all these sites. Then create a frontend to search and view content (and possibly available sources).
I don't know Java so I couldn't develop this further, unless I can find the xpaths within the code.

The thing is, I don't think the sites blocked this script or blocks scraping. Very few sites do that nowadays.
I do think the reason this app doesn't work is due to (possibly slight) code changes on the various websites. This will make a program like this unable to actually find the right data on the pages.

Thinking about it, the way this is programmed is less than ideal. The dev should have included configuration files for the scrapers, making it possible to update the xpaths and expand to other sites. Then it would be a piece of cake to make it work again.

 

[scarlettyoung]

In fact, Content Grabber 2 is better, and it offers much more plus it runs locally.




Uhm, if there was interest, I could set up a Patreon and write a Python app to scrape all these sites. Then create a frontend to search and view content (and possibly available sources).
I don't know Java so I couldn't develop this further, unless I can find the xpaths within the code.

The thing is, I don't think the sites blocked this script or blocks scraping. Very few sites do that nowadays.
I do think the reason this app doesn't work is due to (possibly slight) code changes on the various websites. This will make a program like this unable to actually find the right data on the pages.

Thinking about it, the way this is programmed is less than ideal. The dev should have included configuration files for the scrapers, making it possible to update the xpaths and expand to other sites. Then it would be a piece of cake to make it work again.

If you take over this project I'll gladly support you on Patreon.

 

[Slagmoth]

In fact, Content Grabber 2 is better, and it offers much more plus it runs locally.




Uhm, if there was interest, I could set up a Patreon and write a Python app to scrape all these sites. Then create a frontend to search and view content (and possibly available sources).
I don't know Java so I couldn't develop this further, unless I can find the xpaths within the code.

The thing is, I don't think the sites blocked this script or blocks scraping. Very few sites do that nowadays.
I do think the reason this app doesn't work is due to (possibly slight) code changes on the various websites. This will make a program like this unable to actually find the right data on the pages.

Thinking about it, the way this is programmed is less than ideal. The dev should have included configuration files for the scrapers, making it possible to update the xpaths and expand to other sites. Then it would be a piece of cake to make it work again.

I only recently came upon this thread in the hopes that I finally found out how to get the assets from this site into DAZ3D so I can play around with it more.

And don't be too hard on the Dev, even professional devs at shops I have worked at still use SELECT * in production and use way too many static "helper" classes and don't put all they should have in some sort of config (either file or db).

So if anyone can point me in a direction on a tutorial to get assets from this site to my DAZ IDE, I would appreciate it.

 

[PJWhoopie]

This "was" a beautiful solution, at least on the user facing end.
You scraped the major sites, clicked what you wanted... it unpacked everything and generated DIM-able packages for all the non Daz site stuff.
Sigh....

 

[scarlettyoung]

This "was" a beautiful solution, at least on the user facing end.
You scraped the major sites, clicked what you wanted... it unpacked everything and generated DIM-able packages for all the non Daz site stuff.
Sigh....

Yes, it was beyond awesome and I'm still willing to financially support its continued development.

 

[asgaurd]

Anyone has the source code ? I could try to fix it