Tool Others Vambropy [v1.15.01]: VaM packages & user content database, management, and a complement to VarBrowser

[commondi32]

EDIT: New version 1.15.01

IMPORTANT: Database changed, so a new file vambropy.db will be created followed by a full package scan. Remember that scanning external, non-ssd disks can be slow, some minutes to take a coffee maybe. The previous database file, packages.db, will not be used so you can safely delete it.

After many hours of rewriting parts of the code, debugging and including new ideas by qube21, a big update comes. It brings many changes to the UI and database, as well as minor details here and there.

• User content view allows to browse your scenes and custom preset files, all categories filterable, and their related dependencies with the same options as in the packages view.
• Many improvements to the database queries for better performance in tasks such as sorting columns, scanning, checking dependencies availability in your packages collection, etc.
• Little details reworked for better user experience, like copying missing dependencies directly to clipbard, main window finally opening fine and always centered.
• qube21's new UI functionalities, some of them slightly adapted, included: Older packages filtering options , mouse drag selection with file size sum info on title bar, optional image detaching from the main window and upscaling, new context menu entries, like filtering by creator. Also some performance improvements in thumbnails generation, etc.
• Number of dependencies and Usage number columns. Menu option for scannig related dependency usage (number of packages that require them), only on demand because it is currently slow for all the packages, but then it is stored and persistent in the database.

Main thread app instructions are outdated now. Spoiler sections will be reduced because the workflow is now smoother, mostly self-explanatory. Keep in mind that still it is VaM related stuff. Sort of advanced, inherently complex, quirky, that demands a learning curve to know your stuff, manage it, to and get the most performance from the set of apps and plugins you are using. Sfishere's Varbrowser obvioulsy is the best companion for this app.
As always, comments on bugs, enhancements, anything... are welcome!


Overview
Vambropy is an open source tool for managing your .var collection and enhancing the use of sfishere's VarBrowser plugin. It keeps up to date your currently used packages in VaM as well as packages in external directories/drives. Metadata and tags generated from packages' content are imported to a database, giving you flexible ways of offline browsing your .var files, by sorting, grouping, filtering.
Integration with Varbrowser is provided by allowing to install/uninstall (moving packages from/to its 'AllPackages' directory), with or without dependencies, and is extended with the options to install from external sources (copy/symlink) as well as dealing with Varbrowser's AutoInstall and Favorites tag system in a more fast and flexible way, helping to enhance VaM's performance and responsiveness by dynamically adding and removing custom lists of packages (profiles, a feature VarBrowser lacks). The newest feature is the User content browsing with the same granular dependency install options as with packages.




Features (for v1.12, will be updated later for 1.15):

• A fast sqlite database, even faster in v1.07, updated with a single click, and before each package process
• It uses VarBrowser's concept of available/installed packages. (move paths between AllPackages and AddonPackages folders)
• Integrates VarBrowser's Autoinstall system (toggle on/off "packages always installed no matter what")
• Helps with installing bunchs of dependencies VarBrowser sometimes left uninstalled.(Subdependencies). Couple of clicks and all installed.
• Easy package installation from external directories/drives (copy with full path). No removal of packages, so no package can be accidentally lost.
• Unpacking option for clothing and hair. Local clothing and hair -specially clothing, since we don't use too many hairs to clog the VaM's pipes- works faster than vars, it makes person atom load faster, increasing performance)*.
• VaM and repos directories check, making sure no duplicate packages are within VaM (you manually choose), and bad packages information is displayed when you have packages with bad format.
• Multiple multi-word filters allow you to group, view specific content by creator, info, morphs, etc, and analyze dynamically changing dependencies related to your currently filtered packages, which packages are using some dependency, etc.
• Color-coded information on package and dependency availability.
• View all package's thumbnails in a thumbnail; tags, information and instructions (useful for plugins). i.e. Cannot remember that camera plugin name...? Just type 'camera' in var_info filter and all related packages will show.
• Custom lists (profiles) allows to add/remove your created lists of packages. You can easily uninstall all (like you do from VarBrowser) but at the same time enable a list with hundreds of packages, and then another one and so on.
• A button to view all the dependencies from your saved scenes, so you to get a list of dependencies that can be installed back,i.e. to easily recover your saved scenes' functionality if you previously uninstalled all vars.
• Context menus with all the functions and hotkeys, in the main packages and dependencies views.
• Can generate lists of missing dependencies, to export or copy/paste to the great Boss963's Sharp VaM Tools
• VarBrowser's Favorites are shown in the list with different background color (configurable).
• Can install Sym-Links from external NTFS partition/disks (only when it runs with administrator privileges)
• Since version 1.09 it can install with dependencies in one step like Varbrowser, hopefully better results with sub-dependencies.
• Theme selector. Use config.ini to set any as default,

IMPORTANT: Vambropy does NOT modify any files other than Custom/PluginData/sfishere/Autoinstall.txt (and Favorites,txt). It does NOT modify your sub-directories structures. When you command it to install/uninstall, it moves files with their paths between AllPackages and AddonPackages dirs, so all paths are kept intact; same with installing by copying from another directory. You can configure your external drives/ directory trees with your own structures and the way it copies into VaM, to do it your own way. Vambropy is independent of VaM.exe, it does not need VaM to be running, and changes made with it are not automatically reflected in VaM. if you are running VaM and Vambropy together, after making changes on one of them you will need to refresh the other.
ALSO IMPORTANT: Vambropy always checks for duplicates between AllPackages/AddonPackages, giving the user the option in a dialog box to keep just one. The rest is deleted. In case of duplicate names that are not digitally equal it is warned in a dialog box to proceed with caution, to check the files' content, before dediding which one to keep. As some people tend to click on buttons without reading too much, really realy fast, this could be a bit risky. It could change in future version if you consider it risky.


About the code:

Vambropy is python-coded, open source; code is fully commented on what it does and is easily moddable if you know some programming basics. If there are python programmers in here and want to mod it/fix things/enhance functionality go ahead! If you want to make it bigger or part of another project just mention me as original creator/collaborator on this stuff.
The program will not manage downloads. I just have tested the recent Boss963's Sharp VaM Tools, is already featuring that and much more, and it does it awesomely. Not much more is planned at the moment. I'm after some way of managing morphs so they can be dynamically moved in and out of VaM catalog, to help with performance, That is the reason all morphs are hashed and listed in the db (hidden at the moment but filterable) Feedback and suggestions welcome


Some details on its workflow:

Spoiler: Packages main section

You don't have permission to view the spoiler content. Log in or register now.

Spoiler: Dependencies section

You don't have permission to view the spoiler content. Log in or register now.

Spoiler: Lists section

You don't have permission to view the spoiler content. Log in or register now.

Spoiler: * Unpacked packages

You don't have permission to view the spoiler content. Log in or register now.

Spoiler: Sym-Links

You don't have permission to view the spoiler content. Log in or register now.

Finally there are 3 simple buttons for loading VaM in its 3 main modes.
That is all... I hope you enjoy using it.


IMPORTANT 2: You will see two more .exe files in the release archive: grep and sed from GNU tools for windows are used for very fast scannig on user content, like 10x or more than using python. The provided versions are not the most recent but work fine. You can download updated versions and replace them.

Download
V1.15.01 sources (see attachment)
V1.15.01 binaries:

You must be registered to see the links

You must be registered to see the links

You must be registered to see the links

​

 

[Trebutor]

Nice, but somethings are missing:

- Dependencies, I assume Tk and Image.
- How to run, I assume "python vambropy.pyw"
- What to put in config.ini?
- It doesn't seem to do anything right now. Is there a magic hidden button to press?

 

[commondi32]

Nice, but somethings are missing:
...

Thanks for the feedback. Yes, maybe you need pillow (pip install pillow), for images. Tkinter comes bundled with python 3.11 I think.
I have been testing and fixing bugs in the Dependencies section (like buttons not working yet) among other things. I have been testing as many as I can, and now uploading new version with many changes and fixes, some optimizations, and update on all help about new config.ini (reworked with more settings and comments). I think it is now easier and more solid and interactive, with more messages, warnings to avoid little accidents, and customization options. It should run right from the unzipped files. To run by src, go to src dir and 'python vambrpoy.pyw' should work. Or just open vambropy.exe (no python needed). Start the program, click the "Edit config.ini" button and set the correct VAM_dir. Then restart.

 

[commondi32]

I have not much experience uploading and attaching files, and today I managed to upload the new version with executable + sources in the same file because it is smaller now and it fitted. But the legend [VIRUS] was inserted in the attachment name. What is that? Do I need to upload it again? I am sure the file is not infected, I cannot tell why that happened. If there is problem to download that way please tell me and I'll upload separated binary from source if needed.

 

[Franknar]

Just a suggestion, but why not just use symbolic links instead of moving the files?

 

[commondi32]

Just a suggestion, but why not just use symbolic links instead of moving the files?

It could be an option. I was not thinking about it because I don't use it personally, and the app was first conceived to complement sfishere's VarBrowser. And there is already feelfar's VarManager doing symlinks, and recently BrowserAssist. Also, I thought maybe people do not want to try a unknown tool that is asking for admin privileges, since symlinks cannot work at user level. I'll be releasing an update 1.08 soon to fix some bugs introduced in later versions, and then I will try to do some tests. If I manage to make it work and optionally ask for admin when it enables it, I'll add it.

 

[pirate pete]

Getting this error once all packages are loaded by the program, any idea how to fix?:

 

[commondi32]

Getting this error once all packages are loaded by the program, any idea how to fix?: View attachment 4581308

Thanks for the feedback. I think this was one of the bugs fixed in the new version 1.08
EDIT: If it keeps showing on the new version, the problem seem to be with a dependency in a .var file that requires a minimum version, so it is expecting a number but it gets a 'latest', I imagine an incorrect string like "minlatest", as if it was a manual correction to a meta.json, since some people mess with those files manually and corrupt the file. Could be a little hard to pinpoint. If you have rare vars not from the hub or known sources, you can try inspect those vars with VaM Sharp Tools, it might find and fix some bad stuff.
Another way (you may not like much) is to rename your current packages.db and start again with a limited list of vars. If you previously include repos, do it only with vam directory this time, see if it starts without problems, and then add bunchs of vars to the database until the exception shows again
EDIT AGAIN: I will try to add some code to manage that kind of execption in the next version, so it can skip the (bad) package in case of a malformed string , not throwing you out of the app.

 

[commondi32]

Just a suggestion, but why not just use symbolic links instead of moving the files?

Well it turned to be somewhat easy to implement it as an alternative copy method. The menu now it has the option to symlink your package(s) selection, or by pressing <L> key. It works on packages windows, at the moment. Check info on the release (symlink spoilers section) for details. Remember to run Vambropy as admin. If not, options are available but will not do anything more than create relative subdirectories for the links.
Symlinks will not mix with packages, they will copy in AddonPackages/symlinked. And if uninstalled, they will go to AllPackges/symlinked, preserving relative paths.

 

[Franknar]

Well it turned to be somewhat easy to implement it as an alternative copy method. The menu now it has the option to symlink your package(s) selection, or by pressing <L> key. It works on packages windows, at the moment. Check info on the release (symlink spoilers section) for details. Remember to run Vambropy as admin. If not, options are available but will not do anything more than create relative subdirectories for the links.
Symlinks will not mix with packages, they will copy in AddonPackages/symlinked. And if uninstalled, they will go to AllPackges/symlinked, preserving relative paths.

Amazing! Thanks

 

[yuezy3]

great work, but this config.ini may cz program fail because in line 1006 of modules/ui.py :

with open(config_file_path, 'r') as file:

read config.ini but not sepcify encoding. change this line to :

with open(config_file_path, 'r', encoding='utf-8') as file:
will better. it will now can handle other non assic char as path.

 

[commondi32]

with open(config_file_path, 'r', encoding='utf-8') as file:

Thank you! I'll fix it by changing that line as you specified, in next version.

 

[cdt_drecks]

Hi, I get this error:

Traceback (most recent call last):
File "vambropy.pyw", line 79, in <module>
File "vambropy.pyw", line 68, in main
File "modules\ui.py", line 1184, in refresh
TypeError: 'NoneType' object is not subscriptable

 

[commondi32]

Hi, I get this error:

Traceback (most recent call last):
File "vambropy.pyw", line 79, in <module>
File "vambropy.pyw", line 68, in main
File "modules\ui.py", line 1184, in refresh
TypeError: 'NoneType' object is not subscriptable

Hi, you have found a bug in favorites.txt file handling. The program should create an empty one if it does not exists, but somehow an exception occurs. I will fix it in next release. In the meantime, you can fix it two ways:

Simple way: In VaM, use Varbrowser and set any package you like as favorite (it turns blue). That will create the needed file; or

Edit way: Create an empty "favorites.txt" in Custom/PluginData/sfishere directory (the path is inside VaM directory). Open the empty file with a text editor and type: {"FavoriteNames":[]}. Save the file.

Then Vambropy will start ok.

 

[cdt_drecks]

Hi, you have found a bug in favorites.txt file handling. The program should create an empty one if it does not exists, but somehow an exception occurs. I will fix it in next release. In the meantime, you can fix it two ways:

Simple way: In VaM, use Varbrowser and set any package you like as favorite (it turns blue). That will create the needed file; or

Edit way: Create an empty "favorites.txt" in Custom/PluginData/sfishere directory (the path is inside VaM directory). Open the empty file with a text editor and type: {"FavoriteNames":[]}. Save the file.

Then Vambropy will start ok.

Thansk! Now I have this error:


Traceback (most recent call last):
File "vambropy.pyw", line 85, in <module>
File "vambropy.pyw", line 74, in main
File "modules\ui.py", line 1224, in refresh
File "modules\packages.py", line 66, in find_var_names
File "modules\packages.py", line 47, in find_in_all_packages
UnboundLocalError: cannot access local variable 'latest_key' where it is not associated with a value

 

[commondi32]

Thansk! Now I have this error:


Traceback (most recent call last):
File "vambropy.pyw", line 85, in <module>
File "vambropy.pyw", line 74, in main
File "modules\ui.py", line 1224, in refresh
File "modules\packages.py", line 66, in find_var_names
File "modules\packages.py", line 47, in find_in_all_packages
UnboundLocalError: cannot access local variable 'latest_key' where it is not associated with a value

Another one, thanks! Fixed in new version 1.10!

 

[cdt_drecks]

Another one, thanks! Fixed in new version 1.10!

Hey, this error is still present with 1.10 :

Traceback (most recent call last):
File "vambropy.pyw", line 89, in <module>
File "vambropy.pyw", line 78, in main
File "modules\ui.py", line 1224, in refresh
File "modules\packages.py", line 70, in find_var_names
File "modules\packages.py", line 51, in find_in_all_packages
UnboundLocalError: cannot access local variable 'latest_key' where it is not associated with a value

 

[commondi32]

1.10.1 was the one that fixes it!

 

[doggava]

This tool looks cool!

By the way, check this tool out too for development ideas:

You must be registered to see the links

It e.g. removes old versions or vars and has a mapping functionality for missing vars (using symlinks).

 

[luna.kitty]

" In future relases a button could be included to do either: a) in Dependencies section, install all (or filtered) external dependencies as symlinks, or b) Install everyhing external symlinked, as 'Available' , so Varbrowser could handle everything from VaM. "

I really need this useful function, because my tons of var files are separated across several storage locations. I want to install dependencies as symlinks in just two steps - click a specific var, then install dependencies from a Dependencies list as symlinks.
I think this tool is the best VAM tool for me, thank you very much!