Tool Others Vambropy [v1.16.2]: VaM packages & user content database, management, and a complement to VarBrowser

[commondi32]

EDIT: New version 1.16.2

IMPORTANT: New database column added in v1.16.1. To use the same vambropy.db from v1.16, execute updatedb.py or updatedb.exe once, then delete it.

This update consists of bug fixes , performance, menu and control improvements, and introduces 'ignore' tag for dependencies. It allows to hide unwanted missing dependencies, for example. Also old dependency versions (updated by menu option) are coloured like packages and can be hidden by menu option. See changelog for complete list. Also fixes a bug in 1.16.1 recently released.

EDIT: New version 1.16.0

IMPORTANT: Thumbnails directory structure changed. To update a previous version -before running Vambropy- execute the file sortimgs.exe (or sortimgs.py) once, and all files in the thumbnails directory will move to related creator sub-directories. (you can delete the sortimgs.exe/py file after it ). It helps optimize performance when you have tens of thousands of images, with external apps used when double-clicking the image, and maybe with Vambropy too.

This update has many changes under the hood, preparing for integration with upcoming qube21's Qt user interface. Main user-related changes are bug fixes, improvements on UI, better resizing, some new columns , and better word filtering using negative and exclusive expressions ('-' and '*' prefixes, implemented by qube21 for the upcoming Qt UI)

Features and usage information updated! Spoiler sections rewritten, as workflow is simpler now. Keep in mind that still it is VaM related stuff. Sort of advanced, inherently complex, quirky, that demands a learning curve to manage it and get the most performance from the set of apps and plugins you are using. I recommend reading the great Anand Neemish's guide to Sfishere's Varbrowser , the best companion for this app.
As always, comments on bugs, enhancements, anything... are welcome!


Overview
Vambropy is an open source tool for managing your .var collection and enhancing the use of sfishere's VarBrowser plugin. It keeps up to date your currently used packages in VaM as well as packages in external directories/drives. Metadata and tags generated from packages' content are imported to a database, giving you flexible ways of offline browsing your .var files, by sorting, grouping, filtering.
Integration with Varbrowser is provided by allowing to install/uninstall (moving packages from/to its 'AllPackages' directory), with or without dependencies, and is extended with the options to install from external sources (copy/symlink) as well as dealing with Varbrowser's AutoInstall and Favorites tag system in a more fast and flexible way, helping to enhance VaM's performance and responsiveness by dynamically adding and removing custom lists of packages (profiles, a feature VarBrowser lacks). The newest feature is the User content browsing with the same granular dependency install options as with packages.




Features :

• A high-performance sqlite database, updated on each package process, or with a single button/hotkey.
• VarBrowser's concept of available/installed packages. It moves paths between AllPackages and AddonPackages directories
• Integrates VarBrowser's Autoinstall system (toggle on/off packages "always installed no matter what")
• Easy dependencies view, it helps with installing bunchs of dependencies VarBrowser sometimes left uninstalled, like sub-dependencies.
• Easy package installation (copy) from external package repositories
• No removal of packages, so no package can be accidentally lost.
• Older packages filtering options.
• User content view allows to browse your scenes and custom preset files, all categories filterable, and manage their related dependencies with the same options as in the packages view. (ie. all your custom scenes dependencies can be instantly installed by pressing a single button)
• VaM directories check (making sure no duplicate packages are within VaM) and bad packages information. (search for ERROR on description)
• Multiple group and multi-word filters allowing to view specific content by tags, creator, description and instructions (useful for plugins), morphs, creation date, and analyze dynamically changing dependencies related to filtered packages. i.e. just type 'camera' in description filter and all related packages will show, type 'scripts' in tags to reduce even more, making easy to search that camera-related plugin.
• Mouse drag selection with file size sum info on title bar.
• Configurable color-coded information on package and dependency availability.
• Theme selector. (Use config.ini to set any of them as default)
• All package's thumbnails in a single image.
• Custom lists allow to add/remove your created lists of packages (profiles). You can easily uninstall all (like you do from VarBrowser) but at the same time install/unistall small or big groups of listed packages.
• Context menus with all the functions and their related hotkeys, in the main packages and dependencies views.
• Export missing package lists to clipboard, to easily copy/paste in Sharp VaM Tools.
• Packages with VarBrowser's Favorite items are shown in the list with different background color.
• Sym-Linked package install from Repos (external NTFS partition/disks). This feature only works when running with Administrator privileges.
• Unpacking option for clothing and hair. Local clothing and hair -specially clothing, since we don't use too many hairs to clog the VaM's pipes- is slightly faster than vars, increases performance (with some compromises*).
• Optional image detaching from the main window with image upscaling.
• Number of dependencies and Usage number columns, on demand, but persistent in the database.

IMPORTANT: Vambropy does NOT modify any files other than Custom/PluginData/sfishere/Autoinstall.txt (and Favorites,txt). It does NOT modify your sub-directories structures. When you command it to install/uninstall, it moves files with their paths between AllPackages and AddonPackages dirs, so all paths are kept intact; same with installing by copying from another directory. You can configure your external drives/ directory trees with your own structures and the way it copies into VaM, to do it your own way. Vambropy is independent of VaM.exe, it does not need VaM to be running, and changes made with it are not automatically reflected in VaM. if you are running VaM and Vambropy together, after making changes on one of them you will need to refresh the other.
ALSO IMPORTANT: When Vambropy starts, it checks for duplicates between AllPackages/AddonPackages, giving the user the option in a dialog box to keep just one. The rest is deleted. In case of duplicate names that are not digitally equal it is warned in a dialog box to proceed with caution, to check the files' content, before dediding which one to keep. As some people tend to click on buttons without reading too much, really realy fast, this could be a bit risky. It could change in future version if you consider it risky.


About the code:

Vambropy is python-coded, open source; code is fully commented on what it does and is easily moddable if you know some programming basics. qube21 is also working a lot to enhance functionality and interface, with numerous great features (see Vambropy.pyw changelogs). If there are python coders in here and want to mod it/fix things/enhance functionality just jump aboard! If you want to make it bigger or part of another project just mention me (as original creator) and the collaborators.


Some details on its workflow:

Spoiler: Packages main section

You don't have permission to view the spoiler content. Log in or register now.

Spoiler: Related Dependencies section

You don't have permission to view the spoiler content. Log in or register now.

Spoiler: Custom lists section

You don't have permission to view the spoiler content. Log in or register now.

Spoiler: * Unpacked packages

You don't have permission to view the spoiler content. Log in or register now.

Spoiler: Sym-Links

You don't have permission to view the spoiler content. Log in or register now.

Finally there are 3 simple buttons for loading VaM in its 3 main modes.
That is all... I hope you enjoy using it.


IMPORTANT 2: The binary release has two more .exe files in 'resources' directory: grep and sed from GNU tools for windows are used for very fast scannig on user content, like 10x or more than using python. The provided versions are not the most recent but work fine. You can download updated versions and replace them, or delete the current ones if you are already have them in your environment.

IMPORTANT 3: The binary release has been modified by leaving resources out of the executable, to prevent false VIRUS positives. However, as a pyinstaller executable/uncertified windows app it may still raise alerts in some antiviruses. Please be aware of this. If you are unsure about it, you can check the code and execute it from its sources with python 3.12,( read the Readme.txt file on how to do it and/or check out the thread for help already offered on that)

Download
V1.16.2 sources (see attachment)
V1.16.2 binaries:

You must be registered to see the links

You must be registered to see the links

You must be registered to see the links

​

 

[Trebutor]

Nice, but somethings are missing:

- Dependencies, I assume Tk and Image.
- How to run, I assume "python vambropy.pyw"
- What to put in config.ini?
- It doesn't seem to do anything right now. Is there a magic hidden button to press?

 

[commondi32]

Nice, but somethings are missing:
...

Thanks for the feedback. Yes, maybe you need pillow (pip install pillow), for images. Tkinter comes bundled with python 3.11 I think.
I have been testing and fixing bugs in the Dependencies section (like buttons not working yet) among other things. I have been testing as many as I can, and now uploading new version with many changes and fixes, some optimizations, and update on all help about new config.ini (reworked with more settings and comments). I think it is now easier and more solid and interactive, with more messages, warnings to avoid little accidents, and customization options. It should run right from the unzipped files. To run by src, go to src dir and 'python vambrpoy.pyw' should work. Or just open vambropy.exe (no python needed). Start the program, click the "Edit config.ini" button and set the correct VAM_dir. Then restart.

 

[commondi32]

I have not much experience uploading and attaching files, and today I managed to upload the new version with executable + sources in the same file because it is smaller now and it fitted. But the legend [VIRUS] was inserted in the attachment name. What is that? Do I need to upload it again? I am sure the file is not infected, I cannot tell why that happened. If there is problem to download that way please tell me and I'll upload separated binary from source if needed.

 

[Franknar]

Just a suggestion, but why not just use symbolic links instead of moving the files?

 

[commondi32]

Just a suggestion, but why not just use symbolic links instead of moving the files?

It could be an option. I was not thinking about it because I don't use it personally, and the app was first conceived to complement sfishere's VarBrowser. And there is already feelfar's VarManager doing symlinks, and recently BrowserAssist. Also, I thought maybe people do not want to try a unknown tool that is asking for admin privileges, since symlinks cannot work at user level. I'll be releasing an update 1.08 soon to fix some bugs introduced in later versions, and then I will try to do some tests. If I manage to make it work and optionally ask for admin when it enables it, I'll add it.

 

[pirate pete]

Getting this error once all packages are loaded by the program, any idea how to fix?:

 

[commondi32]

Getting this error once all packages are loaded by the program, any idea how to fix?: View attachment 4581308

Thanks for the feedback. I think this was one of the bugs fixed in the new version 1.08
EDIT: If it keeps showing on the new version, the problem seem to be with a dependency in a .var file that requires a minimum version, so it is expecting a number but it gets a 'latest', I imagine an incorrect string like "minlatest", as if it was a manual correction to a meta.json, since some people mess with those files manually and corrupt the file. Could be a little hard to pinpoint. If you have rare vars not from the hub or known sources, you can try inspect those vars with VaM Sharp Tools, it might find and fix some bad stuff.
Another way (you may not like much) is to rename your current packages.db and start again with a limited list of vars. If you previously include repos, do it only with vam directory this time, see if it starts without problems, and then add bunchs of vars to the database until the exception shows again
EDIT AGAIN: I will try to add some code to manage that kind of execption in the next version, so it can skip the (bad) package in case of a malformed string , not throwing you out of the app.

 

[commondi32]

Just a suggestion, but why not just use symbolic links instead of moving the files?

Well it turned to be somewhat easy to implement it as an alternative copy method. The menu now it has the option to symlink your package(s) selection, or by pressing <L> key. It works on packages windows, at the moment. Check info on the release (symlink spoilers section) for details. Remember to run Vambropy as admin. If not, options are available but will not do anything more than create relative subdirectories for the links.
Symlinks will not mix with packages, they will copy in AddonPackages/symlinked. And if uninstalled, they will go to AllPackges/symlinked, preserving relative paths.

 

[Franknar]

Well it turned to be somewhat easy to implement it as an alternative copy method. The menu now it has the option to symlink your package(s) selection, or by pressing <L> key. It works on packages windows, at the moment. Check info on the release (symlink spoilers section) for details. Remember to run Vambropy as admin. If not, options are available but will not do anything more than create relative subdirectories for the links.
Symlinks will not mix with packages, they will copy in AddonPackages/symlinked. And if uninstalled, they will go to AllPackges/symlinked, preserving relative paths.

Amazing! Thanks

 

[yuezy3]

great work, but this config.ini may cz program fail because in line 1006 of modules/ui.py :

with open(config_file_path, 'r') as file:

read config.ini but not sepcify encoding. change this line to :

with open(config_file_path, 'r', encoding='utf-8') as file:
will better. it will now can handle other non assic char as path.

 

[commondi32]

with open(config_file_path, 'r', encoding='utf-8') as file:

Thank you! I'll fix it by changing that line as you specified, in next version.

 

[cdt_drecks]

Hi, I get this error:

Traceback (most recent call last):
File "vambropy.pyw", line 79, in <module>
File "vambropy.pyw", line 68, in main
File "modules\ui.py", line 1184, in refresh
TypeError: 'NoneType' object is not subscriptable

 

[commondi32]

Hi, I get this error:

Traceback (most recent call last):
File "vambropy.pyw", line 79, in <module>
File "vambropy.pyw", line 68, in main
File "modules\ui.py", line 1184, in refresh
TypeError: 'NoneType' object is not subscriptable

Hi, you have found a bug in favorites.txt file handling. The program should create an empty one if it does not exists, but somehow an exception occurs. I will fix it in next release. In the meantime, you can fix it two ways:

Simple way: In VaM, use Varbrowser and set any package you like as favorite (it turns blue). That will create the needed file; or

Edit way: Create an empty "favorites.txt" in Custom/PluginData/sfishere directory (the path is inside VaM directory). Open the empty file with a text editor and type: {"FavoriteNames":[]}. Save the file.

Then Vambropy will start ok.

 

[cdt_drecks]

Hi, you have found a bug in favorites.txt file handling. The program should create an empty one if it does not exists, but somehow an exception occurs. I will fix it in next release. In the meantime, you can fix it two ways:

Simple way: In VaM, use Varbrowser and set any package you like as favorite (it turns blue). That will create the needed file; or

Edit way: Create an empty "favorites.txt" in Custom/PluginData/sfishere directory (the path is inside VaM directory). Open the empty file with a text editor and type: {"FavoriteNames":[]}. Save the file.

Then Vambropy will start ok.

Thansk! Now I have this error:


Traceback (most recent call last):
File "vambropy.pyw", line 85, in <module>
File "vambropy.pyw", line 74, in main
File "modules\ui.py", line 1224, in refresh
File "modules\packages.py", line 66, in find_var_names
File "modules\packages.py", line 47, in find_in_all_packages
UnboundLocalError: cannot access local variable 'latest_key' where it is not associated with a value

 

[commondi32]

Thansk! Now I have this error:


Traceback (most recent call last):
File "vambropy.pyw", line 85, in <module>
File "vambropy.pyw", line 74, in main
File "modules\ui.py", line 1224, in refresh
File "modules\packages.py", line 66, in find_var_names
File "modules\packages.py", line 47, in find_in_all_packages
UnboundLocalError: cannot access local variable 'latest_key' where it is not associated with a value

Another one, thanks! Fixed in new version 1.10!

 

[cdt_drecks]

Another one, thanks! Fixed in new version 1.10!

Hey, this error is still present with 1.10 :

Traceback (most recent call last):
File "vambropy.pyw", line 89, in <module>
File "vambropy.pyw", line 78, in main
File "modules\ui.py", line 1224, in refresh
File "modules\packages.py", line 70, in find_var_names
File "modules\packages.py", line 51, in find_in_all_packages
UnboundLocalError: cannot access local variable 'latest_key' where it is not associated with a value

 

[commondi32]

1.10.1 was the one that fixes it!

 

[doggava]

This tool looks cool!

By the way, check this tool out too for development ideas:

You must be registered to see the links

It e.g. removes old versions or vars and has a mapping functionality for missing vars (using symlinks).

 

[luna.kitty]

" In future relases a button could be included to do either: a) in Dependencies section, install all (or filtered) external dependencies as symlinks, or b) Install everyhing external symlinked, as 'Available' , so Varbrowser could handle everything from VaM. "

I really need this useful function, because my tons of var files are separated across several storage locations. I want to install dependencies as symlinks in just two steps - click a specific var, then install dependencies from a Dependencies list as symlinks.
I think this tool is the best VAM tool for me, thank you very much!