This thread is in reference to a Virt-a-Mate plugin developed by sfishere called "Var Browser." I posted this topic because I did not want to clutter other asset release threads wherein the topic of slow load times came up. As you may know massive collection of VARs in AddonPackages significantly slow down load times of VAM and causes game instability. The Var Browser plugin helps relieve these problems for inveterate collectors such as myself.
More information from the author can be found on the hub:
I am NOT the developer. Just a satisfied user. And a not-so-expert user at that since I am somewhat new to VAM.
Here's a moderately long explanation of what sfishere's Var Browser plugin does: Essentially, all of the contents of AddonPackages dir (including sub directories) are initially moved to a new directory called AllPackages. (This archive directory is/would be located at the top level of the VAM directory). With the VAR files moved away from AddonPackages, VAM starts very quickly because it has no VAR packages to scan and load. However, this means that VAM will not know about any of the VARs in your collection. (The AllPackages directory is not part of the original game). Once your VAR files are in AllPackages, you must use Var Browser (I'll abbreviate this to VB) to access those files. VB knows all about the VARs in AllPackages because VB scans the entire AllPackages directory at every start up. This process takes time, but is nowhere near as long as loading VAM with massive VAR collections.
When VB loads a VAR (let's call it xyz.var), the really clever thing that VB does is that it checks for all necessary dependencies of xyz.var and moves the required files (at least those which are in your collection in AllPackages) back to AddonPackages. AND THEN most importantly, VB somehow registers these files with VAM so that you can proceed to use xyz.var. The next time you start up VAM, xyz.var and the dependencies are already in AddonPackages. This means that from that point on, VAM's native browser and plugins such as Browser Assist can see and use that xyz.var. (NOTE that in VB, you will be able to tell which VARs are already located in AddonPackages by a blue border instead of the default white. If you tag a file as Favorite, the border is orange).
Hopefully the experienced VAM user will immediately recognize the speed and stability benefit of having a very slim AddonPackages directory, and yet through VB, you continue to have access to your entire 2TB collection of VARs.
CAVEATS:
HOWEVER, there are some caveats with using Var Browser. If I get something wrong and incorrectly characterize VB, please let me know.
1. You initially start with a very skinny AddonPackages directory which is really fast to load. But as you use VB more and more, VAR files are added back to AddonPackages. AddonPackages will start to fill up again and VAM startup time WILL slow down again. VB has a feature that lets you "uninstall" VARs back to the AllPackages directory. This means that periodically, you can revert back to a really skinny AddonPackages dir and get really fast start up times. (sfishere calls this "Uninstall All"). This is slightly misnamed because he also provides the ability for you to tag VARs to always remain in the AddonPackages (i.e. will not get uninstalled). My point is that VB is not a forever solution and will require maintenance. But this is a win for me because I used to manually move VAR file around and scenes were always broken. Now, not so much.
2. VB generally does a good job of gathering all dependencies from your collection in AllPackages. But I have noticed on more than one occasion that loaded scenes are "broken" because some dependencies were labelled as missing when I know for a fact that I have that VAR. I think this comes from nested sub-dependencies that VB is not able to find during the load. In rare cases like this, I find that a second load of that scene will clear up this hiccup.
3. If you use VB, and most of your VARs are now in AllPackages, you may be surprised the native browser, native hub dependency searcher or JayJayWon's awesome Browser Assist will not see any of those VARs (i.e. they do not exist as far as the game is concerned), Don't worry. They're still there. It's just that the game does not know about AllPackages or its contents. You use VB to access those VARs or you can manually move the files back to AddonPackages. Once you have successfully accessed a VAR through VB, it becomes visible to the native browser then NEXT TIME you run the game. (PIN this thought). I just want to note this because when I first started using VB, I didn't really understand it and was really freaked out that most of my VARs were missing.
4. A core concept (and caveat) is that a VAR must be in AddonPackages at VAM load time for its native browsers and numerous plugins to be able to use it. When VB loads a VAR from AllPackages during an open session, you will be able to use it. But VAM browser, Browser Assist etc. will not see it until the next time you start VAM. This was super important insight for me to understand because I use Browser Assist a lot. The first few times I used VB, most of my VARs were missing from Browser Assist so I thought VB was horribly broken. That was until I realized this quirk. (You really don't need to restart each time you load a VAR. But if you do need to restart, keep in mind that VAM will start a lot faster than before.)
5. For me, a normal use case is that I shove my newly downloaded VARs into AllPackages. I start VAM and use VB to load the VAR so I can check it out. If I really like the VAR and plan on using it for future mods (e.g. my own Looks or music or lighting rigs), I tag the VAR as "Auto Install" in the VB interface (gear icon bottom right of each thumbnail). This feature loosely mentioned above in Caveat #1. When a VAR is tagged as Auto Install, it means that the file is not returned to AllPackages during an "Uninstall All" operation. That VAR (and its dependencies) stay in AddonPackages. Again the caveat is that this means your AddonPackages will grow again - albeit more slowly. But there is something else to be aware of in using the Auto Install tag.
When a thumbnail of a VAR is blue, it means that the VAR is in AddonPackages. Tagging that VAR for Auto Install is super quick. HOWEVER, if the VAR is still in AllPackages (meaning the border is white) tagging it as Auto Install is painfully slow. VAM will lock up while VB scans the VAR and gathers all the dependencies. You may need to know this because of the following use case:
- Migrating your VAR collection for use with VB.
- Checking out a new Asset Collection.
In both of these cases, it may be most efficient to manually move your target VARs to AddonPackages first BEFORE starting VAM. In this way, your target VARs (such as core dependencies or session plugins) are blue which means you can quickly tag them and move on with no delay. Otherwise, it is very painful to have VAM freeze for a couple of minutes every time you tag something.
INITIAL USE and DATA MIGRATION
For you to be interested in this tool, you're probably in a similar predicament of having massive VAR collections. I'm talking terabyte or more. But this probably also means that your collection has some sort of structure and may have some tagging (favorites etc.). The question posed by Darkchampx of not breaking things come to mind. Here is a rough roadmap of what you can do to migrate to using VB. Please note that if you have a lot of saved scenes, the steps below can be quite tedious and time consuming. I did it and I do not regret it.
1. Do a test run to see if you even like VB. The simplest way is rename AddonPackages => AllPackages. Then create an empty directory called AddonPackages. Install VB per sfishere's direction and start VAM. You will be amazed at how fast everything starts. But you will be dismayed that there are no scenes in VAM's native browser etc. Don't freak out. When you open VB, you will see all of your VARs. Using VB's UI, go to Custom/Scene and open one of your saved scenes. See if it loads properly. Depending on the complexity of the scene and how many dependencies, you may have to load it again for all of the dependencies to load. Do this a few times to convince yourself that VB will allow you access to your creations, try loading a scene that you downloaded (i.e. authored by someone else). The point of this exercise is to see if you like it and to build confidence that you won't lose your hard work.
1a. I don't tag with VAM's native tools. So I don't know if moving VARs between AllPackages and AddonPackages breaks the JSON record. You may want to see if your tags survive this test run before committing.
2. Close VAM and go to AddonPackages. In it you will notice that it is no longer empty. It will contain VARs (and sub directories if you created any). These are the files that VB moved over during your test. You will also notice that the dir structure mirrors what is contained within AllPackages. This is key because you can easily revert back to not using VB by copying everything back to AllPackages, deleting the now empty AddonPackages, and then renaming AllPackages => AddonPackages. (I guess you can move the files back but renaming dirs is so much quicker than moving so many files).
3. If you do want to dive in and use VB, do a backup of your files. I know that it's pretty hard to find so much disk space in the case of a 2TB collection, but I was in the midst of building a new rig so I had a brand new drive to test VB with. The old drive became the backup. But please back up. I don't want to be responsible for you mucking up years of your VAR collecting. VB has worked out for me and my wuse case, but you may be doing something different than me and something may break.
4. Once you have everything backed up, go back to step 1 and have all of your collection in AllPackages (with an empty AddonPackages).
4a. Identify all of you core plugins and dependencies and move those to AddonPackages. You can place them in various sub-directories if you choose to organize your VARs that way.
4b. Start up VAM and open up VB. Go to Custom/Scene. This should contain the scenes that you saved. Load every one of your favorite scenes. Load each one at least once. This will take time, but what you're doing is causing VB to migrate all the key dependencies from AllPackages to AddonPackages.
4c. Once you're done loading your favorite saved scenes, go to VB's Category/Scene. At the top, there is a check box for "Only Installed". Click this and you will see only blue thumbnails. Tag every one of these as Auto Install. There is no batch function, so this will be quite tedious. But this is necessary so that you can use other tools such as Browser Assist to access your favorite saved scenes.
4d. There may be scenes from other authors which you use frequently. Load those and tag them as well. Per my Caveat #5, it may be quicker and more efficient to move them manually beforehand. Then you would save yourself some time and would only need to tag them.
THAT'S IT FOR NOW
Whew. I did not mean to write so much. There may be a lot that I have not covered. Others including the author may take issue with what I wrote. But I hope this helps.
Seeing as this is not my work product, I cannot really provide any real support. I'll jump in once in a while to see if I can help, but I make no guarantees. I hope others can contribute to the knowledge base (I am eager to learn features I don't know about).
I encourage you to try it. It took me a while with trial and error, but I am glad of it. Var Browser is indispensable for making this version of VAM work with large collections.
P.S. Unfortunately, sfishere did not feel the love for this project, so he has officially stopped development. It's a shame, but it is absolutely his right to focus his energy on what he wants. But the good news is that what exists adds so much value.
More information from the author can be found on the hub:
You must be registered to see the links
I am NOT the developer. Just a satisfied user. And a not-so-expert user at that since I am somewhat new to VAM.
Here's a moderately long explanation of what sfishere's Var Browser plugin does: Essentially, all of the contents of AddonPackages dir (including sub directories) are initially moved to a new directory called AllPackages. (This archive directory is/would be located at the top level of the VAM directory). With the VAR files moved away from AddonPackages, VAM starts very quickly because it has no VAR packages to scan and load. However, this means that VAM will not know about any of the VARs in your collection. (The AllPackages directory is not part of the original game). Once your VAR files are in AllPackages, you must use Var Browser (I'll abbreviate this to VB) to access those files. VB knows all about the VARs in AllPackages because VB scans the entire AllPackages directory at every start up. This process takes time, but is nowhere near as long as loading VAM with massive VAR collections.
When VB loads a VAR (let's call it xyz.var), the really clever thing that VB does is that it checks for all necessary dependencies of xyz.var and moves the required files (at least those which are in your collection in AllPackages) back to AddonPackages. AND THEN most importantly, VB somehow registers these files with VAM so that you can proceed to use xyz.var. The next time you start up VAM, xyz.var and the dependencies are already in AddonPackages. This means that from that point on, VAM's native browser and plugins such as Browser Assist can see and use that xyz.var. (NOTE that in VB, you will be able to tell which VARs are already located in AddonPackages by a blue border instead of the default white. If you tag a file as Favorite, the border is orange).
Hopefully the experienced VAM user will immediately recognize the speed and stability benefit of having a very slim AddonPackages directory, and yet through VB, you continue to have access to your entire 2TB collection of VARs.
CAVEATS:
HOWEVER, there are some caveats with using Var Browser. If I get something wrong and incorrectly characterize VB, please let me know.
1. You initially start with a very skinny AddonPackages directory which is really fast to load. But as you use VB more and more, VAR files are added back to AddonPackages. AddonPackages will start to fill up again and VAM startup time WILL slow down again. VB has a feature that lets you "uninstall" VARs back to the AllPackages directory. This means that periodically, you can revert back to a really skinny AddonPackages dir and get really fast start up times. (sfishere calls this "Uninstall All"). This is slightly misnamed because he also provides the ability for you to tag VARs to always remain in the AddonPackages (i.e. will not get uninstalled). My point is that VB is not a forever solution and will require maintenance. But this is a win for me because I used to manually move VAR file around and scenes were always broken. Now, not so much.
2. VB generally does a good job of gathering all dependencies from your collection in AllPackages. But I have noticed on more than one occasion that loaded scenes are "broken" because some dependencies were labelled as missing when I know for a fact that I have that VAR. I think this comes from nested sub-dependencies that VB is not able to find during the load. In rare cases like this, I find that a second load of that scene will clear up this hiccup.
3. If you use VB, and most of your VARs are now in AllPackages, you may be surprised the native browser, native hub dependency searcher or JayJayWon's awesome Browser Assist will not see any of those VARs (i.e. they do not exist as far as the game is concerned), Don't worry. They're still there. It's just that the game does not know about AllPackages or its contents. You use VB to access those VARs or you can manually move the files back to AddonPackages. Once you have successfully accessed a VAR through VB, it becomes visible to the native browser then NEXT TIME you run the game. (PIN this thought). I just want to note this because when I first started using VB, I didn't really understand it and was really freaked out that most of my VARs were missing.
4. A core concept (and caveat) is that a VAR must be in AddonPackages at VAM load time for its native browsers and numerous plugins to be able to use it. When VB loads a VAR from AllPackages during an open session, you will be able to use it. But VAM browser, Browser Assist etc. will not see it until the next time you start VAM. This was super important insight for me to understand because I use Browser Assist a lot. The first few times I used VB, most of my VARs were missing from Browser Assist so I thought VB was horribly broken. That was until I realized this quirk. (You really don't need to restart each time you load a VAR. But if you do need to restart, keep in mind that VAM will start a lot faster than before.)
5. For me, a normal use case is that I shove my newly downloaded VARs into AllPackages. I start VAM and use VB to load the VAR so I can check it out. If I really like the VAR and plan on using it for future mods (e.g. my own Looks or music or lighting rigs), I tag the VAR as "Auto Install" in the VB interface (gear icon bottom right of each thumbnail). This feature loosely mentioned above in Caveat #1. When a VAR is tagged as Auto Install, it means that the file is not returned to AllPackages during an "Uninstall All" operation. That VAR (and its dependencies) stay in AddonPackages. Again the caveat is that this means your AddonPackages will grow again - albeit more slowly. But there is something else to be aware of in using the Auto Install tag.
When a thumbnail of a VAR is blue, it means that the VAR is in AddonPackages. Tagging that VAR for Auto Install is super quick. HOWEVER, if the VAR is still in AllPackages (meaning the border is white) tagging it as Auto Install is painfully slow. VAM will lock up while VB scans the VAR and gathers all the dependencies. You may need to know this because of the following use case:
- Migrating your VAR collection for use with VB.
- Checking out a new Asset Collection.
In both of these cases, it may be most efficient to manually move your target VARs to AddonPackages first BEFORE starting VAM. In this way, your target VARs (such as core dependencies or session plugins) are blue which means you can quickly tag them and move on with no delay. Otherwise, it is very painful to have VAM freeze for a couple of minutes every time you tag something.
INITIAL USE and DATA MIGRATION
For you to be interested in this tool, you're probably in a similar predicament of having massive VAR collections. I'm talking terabyte or more. But this probably also means that your collection has some sort of structure and may have some tagging (favorites etc.). The question posed by Darkchampx of not breaking things come to mind. Here is a rough roadmap of what you can do to migrate to using VB. Please note that if you have a lot of saved scenes, the steps below can be quite tedious and time consuming. I did it and I do not regret it.
1. Do a test run to see if you even like VB. The simplest way is rename AddonPackages => AllPackages. Then create an empty directory called AddonPackages. Install VB per sfishere's direction and start VAM. You will be amazed at how fast everything starts. But you will be dismayed that there are no scenes in VAM's native browser etc. Don't freak out. When you open VB, you will see all of your VARs. Using VB's UI, go to Custom/Scene and open one of your saved scenes. See if it loads properly. Depending on the complexity of the scene and how many dependencies, you may have to load it again for all of the dependencies to load. Do this a few times to convince yourself that VB will allow you access to your creations, try loading a scene that you downloaded (i.e. authored by someone else). The point of this exercise is to see if you like it and to build confidence that you won't lose your hard work.
1a. I don't tag with VAM's native tools. So I don't know if moving VARs between AllPackages and AddonPackages breaks the JSON record. You may want to see if your tags survive this test run before committing.
2. Close VAM and go to AddonPackages. In it you will notice that it is no longer empty. It will contain VARs (and sub directories if you created any). These are the files that VB moved over during your test. You will also notice that the dir structure mirrors what is contained within AllPackages. This is key because you can easily revert back to not using VB by copying everything back to AllPackages, deleting the now empty AddonPackages, and then renaming AllPackages => AddonPackages. (I guess you can move the files back but renaming dirs is so much quicker than moving so many files).
3. If you do want to dive in and use VB, do a backup of your files. I know that it's pretty hard to find so much disk space in the case of a 2TB collection, but I was in the midst of building a new rig so I had a brand new drive to test VB with. The old drive became the backup. But please back up. I don't want to be responsible for you mucking up years of your VAR collecting. VB has worked out for me and my wuse case, but you may be doing something different than me and something may break.
4. Once you have everything backed up, go back to step 1 and have all of your collection in AllPackages (with an empty AddonPackages).
4a. Identify all of you core plugins and dependencies and move those to AddonPackages. You can place them in various sub-directories if you choose to organize your VARs that way.
4b. Start up VAM and open up VB. Go to Custom/Scene. This should contain the scenes that you saved. Load every one of your favorite scenes. Load each one at least once. This will take time, but what you're doing is causing VB to migrate all the key dependencies from AllPackages to AddonPackages.
4c. Once you're done loading your favorite saved scenes, go to VB's Category/Scene. At the top, there is a check box for "Only Installed". Click this and you will see only blue thumbnails. Tag every one of these as Auto Install. There is no batch function, so this will be quite tedious. But this is necessary so that you can use other tools such as Browser Assist to access your favorite saved scenes.
4d. There may be scenes from other authors which you use frequently. Load those and tag them as well. Per my Caveat #5, it may be quicker and more efficient to move them manually beforehand. Then you would save yourself some time and would only need to tag them.
THAT'S IT FOR NOW
Whew. I did not mean to write so much. There may be a lot that I have not covered. Others including the author may take issue with what I wrote. But I hope this helps.
Seeing as this is not my work product, I cannot really provide any real support. I'll jump in once in a while to see if I can help, but I make no guarantees. I hope others can contribute to the knowledge base (I am eager to learn features I don't know about).
I encourage you to try it. It took me a while with trial and error, but I am glad of it. Var Browser is indispensable for making this version of VAM work with large collections.
P.S. Unfortunately, sfishere did not feel the love for this project, so he has officially stopped development. It's a shame, but it is absolutely his right to focus his energy on what he wants. But the good news is that what exists adds so much value.
