rpdl.net discussion & questions (torrents)
- Thread starter RPDL
- Start date
- Jul 29, 2017
- 9,113
- 106,578
Now that I think about it (I'm a layman about this stuff) - is there any reason you torrent the zip file, instead of just the game folder?
And again, thank you for this service.
And again, thank you for this service.
- Dec 17, 2020
- 5,082
- 14,628
That download speed tho....Now that I think about it (I'm a layman about this stuff) - is there any reason you torrent the zip file, instead of just the game folder?
And again, thank you for this service.
Also yeah, the reason for a zip is a server/linux thing, I go into it a little here.
Also, in other news. I'm going to deprecate my repeated and stubborn attempts to bundle PC/Linux and Mac all into one because it's never quite working properly and it isn't fair to people that download a big game if it doesn't work.
I'll still unzip and scan the files though as that's actually useful.
I don't know if this is a common/standard thing for you to do, but I noticed you used a piece size of 1MiB for a torrent that was over 4000MiB in size (resulting in equally as many pieces).
Any reason you're not following the ~1000 piece guideline? Even uTorrent <3.0 (which no one should be using at this point) supports up to 16MiB pieces.
Having too many pieces in a torrent can slow transfers down noticeably if it gets high enough, and bloats ram usage by the client. There may also be an impact on the disk head when it comes to seeding (though unlike the other two problems, I can't personally confirm this).
There are math equations that can be used to create a .torrent with its ideal piece size such as:
Could easily be capped at a max size as well, for example for 64MiB:
PS. While I understand your reasoning for serving zips, it's a damn shame for people that might want to seed, since they'd have to store the game twice to do so.
Any reason you're not following the ~1000 piece guideline? Even uTorrent <3.0 (which no one should be using at this point) supports up to 16MiB pieces.
Having too many pieces in a torrent can slow transfers down noticeably if it gets high enough, and bloats ram usage by the client. There may also be an impact on the disk head when it comes to seeding (though unlike the other two problems, I can't personally confirm this).
There are math equations that can be used to create a .torrent with its ideal piece size such as:
Code:
piece_size_in_bytes = 2 ** (int(math.log(size_of_data_in_bytes) / math.log(2)) - 9)
Code:
piece_size_in_bytes = 2 ** min((int(math.log(size_of_data_in_bytes) / math.log(2)) - 9), 26)
- Jul 11, 2019
- 73
- 586
since nopy went offline this is my go to way of downloading. a quick question or maybe a suggestion - why no magnet links?
- Dec 17, 2020
- 5,082
- 14,628
I'm 99% sure I already factored in the math for chunk size, shouldn't it be approximately 1.5k chunks per torrent?
IE As the size of the file goes up, the size of the chunks scale up too (but amount of chunks stay the same)
edit: "but I noticed you used a piece size of 1MiB for a torrent that was over 4000MiB in size (resulting in equally as many pieces)."
Wait a sec, you're right... I need to look into this.
Also yeah, the zip thing - It's a trade off, people shouldn't feel bad if they can't seed - they don't need to. That's my job, I'm perfectly fine with people hit-n-running torrents if they please
I'm waiting on a pull request to incorporate this (waiting 2 months, it's a slow project I guess).
Last edited:
Well, at 1MiB per piece, this torrent would be 4372 pieces https://f95zone.to/threads/midlife-crisis-v0-27-public-nefastus-games.20427/post-6982850
As I said originally, I wasn't sure if this was something you did on the regular or not, this just happened to be the first one I saw.
Ideally, yes. 500-1500 chunks is the 'best practice' as far as I'm aware. Or generally, whatever piece size gets you closest to 1k without being obscene (like 2GiB pieces for a 1TiB torrent).
The torrent above should have a piece size of 4-8MiB for it to work out this way.
-10 instead of -9 might give better results actually, -9 errs on the side of not having too many pieces but often having closer to 500 than 1000 in doing so.
It's really not a huge deal, I just know from seeding a lot of torrents in my time that incorrectly sized torrents can start to cause issues when scaling.
- Dec 17, 2020
- 5,082
- 14,628
Yup, you're very correct. I think I've made a mistake in how it's calculated.
if [ $RAWSIZE -lt 512 ];
then
p=18
p1="256KiB"
elif [ $RAWSIZE -lt 1024 ];
then
p=19
p1="512KiB"
elif [ $RAWSIZE -lt 5120 ];
then
p=20
p1="1MiB"
elif [ $RAWSIZE -lt 10240 ];
then
p=21
p1="2MiB"
elif [ $RAWSIZE -lt 15360 ];
then
p=22
p1="4MiB"
else
p=22
p1="4MiB"
fiOoops, I'll fix it shortly (it's a very busy day today for some reason????)
Thanks so much for reporting this, I was 100% unaware!
- Dec 17, 2020
- 5,082
- 14,628
Just doing some spring cleaning:
- I updated the OP of this thread here and my
You must be registered to see the links. In short, I'm no longer attempting to bundle PC/Linux/Mac into one zip, it's now PC/Linux only.
- I added the discord link to the top of the thread too and to the footer on
You must be registered to see the links
- I posted a super short
You must be registered to see the linksthing here about it (boooooooring)
- Fixing the chunk-size discrepancy bigguy_foryou reported (thank you!)
- Work on a concrete funding plan (oof)
- I'm going to migrate all the download links from
https://rpdl.net/torrents/tohttps://files.rpdl.net/static/ - Maybe work on an RSS thing for torrents??
- try out the Being a DIK update lol.
Last edited:
I would love to support you through patreon but even 1 USD is a big deal for me. (Thank you exchange rates) but I have to tell you you are doing good work. Network is great and fast I have 100 Mbps download and I get full speed. You may want to add magnet links too.
My upload is crappy 10 Mbits but I'll seed the games I like forever XD This is the least I can do
My upload is crappy 10 Mbits but I'll seed the games I like forever XD This is the least I can do
- Dec 17, 2020
- 5,082
- 14,628
I promise, the fact that you're getting usage out of all this is enough for me
Yeah, I'm very ready to add magnets, I'm just waiting on the tools I use to support them.
I don't like installing additional packages & programs for minor stuff so once magnets are supported, I'll also add them here ASAP.
Thanks very much for the kind words too
- May 18, 2018
- 241
- 267
Hello rpdl ! Can you try out this game, plase? "Indencent Desires" ---- https://f95zone.to/threads/indecent-desires-the-game-v0-018-vilelab.11623/
Once again Im gonna to THANK YOU for your efforts and persistence. Please keep up the damn good job!
Once again Im gonna to THANK YOU for your efforts and persistence. Please keep up the damn good job!
- Dec 17, 2020
- 5,082
- 14,628
- Dec 17, 2020
- 5,082
- 14,628
bigguy_foryou what do you think of this revision?
using this as an example (you can ignore the p1 variable, it's just what I use to catch the piece size for my comments.) The important part is the rawsize & p variable
IE if bigger than 1gb but smaller than 2gb, use 20 (1mb piece size)
Here are the base numbers too for context:
Additionally, I've been searching about to be sure and every source I can find lists the "optimal" as either ~1500 or between 1200-2200.
Let me know if you'd make any changes, I think maybe in January I might go back through ALL old torrents and update them for the new sizes - along with obviously creating all torrents from now with the new adjusted numbers.
if [ $RAWSIZE -lt 512 ];
then
p=18
p1="256KiB"
elif [ $RAWSIZE -lt 1024 ];
then
p=19
p1="512KiB"
elif [ $RAWSIZE -lt 2048 ];
then
p=20
p1="1MiB"
elif [ $RAWSIZE -lt 4096 ];
then
p=21
p1="2MiB"
elif [ $RAWSIZE -lt 8192 ];
then
p=22
p1="4MiB"
elif [ $RAWSIZE -lt 16384 ];
then
p=23
p1="8MiB"
else
p=24
p1="16MiB"
fiusing this as an example (you can ignore the p1 variable, it's just what I use to catch the piece size for my comments.) The important part is the rawsize & p variable
elif [ $RAWSIZE -lt 2048 ];
then
p=20
p1="1MiB"IE if bigger than 1gb but smaller than 2gb, use 20 (1mb piece size)
Here are the base numbers too for context:
The tl;dr is if between 1-2gb, use piece size of 1mb - which (imo) seems to be the best way of going about things.
Additionally, I've been searching about to be sure and every source I can find lists the "optimal" as either ~1500 or between 1200-2200.
Let me know if you'd make any changes, I think maybe in January I might go back through ALL old torrents and update them for the new sizes - along with obviously creating all torrents from now with the new adjusted numbers.
Yeah that looks completely fine to me.
Regarding the 'optimal' piece count, you've probably looked into it more than me at this point, I was just going off what I remember being suggested to me when I got into it. All I can say is that the more pieces a torrent has: the more fragmented it might be (fixed by preallocating), the more ram the client will use, the worse the penalty is for failing to transfer a piece successfully (either by corruption, connection drop, or just stopping the torrent). I've also seen suggestions of 1000-2000 so you're probably totally fine here, as it seems to be the generally accepted middle ground.
I will note that the 16MiB piece cap is a legacy suggestion due to uTorrent 2's lack of support for anything higher, no one should be using that client anymore so I wouldn't feel bad about breaking it. But you're also probably never going to serve torrents from here that would require anything higher than that anyway.
I don't know how many torrents you've created so far, but I imagine that many of them will eventually get automatically replaced anyway when updates come out.
Regarding the 'optimal' piece count, you've probably looked into it more than me at this point, I was just going off what I remember being suggested to me when I got into it. All I can say is that the more pieces a torrent has: the more fragmented it might be (fixed by preallocating), the more ram the client will use, the worse the penalty is for failing to transfer a piece successfully (either by corruption, connection drop, or just stopping the torrent). I've also seen suggestions of 1000-2000 so you're probably totally fine here, as it seems to be the generally accepted middle ground.
I will note that the 16MiB piece cap is a legacy suggestion due to uTorrent 2's lack of support for anything higher, no one should be using that client anymore so I wouldn't feel bad about breaking it. But you're also probably never going to serve torrents from here that would require anything higher than that anyway.
I don't know how many torrents you've created so far, but I imagine that many of them will eventually get automatically replaced anyway when updates come out.
- Dec 17, 2020
- 5,082
- 14,628
Yeah exactly, 20gb would be an outlier rather than the norm and yeah, when a game is updated i move my servers from the old one to the new one.
As it is right now, it's 510 torrents (i somehow missed the 500 milestone). I've pushed servers past 6,000 in the past but it gets ugly, either way I don't really need to worry about that for a while.
Thanks too, I've pushed those changes live and I'll probably remake all old torrents in the first 1-2 months of 2022, just to be safe.
- Nov 22, 2021
- 15
- 3
- Jul 11, 2019
- 73
- 586