rpdl.net discussion & questions (torrents)

L

loverloverlover

Newbie
Dec 7, 2019
53
84
Now that I think about it (I'm a layman about this stuff) - is there any reason you torrent the zip file, instead of just the game folder?

And again, thank you for this service.

 
RPDL

RPDL

do torrent. go fast.
Donor
Dec 17, 2020
4,843
13,901
loverloverlover said:
Now that I think about it (I'm a layman about this stuff) - is there any reason you torrent the zip file, instead of just the game folder?

And again, thank you for this service.

Click to expand...
That download speed tho....

Also yeah, the reason for a zip is a server/linux thing, I go into it a little here.


Also, in other news. I'm going to deprecate my repeated and stubborn attempts to bundle PC/Linux and Mac all into one because it's never quite working properly and it isn't fair to people that download a big game if it doesn't work.

I'll still unzip and scan the files though as that's actually useful.
 
  • Like
Reactions: loverloverlover
B

bigguy_foryou

Well-Known Member
Jun 8, 2018
1,018
872
I don't know if this is a common/standard thing for you to do, but I noticed you used a piece size of 1MiB for a torrent that was over 4000MiB in size (resulting in equally as many pieces).
Any reason you're not following the ~1000 piece guideline? Even uTorrent <3.0 (which no one should be using at this point) supports up to 16MiB pieces.
Having too many pieces in a torrent can slow transfers down noticeably if it gets high enough, and bloats ram usage by the client. There may also be an impact on the disk head when it comes to seeding (though unlike the other two problems, I can't personally confirm this).

There are math equations that can be used to create a .torrent with its ideal piece size such as:
Code: 
piece_size_in_bytes = 2 ** (int(math.log(size_of_data_in_bytes) / math.log(2)) - 9)
Could easily be capped at a max size as well, for example for 64MiB:
Code: 
piece_size_in_bytes = 2 ** min((int(math.log(size_of_data_in_bytes) / math.log(2)) - 9), 26)
PS. While I understand your reasoning for serving zips, it's a damn shame for people that might want to seed, since they'd have to store the game twice to do so.
 
RPDL

RPDL

do torrent. go fast.
Donor
Dec 17, 2020
4,843
13,901
bigguy_foryou said:
I don't know if this is a common/standard thing for you to do, but I noticed you used a piece size of 1MiB for a torrent that was over 4000MiB in size (resulting in equally as many pieces).
Any reason you're not following the ~1000 piece guideline? Even uTorrent <3.0 (which no one should be using at this point) supports up to 16MiB pieces.
Having too many pieces in a torrent can slow transfers down noticeably if it gets high enough, and bloats ram usage by the client. There may also be an impact on the disk head when it comes to seeding (though unlike the other two problems, I can't personally confirm this).

There are math equations that can be used to create a .torrent with its ideal piece size such as:
Code: 
piece_size_in_bytes = 2 ** (int(math.log(size_of_data_in_bytes) / math.log(2)) - 9)
Could easily be capped at a max size as well, for example for 64MiB:
Code: 
piece_size_in_bytes = 2 ** min((int(math.log(size_of_data_in_bytes) / math.log(2)) - 9), 26)
PS. While I understand your reasoning for serving zips, it's a damn shame for people that might want to seed, since they'd have to store the game twice to do so.
Click to expand...
I'm 99% sure I already factored in the math for chunk size, shouldn't it be approximately 1.5k chunks per torrent?
IE As the size of the file goes up, the size of the chunks scale up too (but amount of chunks stay the same)

edit: "but I noticed you used a piece size of 1MiB for a torrent that was over 4000MiB in size (resulting in equally as many pieces)."

Wait a sec, you're right... I need to look into this.

Also yeah, the zip thing - It's a trade off, people shouldn't feel bad if they can't seed - they don't need to. That's my job, I'm perfectly fine with people hit-n-running torrents if they please :)
red-NINE said:
since nopy went offline this is my go to way of downloading. a quick question or maybe a suggestion - why no magnet links?
Click to expand...
I'm waiting on a pull request to incorporate this (waiting 2 months, it's a slow project I guess).
 
Last edited:
B

bigguy_foryou

Well-Known Member
Jun 8, 2018
1,018
872
rpdl said:
I'm 99% sure I already factored in the math for chunk size, shouldn't it be approximately 1.5k chunks per torrent?
Click to expand...
Well, at 1MiB per piece, this torrent would be 4372 pieces https://f95zone.to/threads/midlife-crisis-v0-27-public-nefastus-games.20427/post-6982850
As I said originally, I wasn't sure if this was something you did on the regular or not, this just happened to be the first one I saw.

rpdl said:
IE As the size of the file goes up, the size of the chunks scale up too (but amount of chunks stay the same)
Click to expand...
Ideally, yes. 500-1500 chunks is the 'best practice' as far as I'm aware. Or generally, whatever piece size gets you closest to 1k without being obscene (like 2GiB pieces for a 1TiB torrent).
The torrent above should have a piece size of 4-8MiB for it to work out this way.

-10 instead of -9 might give better results actually, -9 errs on the side of not having too many pieces but often having closer to 500 than 1000 in doing so.

It's really not a huge deal, I just know from seeding a lot of torrents in my time that incorrectly sized torrents can start to cause issues when scaling.
 
  • Like
Reactions: gojira667
RPDL

RPDL

do torrent. go fast.
Donor
Dec 17, 2020
4,843
13,901
bigguy_foryou said:
Well, at 1MiB per piece, this torrent would be 4372 pieces https://f95zone.to/threads/midlife-crisis-v0-27-public-nefastus-games.20427/post-6982850
As I said originally, I wasn't sure if this was something you did on the regular or not, this just happened to be the first one I saw.


Ideally, yes. 500-1500 chunks is the 'best practice' as far as I'm aware. Or generally, whatever piece size gets you closest to 1k without being obscene (like 2GiB pieces for a 1TiB torrent).
The torrent above should have a piece size of 4-8MiB for it to work out this way.

-10 instead of -9 might give better results actually, -9 errs on the side of not having too many pieces but often having closer to 500 than 1000 in doing so.

It's really not a huge deal, I just know from seeding a lot of torrents in my time that incorrectly sized torrents can start to cause issues when scaling.
Click to expand...
Yup, you're very correct. I think I've made a mistake in how it's calculated.

if [ $RAWSIZE -lt 512 ];
then
p=18
p1="256KiB"
elif [ $RAWSIZE -lt 1024 ];
then
p=19
p1="512KiB"
elif [ $RAWSIZE -lt 5120 ];
then
p=20
p1="1MiB"
elif [ $RAWSIZE -lt 10240 ];
then
p=21
p1="2MiB"
elif [ $RAWSIZE -lt 15360 ];
then
p=22
p1="4MiB"
else
p=22
p1="4MiB"
fi


Ooops, I'll fix it shortly (it's a very busy day today for some reason????)

Thanks so much for reporting this, I was 100% unaware!
 
RPDL

RPDL

do torrent. go fast.
Donor
Dec 17, 2020
4,843
13,901
Just doing some spring cleaning:
  1. I updated the OP of this thread here and my
    You must be registered to see the links
    . In short, I'm no longer attempting to bundle PC/Linux/Mac into one zip, it's now PC/Linux only.
  2. I added the discord link to the top of the thread too and to the footer on
    You must be registered to see the links
  3. I posted a super short
    You must be registered to see the links
    thing here about it (boooooooring)
I've been quashing a lot of bugs & annoyances today so it's been busy. My to-do list is:
  1. Fixing the chunk-size discrepancy bigguy_foryou reported (thank you!)
  2. Work on a concrete funding plan (oof)
  3. I'm going to migrate all the download links from https://rpdl.net/torrents/ to https://files.rpdl.net/static/
  4. Maybe work on an RSS thing for torrents??
  5. try out the Being a DIK update lol.
Thanks everyone! :BootyTime:
 
Last edited:
  • Like
Reactions: loverloverlover and LightmanP
S

Sancrest

Member
Jul 5, 2018
106
159
I would love to support you through patreon but even 1 USD is a big deal for me. (Thank you exchange rates) but I have to tell you you are doing good work. Network is great and fast I have 100 Mbps download and I get full speed. You may want to add magnet links too.

My upload is crappy 10 Mbits but I'll seed the games I like forever XD This is the least I can do
 
RPDL

RPDL

do torrent. go fast.
Donor
Dec 17, 2020
4,843
13,901
Sancrest said:
I would love to support you through patreon but even 1 USD is a big deal for me. (Thank you exchange rates) but I have to tell you you are doing good work. Network is great and fast I have 100 Mbps download and I get full speed. You may want to add magnet links too.

My upload is crappy 10 Mbits but I'll seed the games I like forever XD This is the least I can do
Click to expand...
I promise, the fact that you're getting usage out of all this is enough for me :)

Yeah, I'm very ready to add magnets, I'm just waiting on the tools I use to support them.
I don't like installing additional packages & programs for minor stuff so once magnets are supported, I'll also add them here ASAP.

Thanks very much for the kind words too :love:
 
  • Like
Reactions: Sancrest
RPDL

RPDL

do torrent. go fast.
Donor
Dec 17, 2020
4,843
13,901
bigguy_foryou what do you think of this revision?

if [ $RAWSIZE -lt 512 ];
then
p=18
p1="256KiB"
elif [ $RAWSIZE -lt 1024 ];
then
p=19
p1="512KiB"
elif [ $RAWSIZE -lt 2048 ];
then
p=20
p1="1MiB"
elif [ $RAWSIZE -lt 4096 ];
then
p=21
p1="2MiB"
elif [ $RAWSIZE -lt 8192 ];
then
p=22
p1="4MiB"
elif [ $RAWSIZE -lt 16384 ];
then
p=23
p1="8MiB"
else
p=24
p1="16MiB"
fi

using this as an example (you can ignore the p1 variable, it's just what I use to catch the piece size for my comments.) The important part is the rawsize & p variable
elif [ $RAWSIZE -lt 2048 ];
then
p=20
p1="1MiB"

IE if bigger than 1gb but smaller than 2gb, use 20 (1mb piece size)

Here are the base numbers too for context:
219 = 524 288 = 512 KiB for filesizes between 512 MiB - 1024 MiB
220 = 1 048 576 = 1024 KiB for filesizes between 1 GiB - 2 GiB
221 = 2 097 152 = 2048 KiB for filesizes between 2 GiB - 4 GiB
222 = 4 194 304 = 4096 KiB for filesizes between 4 GiB - 8 GiB
223 = 8 388 608 = 8192 KiB for filesizes between 8 GiB - 16 GiB
224 = 16 777 216 = 16384 KiB for filesizes between 16 GiB - 512 GiB
Click to expand...
The tl;dr is if between 1-2gb, use piece size of 1mb - which (imo) seems to be the best way of going about things.

Additionally, I've been searching about to be sure and every source I can find lists the "optimal" as either ~1500 or between 1200-2200.
Let me know if you'd make any changes, I think maybe in January I might go back through ALL old torrents and update them for the new sizes - along with obviously creating all torrents from now with the new adjusted numbers.
 
B

bigguy_foryou

Well-Known Member
Jun 8, 2018
1,018
872
Yeah that looks completely fine to me.
Regarding the 'optimal' piece count, you've probably looked into it more than me at this point, I was just going off what I remember being suggested to me when I got into it. All I can say is that the more pieces a torrent has: the more fragmented it might be (fixed by preallocating), the more ram the client will use, the worse the penalty is for failing to transfer a piece successfully (either by corruption, connection drop, or just stopping the torrent). I've also seen suggestions of 1000-2000 so you're probably totally fine here, as it seems to be the generally accepted middle ground.

I will note that the 16MiB piece cap is a legacy suggestion due to uTorrent 2's lack of support for anything higher, no one should be using that client anymore so I wouldn't feel bad about breaking it. But you're also probably never going to serve torrents from here that would require anything higher than that anyway.

I don't know how many torrents you've created so far, but I imagine that many of them will eventually get automatically replaced anyway when updates come out.
 
RPDL

RPDL

do torrent. go fast.
Donor
Dec 17, 2020
4,843
13,901
bigguy_foryou said:
Yeah that looks completely fine to me.
Regarding the 'optimal' piece count, you've probably looked into it more than me at this point, I was just going off what I remember being suggested to me when I got into it. All I can say is that the more pieces a torrent has: the more fragmented it might be (fixed by preallocating), the more ram the client will use, the worse the penalty is for failing to transfer a piece successfully (either by corruption, connection drop, or just stopping the torrent). I've also seen suggestions of 1000-2000 so you're probably totally fine here, as it seems to be the generally accepted middle ground.

I will note that the 16MiB piece cap is a legacy suggestion due to uTorrent 2's lack of support for anything higher, no one should be using that client anymore so I wouldn't feel bad about breaking it. But you're also probably never going to serve torrents from here that would require anything higher than that anyway.

I don't know how many torrents you've created so far, but I imagine that many of them will eventually get automatically replaced anyway when updates come out.
Click to expand...
Yeah exactly, 20gb would be an outlier rather than the norm and yeah, when a game is updated i move my servers from the old one to the new one.

As it is right now, it's 510 torrents (i somehow missed the 500 milestone). I've pushed servers past 6,000 in the past but it gets ugly, either way I don't really need to worry about that for a while.

Thanks too, I've pushed those changes live and I'll probably remake all old torrents in the first 1-2 months of 2022, just to be safe.
 
You must log in or register to reply here.
Top Bottom