MiSTer Storage Requirements

For topics which do not fit in other specific forums.
Lamaman1971
Posts: 24
Joined: Fri Aug 06, 2021 11:55 pm
Has thanked: 5 times

MiSTer Storage Requirements

Unread post by Lamaman1971 »

I plan to use my Mister for retro computing - Amiga mainly. Just ordered the parts but left pondering the storage requirements. Would you all recommend a single big SD card or a smaller SD supplemented with a USB hard drive?
ZigZag
Posts: 99
Joined: Sun Jun 28, 2020 7:05 am
Has thanked: 1 time
Been thanked: 5 times

Re: MiSTer Storage Requirements

Unread post by ZigZag »

A big, fast MicroSD. I found HDD speed disappointing.
User avatar
jlancaster86
Posts: 147
Joined: Sat Jun 27, 2020 1:33 pm
Has thanked: 129 times
Been thanked: 35 times

Re: MiSTer Storage Requirements

Unread post by jlancaster86 »

If you have a NAS, you can setup a "MiSTer" SMB share and use that (that's what I do).
Bas
Top Contributor
Posts: 518
Joined: Fri Jan 22, 2021 4:36 pm
Has thanked: 60 times
Been thanked: 225 times

Re: MiSTer Storage Requirements

Unread post by Bas »

Get a big SD. If you're mainly going for Amiga, you'll have plenty of space on anything bigger than 64GB. Big storage mainly comes into play when you start loading CD-ROM images, which isn't really front and center on the Amiga platform. I started my MiSTer with an 8GB card and was actually fine for the most part except CD's. Now I'm on 64GB and have room to spare.
User avatar
bazza_12
Top Contributor
Posts: 404
Joined: Sun May 24, 2020 7:49 pm
Location: Yorkshire, UK
Has thanked: 247 times
Been thanked: 112 times
Contact:

Re: MiSTer Storage Requirements

Unread post by bazza_12 »

defo if you're only using amiga use get a big SDcard.. my amiga folder is 8gig ish.. that includes MegaAGS, Shapeshifter, kickstart roms and 21meg of demos. I agree with Bas above.. 64 gig+
The music is reversible but time is not. Turn back. Turn back
Bas
Top Contributor
Posts: 518
Joined: Fri Jan 22, 2021 4:36 pm
Has thanked: 60 times
Been thanked: 225 times

Re: MiSTer Storage Requirements

Unread post by Bas »

Regarding eggs and baskets.. SD isn't exactly reliable storage, so make backup copies of anything valuable.
User avatar
kathleen
Top Contributor
Posts: 409
Joined: Fri Jun 26, 2020 4:23 am
Location: Belgium
Has thanked: 224 times
Been thanked: 131 times

Re: MiSTer Storage Requirements

Unread post by kathleen »

ZigZag wrote: Sat Aug 07, 2021 2:27 am A big, fast MicroSD. I found HDD speed disappointing.
Is using an external SSD HDD could help the speed concern ? Or is it due to the USB transfer ?

かすりん

ZigZag
Posts: 99
Joined: Sun Jun 28, 2020 7:05 am
Has thanked: 1 time
Been thanked: 5 times

Re: MiSTer Storage Requirements

Unread post by ZigZag »

I think it's USB hub speed. Could just be my set up, but I find MicroSD faster than an external SSD. It's noticeable mostly with cores like PC & Amiga. This is just my personal experience though, and I haven't updated in a few months.
Solskogen
Posts: 89
Joined: Mon May 25, 2020 5:33 am
Has thanked: 11 times
Been thanked: 6 times

Re: MiSTer Storage Requirements

Unread post by Solskogen »

I've found that ethernet is faster than USB and SD card, so I use a small/cheap sdcard for the cores and data on SMB(NAS).
User avatar
RealLarry
Top Contributor
Posts: 767
Joined: Mon May 25, 2020 4:04 am
Location: San Junipero/DE/Earth
Has thanked: 85 times
Been thanked: 328 times

Re: MiSTer Storage Requirements

Unread post by RealLarry »

Solskogen wrote: Mon Aug 09, 2021 10:55 am I've found that ethernet is faster than USB and SD card, so I use a small/cheap sdcard for the cores and data on SMB(NAS).
An ethernet connection is faster. That's why I'm using an old and decommissioned mini PC for MiSTer's data, which is booting up a stripped down Debian for max. speed and greatest flexibility. I've tested 256GB SD cards and ready-to-run NAS' but wasn't ever satisfied. Better building such things by myself, also in terms of data privacy and trustworthiness.
Contributor of tty2oled, author of tty2tft, tty2rpi and update_tty2xxx
akeley
Top Contributor
Posts: 1303
Joined: Mon May 25, 2020 7:54 pm
Has thanked: 416 times
Been thanked: 399 times

Re: MiSTer Storage Requirements

Unread post by akeley »

I'm fine with my SD card's speed. I mean, how much faster can the other methods be, when you consider the kind of data we're shifting? ;)
Bas
Top Contributor
Posts: 518
Joined: Fri Jan 22, 2021 4:36 pm
Has thanked: 60 times
Been thanked: 225 times

Re: MiSTer Storage Requirements

Unread post by Bas »

The LAN interface is 100Mbit right? That caps the network at around 12MB/s throughput, which is faster than just about anything you can expect from a retro core except maybe big box Amiga (which MiSTer doesn't have) or AO486. Any faster won't be useful until you reach the extremes like Saturn or Playstation, but probably not even those.
elvis
Posts: 62
Joined: Sun May 24, 2020 9:25 pm
Has thanked: 41 times
Been thanked: 35 times

Re: MiSTer Storage Requirements

Unread post by elvis »

Bas wrote: Mon Aug 09, 2021 7:01 pm The LAN interface is 100Mbit right?
The DE10nano's Ethernet port is gigabit, full duplex.

The USB port is USB2, and the issue that USB2 is half duplex. For storage, that's a larger impact than its 480Mbit/s bandwidth.

I already had a NAS configured for use centrally in my house across multiple gaming systems, modded consoles, arcade machines, TVs, etc. So that was a simple choice for MiSTer. The speed is certainly nicer than either SD cards or USB2 connected storage.

Additional features like transparent compression, deduplication, snapshots and redundancy courtesy of BtrFS are all nice too.
User avatar
aberu
Core Developer
Posts: 1144
Joined: Tue Jun 09, 2020 8:34 pm
Location: Longmont, CO
Has thanked: 244 times
Been thanked: 388 times
Contact:

Re: MiSTer Storage Requirements

Unread post by aberu »

I agree with most people here, a big MicroSD is still pretty cheap. A 128GB if you are primarily just doing Amiga and some other stuff. If you get into AO486, then 256GB IMO. If you need more you just plop the extras on a usb stick and navigate to the extra files there.
birdybro~
Bas
Top Contributor
Posts: 518
Joined: Fri Jan 22, 2021 4:36 pm
Has thanked: 60 times
Been thanked: 225 times

Re: MiSTer Storage Requirements

Unread post by Bas »

Elvis, did you benchmark the speeds?
User avatar
bazza_12
Top Contributor
Posts: 404
Joined: Sun May 24, 2020 7:49 pm
Location: Yorkshire, UK
Has thanked: 247 times
Been thanked: 112 times
Contact:

Re: MiSTer Storage Requirements

Unread post by bazza_12 »

Bas wrote: Tue Aug 10, 2021 5:23 am Elvis, did you benchmark the speeds?
he's left the building...




sorry.. couldn't resist :lol: :lol:
The music is reversible but time is not. Turn back. Turn back
elvis
Posts: 62
Joined: Sun May 24, 2020 9:25 pm
Has thanked: 41 times
Been thanked: 35 times

Re: MiSTer Storage Requirements

Unread post by elvis »

Bas wrote: Tue Aug 10, 2021 5:23 am Elvis, did you benchmark the speeds?
Using iperf3 for raw TCP performance in both directions:

MiSTer to NAS:

Code: Select all

/root# iperf3 -i1 -c 192.168.3.254
Connecting to host 192.168.3.254, port 5201
[  5] local 192.168.3.47 port 57584 connected to 192.168.3.254 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec  75.0 MBytes   623 Mbits/sec    0    167 KBytes       
[  5]   1.01-2.01   sec  73.8 MBytes   619 Mbits/sec    0    177 KBytes       
[  5]   2.01-3.01   sec  75.0 MBytes   631 Mbits/sec    0    177 KBytes       
[  5]   3.01-4.01   sec  75.0 MBytes   630 Mbits/sec    0    177 KBytes       
[  5]   4.01-5.01   sec  75.0 MBytes   629 Mbits/sec    0    177 KBytes       
[  5]   5.01-6.01   sec  75.0 MBytes   630 Mbits/sec    0    177 KBytes       
[  5]   6.01-7.01   sec  73.8 MBytes   618 Mbits/sec    0    177 KBytes       
[  5]   7.01-8.00   sec  75.0 MBytes   631 Mbits/sec    0    177 KBytes       
[  5]   8.00-9.00   sec  75.0 MBytes   632 Mbits/sec    0    177 KBytes       
[  5]   9.00-10.02  sec  76.2 MBytes   630 Mbits/sec    0    177 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec   749 MBytes   627 Mbits/sec    0             sender
[  5]   0.00-10.05  sec   749 MBytes   625 Mbits/sec                  receiver
iperf Done.
NAS to MiSTer

Code: Select all

/root# iperf3 -i1 -R -c 192.168.3.254
Connecting to host 192.168.3.254, port 5201
Reverse mode, remote host 192.168.3.254 is sending
[  5] local 192.168.3.47 port 57588 connected to 192.168.3.254 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  85.6 MBytes   718 Mbits/sec                  
[  5]   1.00-2.00   sec  85.1 MBytes   714 Mbits/sec                  
[  5]   2.00-3.00   sec  87.1 MBytes   730 Mbits/sec                  
[  5]   3.00-4.00   sec  86.1 MBytes   722 Mbits/sec                  
[  5]   4.00-5.01   sec  87.0 MBytes   726 Mbits/sec                  
[  5]   5.01-6.00   sec  87.9 MBytes   738 Mbits/sec                  
[  5]   6.00-7.01   sec  84.9 MBytes   712 Mbits/sec                  
[  5]   7.01-8.01   sec  87.4 MBytes   732 Mbits/sec                  
[  5]   8.01-9.00   sec  86.7 MBytes   732 Mbits/sec                  
[  5]   9.00-10.00  sec  86.4 MBytes   724 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   867 MBytes   725 Mbits/sec  222             sender
[  5]   0.00-10.00  sec   864 MBytes   725 Mbits/sec                  receiver
iperf Done.
And a real-world test: Here's my CIFS mount with my ROMS. 235MB of files in this particular directory. I use the Linux command "cat" to stream them over the CIFS mount, pass them through a tool called "pv" (short for "pipe view") which just reports the transfer speeds, and then throws them away (passed to /dev/null). This is the same as reading files into memory on MiSTer.

Before any disk-read tests, I send "3" to a special file called /proc/sys/vm/drop_caches. This tells Linux's local file system cache to invalidate all entries and empty the cache, forcing a new read from the backing storage. Note that this doesn't clear the cache on the remote system (i.e: my NAS still has it's own cache, independent of clients connected).

Code: Select all

/media/fat/SNES/1 US - A-E# echo '3' > /proc/sys/vm/drop_caches ; cat * | pv >/dev/null
 235MiB 0:00:05 [45.8MiB/s] 
 
45.8MB/s, or around 400Mbit/s real world. My NAS's storage is much faster than that (can quite happily saturate a couple of 1GbE PCs in the house while feeding media to my TVs - storage was idle at time of benchmark), so the bottleneck there is definitely the DE10-Nano itself.

Let's compare reading off the MicroSD card I have. It's not particularly impressive or high speed, a SanDisk Ultra 64GB Micro SDXC UHS Speed Class 1, rated to 10MB/s. Using MiSTer's local Linux image:

Code: Select all

media/fat/linux# echo '3' > /proc/sys/vm/drop_caches ; cat linux.img | pv >/dev/null
 312MiB 0:00:14 [22.3MiB/s]
Exceeds the minimum 10MB/s guarantee by a factor of 2, but still slower than my NAS. I don't have faster MicroSD cards to test, as again I can't be stuffed buying them when my NAS has more convenient features than just raw speed. So I'm not sure what the maximum speed through the MiSTer's MicroSD slot is.
User avatar
drpaneas
Posts: 7
Joined: Thu Aug 12, 2021 5:06 pm
Has thanked: 2 times

Re: MiSTer Storage Requirements

Unread post by drpaneas »

Easier thing is to use an SD Card.

But I second what @Elvis said -- the network connection is faster. I did lot's of benchmarks with iperf3 and game streaming (using Moonlight -- this is outside of the context of Mister though) and the speed was mindblowing, even with 5G wifi. If you can setup a network storage, go for it ;)
german_user
Posts: 97
Joined: Wed Sep 08, 2021 12:09 pm
Has thanked: 25 times
Been thanked: 13 times

Re: MiSTer Storage Requirements

Unread post by german_user »

Only or mainly for the Amiga Core I would stick with an SD card. I am currently using inexpensive 64GB cards myself, these cost less than € 4 each and are relatively easy to copy. CD32 games isos are the real memory hogs, the same games as the A1200 version rarely use more than 200MB.
elvis
Posts: 62
Joined: Sun May 24, 2020 9:25 pm
Has thanked: 41 times
Been thanked: 35 times

Re: MiSTer Storage Requirements

Unread post by elvis »

PorkChopExpress from MiSTeraddons has published a great breakdown on MiSTer storage. Confirming my previous findings that wired Ethernet is by far the fastest option for storage.

https://misteraddons.com/blogs/news/mis ... d-shootout

I've got a project in the works to assist people with configuring network storage for MiSTer. When that's ready for public consumption, I'll make a separate post.
Post Reply