Page 1 of 1
MiSTer Storage Requirements
Posted: Fri Aug 06, 2021 11:57 pm
by Lamaman1971
I plan to use my Mister for retro computing - Amiga mainly. Just ordered the parts but left pondering the storage requirements. Would you all recommend a single big SD card or a smaller SD supplemented with a USB hard drive?
Re: MiSTer Storage Requirements
Posted: Sat Aug 07, 2021 2:27 am
by ZigZag
A big, fast MicroSD. I found HDD speed disappointing.
Re: MiSTer Storage Requirements
Posted: Sat Aug 07, 2021 2:40 am
by jlancaster86
If you have a NAS, you can setup a "MiSTer" SMB share and use that (that's what I do).
Re: MiSTer Storage Requirements
Posted: Sat Aug 07, 2021 6:50 am
by Bas
Get a big SD. If you're mainly going for Amiga, you'll have plenty of space on anything bigger than 64GB. Big storage mainly comes into play when you start loading CD-ROM images, which isn't really front and center on the Amiga platform. I started my MiSTer with an 8GB card and was actually fine for the most part except CD's. Now I'm on 64GB and have room to spare.
Re: MiSTer Storage Requirements
Posted: Sat Aug 07, 2021 10:12 am
by bazza_12
defo if you're only using amiga use get a big SDcard.. my amiga folder is 8gig ish.. that includes MegaAGS, Shapeshifter, kickstart roms and 21meg of demos. I agree with Bas above.. 64 gig+
Re: MiSTer Storage Requirements
Posted: Sat Aug 07, 2021 10:31 am
by Bas
Regarding eggs and baskets.. SD isn't exactly reliable storage, so make backup copies of anything valuable.
Re: MiSTer Storage Requirements
Posted: Sat Aug 07, 2021 12:53 pm
by kathleen
ZigZag wrote: ↑Sat Aug 07, 2021 2:27 am
A big, fast MicroSD. I found HDD speed disappointing.
Is using an external SSD HDD could help the speed concern ? Or is it due to the USB transfer ?
Re: MiSTer Storage Requirements
Posted: Sun Aug 08, 2021 4:06 am
by ZigZag
I think it's USB hub speed. Could just be my set up, but I find MicroSD faster than an external SSD. It's noticeable mostly with cores like PC & Amiga. This is just my personal experience though, and I haven't updated in a few months.
Re: MiSTer Storage Requirements
Posted: Mon Aug 09, 2021 10:55 am
by Solskogen
I've found that ethernet is faster than USB and SD card, so I use a small/cheap sdcard for the cores and data on SMB(NAS).
Re: MiSTer Storage Requirements
Posted: Mon Aug 09, 2021 2:25 pm
by RealLarry
Solskogen wrote: ↑Mon Aug 09, 2021 10:55 am
I've found that ethernet is faster than USB and SD card, so I use a small/cheap sdcard for the cores and data on SMB(NAS).
An ethernet connection
is faster. That's why I'm using an old and decommissioned mini PC for MiSTer's data, which is booting up a stripped down Debian for max. speed and greatest flexibility. I've tested 256GB SD cards and ready-to-run NAS' but wasn't ever satisfied. Better building such things by myself, also in terms of data privacy and trustworthiness.
Re: MiSTer Storage Requirements
Posted: Mon Aug 09, 2021 3:34 pm
by akeley
I'm fine with my SD card's speed. I mean, how much faster can the other methods be, when you consider the kind of data we're shifting?
Re: MiSTer Storage Requirements
Posted: Mon Aug 09, 2021 7:01 pm
by Bas
The LAN interface is 100Mbit right? That caps the network at around 12MB/s throughput, which is faster than just about anything you can expect from a retro core except maybe big box Amiga (which MiSTer doesn't have) or AO486. Any faster won't be useful until you reach the extremes like Saturn or Playstation, but probably not even those.
Re: MiSTer Storage Requirements
Posted: Mon Aug 09, 2021 9:21 pm
by elvis
Bas wrote: ↑Mon Aug 09, 2021 7:01 pm
The LAN interface is 100Mbit right?
The DE10nano's Ethernet port is gigabit, full duplex.
The USB port is USB2, and the issue that USB2 is half duplex. For storage, that's a larger impact than its 480Mbit/s bandwidth.
I already had a NAS configured for use centrally in my house across multiple gaming systems, modded consoles, arcade machines, TVs, etc. So that was a simple choice for MiSTer. The speed is certainly nicer than either SD cards or USB2 connected storage.
Additional features like transparent compression, deduplication, snapshots and redundancy courtesy of BtrFS are all nice too.
Re: MiSTer Storage Requirements
Posted: Tue Aug 10, 2021 12:50 am
by aberu
I agree with most people here, a big MicroSD is still pretty cheap. A 128GB if you are primarily just doing Amiga and some other stuff. If you get into AO486, then 256GB IMO. If you need more you just plop the extras on a usb stick and navigate to the extra files there.
Re: MiSTer Storage Requirements
Posted: Tue Aug 10, 2021 5:23 am
by Bas
Elvis, did you benchmark the speeds?
Re: MiSTer Storage Requirements
Posted: Tue Aug 10, 2021 7:41 am
by bazza_12
Bas wrote: ↑Tue Aug 10, 2021 5:23 am
Elvis, did you benchmark the speeds?
he's left the building...
sorry.. couldn't resist
Re: MiSTer Storage Requirements
Posted: Tue Aug 10, 2021 11:44 pm
by elvis
Bas wrote: ↑Tue Aug 10, 2021 5:23 am
Elvis, did you benchmark the speeds?
Using iperf3 for raw TCP performance in both directions:
MiSTer to NAS:
Code: Select all
/root# iperf3 -i1 -c 192.168.3.254
Connecting to host 192.168.3.254, port 5201
[ 5] local 192.168.3.47 port 57584 connected to 192.168.3.254 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.01 sec 75.0 MBytes 623 Mbits/sec 0 167 KBytes
[ 5] 1.01-2.01 sec 73.8 MBytes 619 Mbits/sec 0 177 KBytes
[ 5] 2.01-3.01 sec 75.0 MBytes 631 Mbits/sec 0 177 KBytes
[ 5] 3.01-4.01 sec 75.0 MBytes 630 Mbits/sec 0 177 KBytes
[ 5] 4.01-5.01 sec 75.0 MBytes 629 Mbits/sec 0 177 KBytes
[ 5] 5.01-6.01 sec 75.0 MBytes 630 Mbits/sec 0 177 KBytes
[ 5] 6.01-7.01 sec 73.8 MBytes 618 Mbits/sec 0 177 KBytes
[ 5] 7.01-8.00 sec 75.0 MBytes 631 Mbits/sec 0 177 KBytes
[ 5] 8.00-9.00 sec 75.0 MBytes 632 Mbits/sec 0 177 KBytes
[ 5] 9.00-10.02 sec 76.2 MBytes 630 Mbits/sec 0 177 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.02 sec 749 MBytes 627 Mbits/sec 0 sender
[ 5] 0.00-10.05 sec 749 MBytes 625 Mbits/sec receiver
iperf Done.
NAS to MiSTer
Code: Select all
/root# iperf3 -i1 -R -c 192.168.3.254
Connecting to host 192.168.3.254, port 5201
Reverse mode, remote host 192.168.3.254 is sending
[ 5] local 192.168.3.47 port 57588 connected to 192.168.3.254 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 85.6 MBytes 718 Mbits/sec
[ 5] 1.00-2.00 sec 85.1 MBytes 714 Mbits/sec
[ 5] 2.00-3.00 sec 87.1 MBytes 730 Mbits/sec
[ 5] 3.00-4.00 sec 86.1 MBytes 722 Mbits/sec
[ 5] 4.00-5.01 sec 87.0 MBytes 726 Mbits/sec
[ 5] 5.01-6.00 sec 87.9 MBytes 738 Mbits/sec
[ 5] 6.00-7.01 sec 84.9 MBytes 712 Mbits/sec
[ 5] 7.01-8.01 sec 87.4 MBytes 732 Mbits/sec
[ 5] 8.01-9.00 sec 86.7 MBytes 732 Mbits/sec
[ 5] 9.00-10.00 sec 86.4 MBytes 724 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 867 MBytes 725 Mbits/sec 222 sender
[ 5] 0.00-10.00 sec 864 MBytes 725 Mbits/sec receiver
iperf Done.
And a real-world test: Here's my CIFS mount with my ROMS. 235MB of files in this particular directory. I use the Linux command "cat" to stream them over the CIFS mount, pass them through a tool called "pv" (short for "pipe view") which just reports the transfer speeds, and then throws them away (passed to /dev/null). This is the same as reading files into memory on MiSTer.
Before any disk-read tests, I send "3" to a special file called /proc/sys/vm/drop_caches. This tells Linux's local file system cache to invalidate all entries and empty the cache, forcing a new read from the backing storage. Note that this doesn't clear the cache on the remote system (i.e: my NAS still has it's own cache, independent of clients connected).
Code: Select all
/media/fat/SNES/1 US - A-E# echo '3' > /proc/sys/vm/drop_caches ; cat * | pv >/dev/null
235MiB 0:00:05 [45.8MiB/s]
45.8MB/s, or around 400Mbit/s real world. My NAS's storage is much faster than that (can quite happily saturate a couple of 1GbE PCs in the house while feeding media to my TVs - storage was idle at time of benchmark), so the bottleneck there is definitely the DE10-Nano itself.
Let's compare reading off the MicroSD card I have. It's not particularly impressive or high speed, a SanDisk Ultra 64GB Micro SDXC UHS Speed Class 1, rated to 10MB/s. Using MiSTer's local Linux image:
Code: Select all
media/fat/linux# echo '3' > /proc/sys/vm/drop_caches ; cat linux.img | pv >/dev/null
312MiB 0:00:14 [22.3MiB/s]
Exceeds the minimum 10MB/s guarantee by a factor of 2, but still slower than my NAS. I don't have faster MicroSD cards to test, as again I can't be stuffed buying them when my NAS has more convenient features than just raw speed. So I'm not sure what the maximum speed through the MiSTer's MicroSD slot is.
Re: MiSTer Storage Requirements
Posted: Thu Aug 12, 2021 5:21 pm
by drpaneas
Easier thing is to use an SD Card.
But I second what @Elvis said -- the network connection is faster. I did lot's of benchmarks with iperf3 and game streaming (using Moonlight -- this is outside of the context of Mister though) and the speed was mindblowing, even with 5G wifi. If you can setup a network storage, go for it
Re: MiSTer Storage Requirements
Posted: Tue Sep 21, 2021 2:36 pm
by german_user
Only or mainly for the Amiga Core I would stick with an SD card. I am currently using inexpensive 64GB cards myself, these cost less than € 4 each and are relatively easy to copy. CD32 games isos are the real memory hogs, the same games as the A1200 version rarely use more than 200MB.
Re: MiSTer Storage Requirements
Posted: Fri Dec 31, 2021 3:19 am
by elvis
PorkChopExpress from MiSTeraddons has published a great breakdown on MiSTer storage. Confirming my previous findings that wired Ethernet is by far the fastest option for storage.
https://misteraddons.com/blogs/news/mis ... d-shootout
I've got a project in the works to assist people with configuring network storage for MiSTer. When that's ready for public consumption, I'll make a separate post.