MiSTer FPGA Forum

Like my work?
Support me on https://ko-fi.com/natrox!

[Changelog]
2022-11-04 - v0.2.0 - Added functionality to exclude folders from the migration, users can pick which ones to skip from an optional menu.
2022-10-30 - v0.1.0 - Initial release.

Original post:

Hello everyone!
I am happy to announce that I have released a new utility for MiSTer; the SD Card Migration Utility!

You can use this utility to migrate to another SD card. You can even migrate to a smaller card if you wish - as long as the space used on your source SD card does not exceed the capacity of the new SD card.

No more will you need to use MrFusion to set up a new SD card and copy files over manually! This utility runs from your MiSTer and does everything for you. After it finishes, the SD card is ready for use right away.

Limitation: It won't work with SD cards that are 2TB or larger - but good luck finding one of those!!

Description: (copied from my repository):
This utility can be used on a running MiSTer system to migrate to a new SD card.
A USB SD card adapter is required to be connected to your MiSTer.
The target SD card can be of any size - as long as it can fit all of the data on the source SD card.
It performs a number of checks to make sure that the migration will be successful.

A key feature of this utility is that it looks at the source SD card's disk geometry and reproduces the required geometry on the new SD card.
This means that the U-Boot partition located at the tail-end of the disk is copied over, while the rest of the card is formatted to exFAT for data use.
All of the original data is copied over by default - version 0.2.0 introduces folder filtering, giving you the option to skip folders if you so desire. This utility is designed to be run from the MiSTer main menu, it will guide you through disk selection and it performs the necessary checks. During the process, it will freeze the main MiSTer binary to ensure file integrity.
It can also be used to make a backup SD - although it is not suitable for incremental backups.

Download:
For wget: https://raw.githubusercontent.com/Natro ... rate_sd.sh
GitHub viewer: https://github.com/Natrox/MiSTer_Utils_ ... rate_sd.sh
Repository: https://github.com/Natrox/MiSTer_Utils_Natrox

Installation:
Download "migrate_sd.sh" in whatever way you please, and put it in the "/media/fat/Scripts" folder on your MiSTer!
You can then run it from the main menu like any other script.

I hope you will find this script useful. Please let me know if you have any questions, problems or nice words.

Neocaron wrote: ↑Sun Oct 30, 2022 1:38 pm So it works absolutely great, I think it does an amazing job at creating additional sd cards and It's easier than mr fusion and faster, because everything is setup from the og card directly from the Mister with all my custom settings and content already! It's gonna be my new go to on that front.

However since the usb 2 is very slow, we should have an option to copy everything or having the choice to pick game categories. Because each PSX games/ Saturn/MSU/Ao486 are taking minutes to copy. The script is awesome but very much limited by the usb2 specification of the board.

Still it's possible if you don't have a huge cd library, or you are in no rush (I'm talking hours) to just start the process and let it be until it's done.

toastboy wrote: ↑Mon Oct 31, 2022 8:35 pm Migration finished, SDcard swapped and works perfectly.

Longtime4321 wrote: ↑Tue Nov 01, 2022 5:44 am Mine won't detect my SD card. I'm using a sandisk Micro sd to usb converter, so I can't imagine it would be the converters fault.
Would it be possible to add support for the sd card slot on the I/O Board?
Edit: Does the SD card have to be formated as fat32 or anything? Mine is currently a corrupt MiSTer install
Edit2: Formating it as FAT32 did the trick.

kubbie wrote: ↑Mon Oct 31, 2022 10:27 pm How long did it take to migrate 220GB? I'm thinking of migrating from 128GB to 256GB.

ntt wrote: ↑Thu Nov 10, 2022 8:53 pm HI Natrox
I've just copied the mister SD card with your script: everything went well, I could even skip the contents of the games folder, I couldn't be happier!
Just one question - or perhaps a proposal for next version: would it be possible to flag the entire games folder as not to be copied, instead of having to go through all of its subfolders one by one? Curiously enough I could toggle off any other folders but apparently not the entire games folder itself (or possibly I've missed it among the other 120+ subfolders being listed )
Again, thank you for this wonderful piece of software!

Hi Natrox,

what a wonderful script, great idea too, nothing else does THIS job.
I love nothing more than a functional, well written bash script

I have a question, I would like to simply use this script, to "prepare" an SD card, then I can copy everything over myself on a device other than the mister itself.

To do this, if there were a 'de-select all' option when the listing of directories (step 3.5) to choose to migrate comes up, I could choose that - or an option to simply skip copying the 'other' folders. Then only the critical folders would be copied and the sdcard 'prepared'.

Once the process is finished (paritioning, u-boot, copying linux folder etc) then I can immediately check if the SD card works on a mister. I can then copy all the data myself on another linux/windows box, which will make copying the data much faster (if you use good quality sdcard and good quality sdcard read/write devices). In Windows I might use fastcopy or robocopy and in Linux rsync.

Fantastic tool, thanks so much for all your work on this

edit: I also noticed that the exfat partition has no label, this could possible be a feature added to the script as well.

Hi Natrox,

I tried running the script however I get the following error messages:

line 0: syntax error near unexpected token 'newline'
line 0: '<!DOCTYPE HTML>'

I'm a complete newb when it comes to programming so any help you can provide would be greatly appreciated. Thanks!

EDIT: I GOT IT WORKING!! This script is a godsend. THANK YOU FOR ALL OF YOUR HARD WORK!!!!

Who were you, DenverCoder9? What did you see?!

I love me a good xkcd reference..

evilsim wrote: ↑Wed Dec 21, 2022 4:06 am

I have a question, I would like to simply use this script, to "prepare" an SD card, then I can copy everything over myself on a device other than the mister itself.

Hm, to be honest, it would probably be easier to just run Mr. Fusion again. If you need to prepare multiple MiSTers, you could image the SD after using Mr. Fusion and do it like that. Right now, you can still deselect everything by hand and it should work. You can also interrupt the file copy operation and then mount the SD manually to copy over what you need.

unclefunkle999 wrote: ↑Sat Dec 24, 2022 3:15 am

Hi Natrox,

I tried running the script however I get the following error messages:

line 0: syntax error near unexpected token 'newline'
line 0: '<!DOCTYPE HTML>'

I'm a complete newb when it comes to programming so any help you can provide would be greatly appreciated. Thanks!

EDIT: I GOT IT WORKING!! This script is a godsend. THANK YOU FOR ALL OF YOUR HARD WORK!!!!

Glad to see you got it working, and you're welcome.

For others running into this problem: you have to use "View Raw" on GitHub and download that file. Otherwise, you will download the HTML file instead.

Hi Natrox.

I've a 128G SD card in my Mister and want to get a smaller SD card, like 32GB, cuz I moved all the games (/Games) folder to a USBKey (/media/USB0)

If I run your script, will It only look at SD card at (/media/fat) and copy it to a new SD card, or will it scan for other attachments like my USBKEY??

Perhaps its a good idea to pull out my USBKEY before running your script???

Regards
TuffyDK

TuffyDK wrote: ↑Mon Feb 06, 2023 9:36 am

Hi Natrox.

I've a 128G SD card in my Mister and want to get a smaller SD card, like 32GB, cuz I moved all the games (/Games) folder to a USBKey (/media/USB0)

If I run your script, will It only look at SD card at (/media/fat) and copy it to a new SD card, or will it scan for other attachments like my USBKEY??

Perhaps its a good idea to pull out my USBKEY before running your script???

Regards
TuffyDK

Hey there Tuffy,

Technically it should only look at the /media/fat folder, but as far as I understand there is some functionality to overlap folders from USB storage onto the SD card root.

For safety, I recommend disconnecting your USB key before running the script.

Let me know if you run into any issues!

Natrox wrote: ↑Mon Feb 06, 2023 10:42 am

TuffyDK wrote: ↑Mon Feb 06, 2023 9:36 am

Hi Natrox.

I've a 128G SD card in my Mister and want to get a smaller SD card, like 32GB, cuz I moved all the games (/Games) folder to a USBKey (/media/USB0)

If I run your script, will It only look at SD card at (/media/fat) and copy it to a new SD card, or will it scan for other attachments like my USBKEY??

Perhaps its a good idea to pull out my USBKEY before running your script???

Regards
TuffyDK

Hey there Tuffy,

Technically it should only look at the /media/fat folder, but as far as I understand there is some functionality to overlap folders from USB storage onto the SD card root.

For safety, I recommend disconnecting your USB key before running the script.

Let me know if you run into any issues!

I'll get back when I have tried it - Not today, but sometime in the future - I think I will make a backup with Clonezilla first - Just to be sure

/Tuffy

Thank you Natrox for a fantastic script. It was painless swapping over SD cards.

It's a real shame that the Linux side of MiSTer can't access the second SD card slot, that's a huge waste of potential. I suppose it is what it is but it's still unfortunate. Regardless, excellent script.
Has the issue been fixed where it won't detect the SD card if the SD card isn't Fat32?