DOSContainer Alpha Release

[Bas]

I'm taking a small detour now. The current state of DOSContainer allows me to package just about anything from the dawn of time until about 1985. That's where I'll start, and I want the process be friendly to outside contribution. So in short:

1. I'm completing the zipfile downloader feature on DOSContainer to enable the layered build architecture that I envisioned from the start.
2. Set up robust and fast hosting for everything. (done)
3. Create a friendly webform for contributed games and YAML's.
4. Publish tested and validated games on a new DOSContainer website.
5. Put DOSContainer itself behind a web form. Upload your YAML, it spits out a VHD just for you. (stretch goal)

Once that flow is up, I'm starting in 1981 and working my way up through history. Implementing DOS versions and FAT variations follows as I reach the games for which they are period appropriate. Starting the packaging early helps me learn and figure out optimal workflow.

Another feature that I feel is a hard requirement when games become configurable themselves (1985-ish), is a "pull" option. This means you run a VHD on MiSTer and have a shell open at the same time. As you make the needed changes in the DOS environment to set up all the different hardware settings, you run a simple "pull" command on the VHD every time to extract a delta that captures only the config changes you made in DOS (yes, I think I can detect open/uncommitted files with reasonable confidence). Those changes then become a layer in the YAML. Once done, the tool wraps the whole lot up in a zipfile and a YAML manifest. The goal: instantly allow custom builds that mix and match all settings for the game itself where it makes sense. Just tweak the YAML to suit your wishes, or just build VHD's for every permutation you can think of in the blink of an eye. This should fix the huge amount of manual toil to configure each individual game.

By the way: this is all a hobby and I only have a few hours each week for it. I do welcome any help I can get in this! If you know Rust, awesome! If you're good with AWS serverless code: please raise your hand. If you want to design a website, logo or anything else: yay! I am very partial to static hosting with things like Hugo due to the low maintenance and no security/maintenance headaches like WordPress would bring.

Another option that I'm very open to, is to join with the 0Mhz effort and provide the plumbing for that. My own scope is wider than MiSTer itself.

[AmintaMister]

I'm taking a small detour now. The current state of DOSContainer allows me to package just about anything from the dawn of time until about 1985. That's where I'll start, and I want the process be friendly to outside contribution. So in short:

1. I'm completing the zipfile downloader feature on DOSContainer to enable the layered build architecture that I envisioned from the start.
2. Set up robust and fast hosting for everything. (done)
3. Create a friendly webform for contributed games and YAML's.
4. Publish tested and validated games on a new DOSContainer website.
5. Put DOSContainer itself behind a web form. Upload your YAML, it spits out a VHD just for you. (stretch goal)

Once that flow is up, I'm starting in 1981 and working my way up through history. Implementing DOS versions and FAT variations follows as I reach the games for which they are period appropriate. Starting the packaging early helps me learn and figure out optimal workflow.

Another feature that I feel is a hard requirement when games become configurable themselves (1985-ish), is a "pull" option. This means you run a VHD on MiSTer and have a shell open at the same time. As you make the needed changes in the DOS environment to set up all the different hardware settings, you run a simple "pull" command on the VHD every time to extract a delta that captures only the config changes you made in DOS (yes, I think I can detect open/uncommitted files with reasonable confidence). Those changes then become a layer in the YAML. Once done, the tool wraps the whole lot up in a zipfile and a YAML manifest. The goal: instantly allow custom builds that mix and match all settings for the game itself where it makes sense. Just tweak the YAML to suit your wishes, or just build VHD's for every permutation you can think of in the blink of an eye. This should fix the huge amount of manual toil to configure each individual game.

By the way: this is all a hobby and I only have a few hours each week for it. I do welcome any help I can get in this! If you know Rust, awesome! If you're good with AWS serverless code: please raise your hand. If you want to design a website, logo or anything else: yay! I am very partial to static hosting with things like Hugo due to the low maintenance and no security/maintenance headaches like WordPress would bring.

Another option that I'm very open to, is to join with the 0Mhz effort and provide the plumbing for that. My own scope is wider than MiSTer itself.

This is an extraordinary effort, a perfect strategy, and a fundamental openess to collaboration, THANKS!

[Bas]

Build 64 actually builds a working (in PCem, MiSTer untested) bootable Alley Cat VHD based on IBM PC DOS 2.00.

This build is a partly hardcoded proof of concept that should prove a number of items. I would VERY MUCH appreciate some testing of the below functionality:

• Does this even run properly on Windows at all? I don't know, I don't have Windows.
• Does the ARMv7 build run properly on the MiSTer? (stretch goal: 32-bit Raspberry Pi's should work in theory.. test please?)
• This is the first build with the ZIP downloader/extraction code hooked into the build phase.
• The downloader pulls files in over HTTPS from my under heavy construction AWS-based hosting location.

So please, anyone adventurous, have a look at build 64 and see how it goes. Use the YAML I posted above as an input file. The result should be alleycat.vhd which should work as advertised.

WARNING: This build contains a command to delete a single file. What it does is it downloads the zip file for Alley Cat, extracts that to a platform-appropriate temp location and then deletes the zip file. While this should all be very safe (I'm using standard-fare Rust crates for the handling of tempfiles/dirs, my own tests on Linux turned out fine) I can not guarantee anything here. Do NOT test this build on the master control station of your in-house nuclear reactor please, or on any other system you are particularly attached to. Chances of anything going badly wrong are practically non-existent, but I do feel I should warn you about being a guinea pig.

To satisfy some people's curiosity. The binaries for IBM DOS are currently still compiled as raw bytes into the executable itself. It was done this way because I didn't have a working downloader yet. When this build proves stable, my next step will be to remove these embedded files and make them into downloads just like the game files. When that's done you'll start seeing multiple progress bars as a VHD-build proceeds.

I do like how going from zero to working bootable VHD including this tiny game's download is done in 0.16 seconds on my laptop. This opens the possiblity of stuffing DOSContainer into an AWS Lambda function eventually.

[rcblanke]

Hi Bas,
Appears to be working fine on Win11:

win11 output

doscontainer.png (108.95 KiB) Viewed 709 times

though as you can see leaving out the .yml extension results in a funky message. Please note that I haven't tried to actually run the vhd on PCem or MiSTer.

output alleycat.vhd:
MD5 9cb6b2bb9d315726a4b6ab7de5a401a1
SHA 0fffca953b3243063b4059c0cd09a35ad4ba173f

Regards,
Ronald

[NML32]

Confirmed working on Windows 10
Used doscontainer.exe windows_x86_64

[Bas]

Thanks for your testing! The error messages are funky because you're looking at Rust passing the error down the chain and giving up because I didn't yet write a nice formatter for it yet. That's my bad, but mostly cosmetic. You giving me checksums makes a huge amount of sense, but I just realized they aren't meaningful yet. Files are dropped into the VHD with the current system date/time stamped on them so every VHD will in fact be slightly different. I hadn't thought of that yet (just needed some kind of date/time to work and even that is wrong right now). I'll pick that up as an issue and use the source files' timestamps so builds become reproducible and actually correct.

[NML32]

I get this error when running from MiSTer

root@MiSTer:/media/fat/doscontainer # ./doscontainer build allycat.yaml
./doscontainer: /lib/libc.so.6: version GLIBC_2.32' not found (required by ./doscontainer)
./doscontainer: /lib/libc.so.6: versionGLIBC_2.33' not found (required by ./doscontainer)
./doscontainer: /lib/libc.so.6: version `GLIBC_2.34' not found (required by ./doscontainer)

[NML32]

Success with build #68 creating allycat.vhd from MiSTer FPGA.

https://youtu.be/4ghmIWLhk_M

[Bas]

Wow, that was fast! I changed the build target over from glibc to musl, that helps apparently. Impressed with the speed of this thing on dinky MiSTer ARM!!

Last build for today will probably be 72 if it succeeds in about 15 minutes or so, after that it's back to work for a whole week again so not sure how much time I'll have to keep going. The biggest hurdle now is to support subdirectory's on the VHD. I did clean up and expand the documentation, added a section on the updated YAML format. I also do fully intend to support the Bash version's YAML format as well, but things may break there for a while. I will port over any games from the old version to the new though.

Change log lives here: https://code.area536.com/DOSContainers/ ... ranch/main

[Bas]

Just a quick update on progress. Lots of regression as I'm generalizing the method of writing files/dirs to the file system. So while it seemed that build 64 was really close, I had to rip out ever growing parts of the file system code to allow for the general pattern of: extract this ZIP file to a temp location, then copy everything from that location over to the VHD recursively.

Why is this difficult? Mainly because ancient DOS versions have divergent ideas on how to write the FAT's root directory versus regular subdirectories. Flexibility in the core code is required but that runs the risk of creating a ball of spaghetti.

Writing a file on FAT entails allocating clusters for it, checking and creating the correct directory entry for it, and then writing the bytes to the previously allocated clusters in the data area of the partition/volume. What I'm seeing now is slight fluctuations on where the data area is supposed to actually begin.

Now I remember the rumors back in the day of how Commodore and Olivetti supposedly weren't exactly IBM compatible but I never got more than amecdotal hearsay on those. As this project progresses I get to see a lot of the icky underbelly of how MS-DOS was marketed before MS released its own builds directly. Great stuff for a blog series, but very annoying right now.

[limi]

[Bas]

Interesting. This looks a lot more complete than the previous code, which was a steaming mess of versions all mixed together.

[RealLarry]

If anyone is interested: MS published their MS-DOS v4.0 on GitHub.