Short: Fast file/dir tree copier (EARLY BETA) Author: Pancho Manera / PAL-N Studios Uploader: josemanera gmail com (Pancho Manera, PAL-N Studios) Type: util/cli Version: 1.3 Requires: AmigaOS 2.04 (Kickstart 37) or newer Architecture: m68k-amigaos >= 2.0.4 Distribution: Aminet MegaCopy ======== A fast copier for files and whole directory trees on AmigaOS. It preserves protection bits, comments and datestamps, understands the standard AmigaDOS copy keywords, and is built to be quick on the case that hurts most: trees made of thousands of small files. *** READ THIS FIRST: THIS IS AN EARLY BETA *** =============================================== MegaCopy is new code. It has been reviewed hard and tested, but it has had very little time on real hardware. It is a tool that WRITES FILES. A bug in it can destroy data. YOU USE THIS AT YOUR OWN RISK. THERE IS NO WARRANTY OF ANY KIND. The author is not responsible for lost files, corrupted volumes, broken installations or wasted time. If your data matters to you, you must have a backup BEFORE you point this at anything. That is your job, not the program's. Nobody is going to be able to give your files back to you. If you are not willing to accept that, do not use it. That is a completely reasonable decision and no one will think less of you for it. - Do not point it at anything you do not have a backup of. - Do not put it in a boot script yet. - Test it on something you can afford to lose first. Bug reports are very welcome. WHAT'S NEW IN 1.3 ================= - **Critical fix: 1.2 would not start at all.** If you have 1.2, replace it with this one. What happened: a startup check meant to catch a truncated argument template compared against the wrong bound and fired on every run, so every invocation died with "internal argument template overflow" and copied nothing. It could not have worked as written either -- the value it tested cannot exceed the limit it was tested against, so the condition was never going to detect anything. The check now verifies that the template's tail is intact, which is what a truncation would actually destroy. We caught it within the hour and this release went out straight away. 1.2 was up very briefly; apologies to anyone who downloaded it in that window. The lesson taken: a safety check that has never been executed is not a safety check, and it does not ship again without being run first. WHAT'S NEW IN 1.2 ================= - Built-in help. `MegaCopy HELP` (or `?`) prints every keyword with a one-line explanation and a handful of worked examples, so you do not have to come back to this file to remember what ARC does. - Fix: the argument template sat at exactly the 255-character ceiling of a short string. One more keyword would have truncated it silently, leaving the arguments looking valid while the parser read garbage -- the worst possible failure mode in a program that writes files. There is now a startup check that refuses to run instead. - Cleanups in the bundled test scripts and in the source comments. THE SAFE WAY TO TRY IT ====================== Just drop the `MegaCopy` binary into your `C:` drawer and call it by its own name: MegaCopy DH0:Work DH1:Backup ALL CLONE Nothing on your system is replaced or renamed, and there is nothing to undo. If MegaCopy turns out to have a bug, it can only affect the commands where you typed `MegaCopy` yourself. **Start here.** Use it by hand for a while, on trees you have backed up. There is an optional installer (see below) but you do not need it, and it is not the thing to start with. USAGE ===== MegaCopy [...] TO [options] Type `MegaCopy HELP` for the full list. In short: ALL copy subdirectories too QUIET (Q) no per-file output BUFFER (BUF) buffer size, n x 512 bytes (0 = as big as possible) CLONE keep protection bits, comment and datestamp DATES keep the datestamp only COMMENT (COM) keep the comment only NOPROTECTION do not copy protection bits (NOPRO) NOREQUESTERS no disk requesters (NOREQ) NOREPLACE skip files already present in the destination (NOREP) NEWER copy only if the source is newer INTERACTIVE ask before replacing (INTER) FORCE overwrite protected destinations ARCHIVE (ARC) only files with the archive bit clear, then set it COPYLINKS follow links instead of skipping them STATS print the summary even with QUIET PROGRESS progress display even with QUIET TURBO raise task priority while copying CACHE ask both volumes for more filesystem buffers Wildcards (`#?`, `*`, `%`, `|`, `~`) work in FROM, in any path component, not just the last one. With no FROM at all it copies the current directory. Several directories as FROM get their contents merged into TO. Examples: MegaCopy DH0:Work DH1:Backup ALL CLONE MegaCopy #?.info RAM:T MegaCopy DH0:Work DH1:Backup ALL CLONE NOREP (re-sync) MegaCopy DH0:Work DF0: ALL CLONE BUF=64 (low memory) WHY IT IS FAST ============== Not magic, and not a bigger buffer. The reason is boring and countable: **it sends fewer DOS packets per file.** Every operation on an AmigaDOS file is a message to a filesystem handler process: a round trip with a task switch at each end. On a tree of small files that traffic, not the data, is what takes the time. Per file, with CLONE, MegaCopy gets down to four packets: open the destination, write, close, stamp the date. It gets there by skipping the metadata calls it can prove would change nothing: - a file created with MODE_NEWFILE is born with protection 0, so if the source protection (masked of the archive bit) is also 0 -- which is the case for the vast majority of files -- then setting it would write the value that is already there; - a newly created file has no comment, so if the source has none either, setting the comment is a packet that changes nothing. Both skips only apply when the destination is genuinely new. If the file already existed, the old bits and the old comment are still there and must be overwritten, so both calls are made. Directory scanning uses `ExAll` with `ED_COMMENT`, which returns dozens of entries per packet complete with type, size, protection, date and comment. In a 500-file directory that is roughly 10 packets instead of about 1000. Measured under emulation, with the destination on a separate volume: 334,649 files · 17,577 dirs · 8.5 GB 1948 s · 4162 KB/s 5,758 files · 414 dirs · 229 MB 33 s · 6852 KB/s 807 files · 77 dirs · 51 MB 4.88 s · 10700 KB/s re-sync of an unchanged 229 MB tree (NOREP) 9 s Emulator numbers are good for comparing one build against another, not as absolutes: the host's own disk cache is in charge and no program inside the Amiga can flush it. WHERE IT COULD BE FASTER, BUT IS NOT PROVEN =========================================== Everything above is measured. This section is theory, written down precisely so nobody mistakes it for a result. **Overlapped I/O.** MegaCopy keeps two DOS packets in flight: while the destination writes block N, the source is already reading N+1. Between two different devices this should approach 2x on the data. It has NOT been demonstrated. Under emulation the host's cache removes exactly the latency the overlap is supposed to hide. And on an A1200 with IDE the transfer is PIO -- reading and writing consume the *same* CPU -- so there is no physical overlap of data there at all; the honest expectation is 1.1-1.3x. Real DMA is where this should show. **Lockstep re-sync.** When re-syncing, the source directory and the destination directory are served by two *different* handler processes, so their two `ExAll` scans can be in flight simultaneously instead of one after the other. Implemented; predicted around 2x on re-sync; not yet measured on hardware where it could matter. **MaxTransfer and Mask.** A filesystem will only DMA straight out of your buffer if the buffer respects the device's `de_Mask` and `de_MaxTransfer`. If it does not, it quietly bounce-copies through its own buffer and you pay for every byte twice. MegaCopy does not read those values yet, and it cannot be tested under a directory-mount emulator, which has no FileSysStartupMsg at all. **Where the real floor is.** On the destination side, an FFS writing a small file does several block writes plus rewrites of the same header, because closing re-stamps the date and the metadata calls are separate packets with no single action that merges them. On a period hard disk that is 50-70 ms per file. At that point the limit is the destination filesystem, not the copier. Getting past it means changing the approach (raw writes to a verified-empty volume) or changing the filesystem (PFS3 handles metadata atomically and in batches). Neither is in here. Two things that sound like they should help and were measured as doing **nothing**: raising task priority (TURBO) and asking the filesystem for more cache buffers (CACHE). Both are still available as switches because on real hardware they may behave differently, but do not expect anything from them. MEMORY AND CPU ============== Built for a plain 68000, so it runs on an A500 with a 2.04 ROM as well as on an 060. Verified by disassembling the shipped binaries: 11,765 instructions in MegaCopy and not one of them is a 68010, 68020+ or FPU instruction. There is no hand-written assembly in the sources. Footprint, from the binary's own hunk header: code + data + BSS at load 48.2 KB RTL stack 32 KB pipeline buffers 2 x 256 KB by default, halving until they fit, floor 2 x 8 KB directory scan buffer 16 KB entry list per directory 13.6 KB initially, doubling So the minimum working set is around 126 KB, which is where it ends up if memory is tight -- the buffer allocation halves (256 -> 128 -> 64 -> 32 -> 16 -> 8 KB each) until the system says yes. It runs on chip-RAM-only machines. On a 1 MB machine it is worth passing `BUF=64` (32 KB buffers) rather than letting it take 512 KB while it works. The RTL is built with the OS heap in small puddles so it does not demand one large contiguous block. THE OPTIONAL INSTALLER ====================== `InstallMegaCopy` has an icon: double-click it, or run it from a shell. It can install MegaCopy as your `C:Copy`, so that scripts and programs that shell out to Copy start using it without knowing. Directory Opus configured to copy via the shell, Installer scripts, and your own boot and backup scripts are the usual beneficiaries. **You do not need this step, and it is the only one with consequences.** A bug now reaches everything, including your Startup-Sequence. **IT IS FULLY REVERSIBLE.** Installing saves your existing command to `C:Copy_original` first; option 2 in the installer puts it back byte for byte, with its protection bits, date and comment intact. An existing backup is never overwritten, so installing twice cannot bury the original. And if the installer itself will not run, one shell command undoes everything: Rename C:Copy_original C:Copy Nothing is hidden, encrypted or scattered around the system: exactly one file is moved and one is added, and you can undo both by hand. Safety rules it follows: - Nothing is written over the target until the replacement has been written to a temporary file, closed with the close *checked*, and verified by size. A close that fails to flush is exactly the case "everything wrote fine and the file is wrong". - The swap is rename-rename-delete, never delete-then-rename. There is no instant when `C:Copy` does not exist, and if anything fails the original goes straight back. - Protection bits, date and comment are preserved. - A pre-existing backup is validated as a real executable before being trusted. If your startup scripts use `Resident`, it warns you. MegaCopy is NOT reentrant and must not be made resident. **Write this down before installing.** If you ever boot the same disk under Kickstart 1.3, neither MegaCopy nor the installer will run -- both need 2.04 -- and `C:Copy` will be dead. The `Rename` above is the way out. The installer prints it on screen before it asks you anything. THE BENCHMARKING TOOLS ====================== Included because the speed claims should be something you can check yourself, not something you have to believe. **MegaBench** is a program, not a script, and that is deliberate: a shell script can only time with `Date`, which has one-second resolution. On a six-second run that is +-17% of error -- the measurement would be pure noise. MegaBench uses `DateStamp`, so 1/50 s. MegaBench [PASSES ] [KEEP] - Flushes both volumes between runs, so the second program to run does not find the first one's metadata still in the filesystem buffers and win for the wrong reason. - Alternates the order and discards the first pass as warm-up. - Deletes and recreates the destination before every run. - Checks the return code of every run: a copy that failed also "finishes quickly", and that number is worthless. Note that `PASSES 3` means eight full copies of the tree. For a large tree use `PASSES 1`. Also included, as AmigaDOS scripts: **Bench** (a timed comparison, logged to a file so you need not photograph the screen), **Patrones** (eight wildcard cases), **Verificar** (nine behaviour cases, each printing OK or MAL) and **Probar** (a quick smoke test with edge cases: zero-byte file, empty directory, a file larger than the buffer, four levels deep). CREDITS ======= Pancho Manera / PAL-N Studios. Written with the Macumba Editor, the same environment used for PocketLauncher and the rest of the PAL-N Studios tools. Built with FPC for m68k, targeting plain 68000 with the small-puddle OS heap so it runs on chip-RAM-only machines.