rsync for AmigaOS 3.x
=====================
This is current rsync (3.5.1, protocol 33) for classic AmigaOS 3.1 and
later on a 68020 or better, built on ixemul.library. It talks to the
rsync on any current Linux, BSD or macOS machine, so you can back up your
Amiga to a server and restore it again, send only what has changed, and
run an rsync daemon on the Amiga itself.
Unlike older Amiga ports it keeps what makes Amiga files Amiga files:
file comments and protection bits (hsparwed) travel with -X and come back
on a restore, soft and hard links are kept with -l and -H, and file
dates are converted with your time zone, so they are right on the server
all year round.
It runs in FS-UAE and on real hardware (Vampire, 68080, Roadshow).
Contents of the archive:
- rsync: the program (68020, no FPU needed)
- rsync.readme: this text
- rbackup.rexx: ARexx script that backs up several drives, with retries
- rsync.1.txt: the rsync manual
- rsyncd.conf.5.txt: the manual of the daemon configuration
- COPYING: GNU General Public License version 3
Quick start
-----------
For backups and restores between the Amiga and another machine, use
-rtX. Backup:
rsync -rtX /Work/ rsync://server/module/Work/
Restore:
rsync -rtX rsync://server/module/Work/ /Work/
Add -lH if the disk contains soft or hard links (see "Links").
- -r recurses, -t keeps the dates (needed so the next run only sends
changes), -X keeps the file comments and protection bits.
- Without -X, restored files get their protection bits from the Unix
mode on the server, and programs usually lose their e bit: they no
longer start. Back up and restore with -X, both ways.
- Do NOT use -a or -p. They set the protection bits from the Unix mode
after -X has set them, so h, s, p, a and usually e are lost again.
- -X needs rsync 3.0 or newer with xattr support on the other side
(e.g. the rsync of Ubuntu or Debian). With an older server, leave -X
out and expect to fix protection bits by hand after a restore.
- Set your time zone once, otherwise the dates on the server are one or
two hours off (see "Dates and time zones"). For Central Europe:
SetEnv TZ CET-1CEST,M3.5.0,M10.5.0/3
Copy ENV:TZ ENVARC:
- Write paths Unix style: /Work/src/ for Work:src/, /RAM/ for RAM:.
"Work:src" does not work, rsync takes it for a remote host.
- Or simply edit the settings in rbackup.rexx and start it with
"rx rbackup.rexx".
Installation
------------
1. Copy rsync somewhere in your path, e.g. C:.
2. Install ixemul 80.1 (dev/lib/ixemul-80.1-m68k.lha). Copy a MATCHING
PAIR of libraries to LIBS:, renamed:
Sys/libs/ixemul020.library as LIBS:ixemul.library
Sys/libs/ixnet020.library as LIBS:ixnet.library
(or the 040/060 pair for those CPUs). Do not use ixnet000.library
with ixemul000.library: in 80.1 they are different build variants and
networking silently stays off ("Function not implemented"). rsync is
built without FPU code, so use the non-fpu ixemul.
3. A TCP/IP stack providing bsdsocket.library (Roadshow, AmiTCP, Miami,
or UAE's emulated one).
4. ixnet opens AmiTCP:libs/usergroup.library in every ixemul program. If
AmiTCP: does not exist you get a "Please insert volume AmiTCP"
requester, so assign it, e.g. "Assign AmiTCP: SYS:" (the library
itself is optional).
5. T: and ENV: must exist (the standard Startup-Sequence makes them).
rsync needs some room in T:, see "How it works".
6. Set TZ to your time zone, see "Dates and time zones".
On the server, an rsync daemon module for the backup, for example in
/etc/rsyncd.conf on Linux:
[amiga]
path = /srv/backup/amiga
read only = no
uid = youruser
gid = youruser
auth users = amiga
secrets file = /etc/rsyncd.secrets
with a line "amiga:yourpassword" in /etc/rsyncd.secrets (mode 600), and
the backup goes to rsync://amiga@server/amiga/. The filesystem there
must support extended attributes for -X (ext4, xfs and btrfs do).
The backup script rbackup.rexx
------------------------------
rbackup.rexx backs up several drives in one go and tries again when the
network fails. Edit the settings at the top, then start it from a Shell
with "rx rbackup.rexx" (RexxMast must be running):
server = 'myserver.example.org'
module = 'amiga'
user = 'amiga'
password = 'yourpassword'
backup.1 = 'Work:'
backup.2 = 'Data:Pictures'
utcoffset = 1
summertime = 'EU'
Each drive or drawer lands in a folder of the same name in the module:
Work: in amiga/Work/, Data:Pictures in amiga/Data/Pictures/. The server
may be given as name:port.
- Network errors, broken connections and timeouts (exit codes 10, 12,
30, 35) are retried, by default 5 times with growing pauses. A failed
login stops the script; other errors are reported and not retried.
- --partial --timeout=120 are in the default options, so a retry
continues a large file and a stalled connection becomes an error
instead of hanging.
- The time zone is two settings: utcoffset (1 for Germany, 0 for the
UK, 2 for Finland, -5 for New York) and summertime. The script turns
them into TZ.
- fixdates = 1 is for ONE run after you set the time zone for the first
time, when the server already holds a backup made without it: that
run only corrects the dates (rsync --size-only) instead of reading
every file again. Then set it back to 0. A file that changed since the
old backup but kept exactly its size is not sent in that run.
- The password goes to rsync through RSYNC_PASSWORD, which the script
removes again at the end. Anyone who can read the script can read the
password.
- Drives that are not there are reported instead of bringing up an
"insert volume" requester. A drawer needs --mkpath on the server side
(rsync 3.2.3 or newer); the script adds it when needed.
- Results go to the console, to the log file (with rsync's own log) and
to a summary window at the end.
File comments and protection bits (-X)
--------------------------------------
AmigaDOS keeps a comment (FileNote) and protection bits (hsparwed,
Protect) with every file and directory. Unix filesystems have neither,
so by default rsync transfers only contents and dates. With -X the
Amiga rsync presents them as two extended attributes, which a Linux
server stores in its filesystem. "getfattr -d file" on Linux shows them:
- the file comment as user.amiga.comment="Das List-Kommando"
- the protection bits as user.amiga.protection="-sp-rwed"
The protection value is written the way List shows it: the letters
hsparwed in that order, "-" for a flag that is not set (for rwed: not
allowed). The default ----rwed is not stored; a file without the
attribute gets ----rwed when it comes back. Comments are stored as raw
bytes (Latin-1), so umlauts look odd in a UTF-8 terminal but return
unchanged.
- Both sides need -X; the client passes it on to the server. A server
older than rsync 3.0, or one built without xattr support, refuses it.
- Files backed up without -X have no attributes on the server; restoring
them with -X gives them the default ----rwed.
- Changing only a comment or the bits is detected: rsync itemizes it as
".f........x" and sends just the attribute.
- Other user.* attributes coming from Linux are ignored on the Amiga.
-X against -p:
- Comment: -rtX keeps it, -rtp loses it.
- r, w, e, d: -rtX keeps them exactly. -rtp keeps r, w and e; w and d
can only be set together.
- h, s, p, a: -rtX keeps them, -rtp loses them.
- Mode on the server: with -rtX new files get 0400 to 0700, existing
files keep theirs. -rtp always sets 0400 to 0700.
- The server needs rsync 3.0 or newer with xattr support for -rtX, and
nothing special for -rtp.
What -p does (measured on FFS with all 16 combinations of r, w, e, d):
Amiga r, w (or d), e become the owner's read, write, execute on Linux;
group and other get nothing. Back on the Amiga only the owner bits
count: 0755 and 0700 give rwed, 0644 and 0600 give rw-d (not
executable!), 0555 gives r-e-, 0444 gives r---. Files without r cannot
be read by rsync at all (there is no "root"): they are skipped with
"Permission denied", exit code 23.
Dates and time zones
--------------------
AmigaDOS keeps file dates in local time; Unix systems keep them in UTC.
ixemul converts with one fixed offset that is normally not set, so
without help a file saved at 13:00 Central European Time shows up on a
Linux server as 14:00 (15:00 in summer).
When the variable TZ says which time zone the Amiga clock is in, rsync
converts the dates itself, with the rules that apply on each file's own
date:
SetEnv TZ CET-1CEST,M3.5.0,M10.5.0/3
Copy ENV:TZ ENVARC:
That is Central Europe; the second line keeps it after a reboot. For
the UK use GMT0BST,M3.5.0/1,M10.5.0, for US Eastern EST5EDT,M3.2.0,M11.1.0.
Set it on every Amiga that runs rsync, also for the daemon. The Amiga
clock has to show local time, including summer time.
With TZ set, the dates on the server are right all year round, they come
back unchanged on a restore, and nothing is sent again when the clock
goes to or from summer time. Without TZ rsync behaves like the earlier
versions: dates one or two hours ahead on the server, but stable.
Existing backups: after setting TZ, the dates of all files on the server
differ from the Amiga, and a normal run would read every file again.
Run once with --size-only instead (or fixdates = 1 in rbackup.rexx):
rsync -rtX --size-only /Work/ rsync://server/module/Work/
Limits: once a year the clock shows 02:00 to 03:00 twice. An Amiga file
from that hour is taken as the first 02:30 (summer time); a file from the
server dated in the second one arrives as 02:30 and gets its date set
again on every run. A date in the hour that does not exist in spring
comes back two hours later.
rsync reads TZ and RSYNC_PASSWORD through dos.library itself: ixemul
keeps a copy of ENV: and may not notice a later SetEnv until the next
reboot, so other ixemul programs can still see an old value.
Links (-l, -H)
--------------
- Without -l and -H (plain -rtX): soft links are skipped ("skipping
non-regular file"), hard links are copied as independent files.
- -l transfers soft links as links, in both directions. Targets are
converted between Amiga and Unix form (Work:x <-> /Work/x; relative
targets stay relative).
- -H keeps hard links to files, in both directions. A hard link to a
directory (Unix has none) arrives as a directory of its own whose files
are hard links to the originals.
For a full backup use "rsync -rtXlH". Soft links sent from the Amiga
carry no date (1970 on Linux). A Linux daemon with "use chroot = no"
prefixes link targets with /rsyncd-munged/; set "munge symlinks = no" in
a trusted setup. MakeLink of OS 3.1 creates hard links.
Tested options
--------------
Everything listed was run in the emulator and the result checked. An
option that is not listed may well work; it has just not been tested.
In brackets: where it was tested. "local" is a copy on the Amiga, "push"
the Amiga sending to a Linux daemon, "pull" the Amiga fetching from a
Linux daemon, "daemon" a Linux client using the rsync daemon on the
Amiga, "Amiga-Amiga" two Amigas.
Works (tested in every direction where it makes a difference):
- -r -t (local, push, pull, daemon, Amiga-Amiga). Set TZ, otherwise the
dates on the server are one or two hours off.
- Local copies (local).
- Daemon client (push, pull), against rsync 3.5.1, 3.2.7, 3.1.3, 3.0.9
and 2.6.9.
- Daemon on the Amiga (daemon, Amiga-Amiga). "max connections" is not
enforced.
- -X (local, push, pull, daemon).
- -H (local, push, pull).
- -i (local, push, pull, daemon).
- --delete (local, push, pull, daemon).
- -c, --checksum (local, push, pull).
- --password-file=/RAM/pw (push). For modules with "auth users", use
rsync://user@host/module/. A wrong password gives "auth failed", exit
code 5.
- --stats, -h, --log-file (local, push).
- --progress (local, push, pull) and --info=progress2 (pull).
- --list-only, and "rsync rsync://host/" to list the modules (push,
Amiga-Amiga).
Works, with a limitation:
- TZ, the time zone (push, pull, daemon). See "Dates and time zones" for
the limits. Not tested with local copies and between two Amigas.
- -p (push, pull). Only r, w, e (and d together with w), owner bits
only.
- -l (local, push, pull). Links sent from the Amiga have no date.
Partly tested (worked where it was tested):
- -R (local). Not tested with a daemon.
- --exclude and --delete-excluded (local, push, Amiga-Amiga). Not tested
when pulling.
- --include, --exclude-from, -m (push). Write patterns without quotes,
the Amiga shell does not expand "*". To put a "*" into a file with
Echo, double it: Echo "**.tmp".
- --files-from=/RAM/list (push).
- -z, compression (local, Amiga-Amiga). Not tested against Linux.
- -n, --dry-run (push).
- --max-size, --min-size (push).
- --size-only, -u, --ignore-existing (push).
- --delete-before, --delete-after (push).
- --backup --suffix=.bak (push). Backups on the Amiga side not tested.
- --remove-source-files (push).
- --inplace, -W, --partial (push). --partial not with an interrupted
transfer.
- --bwlimit (push).
- -x (push). Stopping at other volumes not tested.
- --timeout, --contimeout, --port (push). No timeout was actually
triggered.
"Partly tested" mostly means that the Amiga as the receiving side
(local copies, restores) has not been tried with that option yet.
Avoid, or not supported:
- -a: it contains -p. Use -rtX instead.
- Files of 2 GB or more: skipped with an error, exit code 23.
What does not work
------------------
- Files of 2 GB or more. AmigaOS 3.x and ixemul use 32-bit file sizes.
rsync skips such files with "files of 2 GB or more are not supported
on AmigaOS" and exits with code 23; everything else is transferred.
Files of 4 GB or more on PFS3/SFS may report a wrapped-around size and
cannot be detected; keep them out of rsync transfers.
- Transfers over a remote shell (host:path via ssh/rsh): there is no
usable ssh for AmigaOS 3.x to start. Use an rsync daemon.
- pre-xfer exec, post-xfer exec, early exec, name converter and
RSYNC_CONNECT_PROG: they need fork(). rsync reports an error.
- "max connections" in rsyncd.conf is not enforced.
- Ownership, uid/gid mapping, chroot: AmigaOS has no users.
- ACLs, Linux-style xattrs (other than the two above), iconv, IPv6.
Linux files on an Amiga keep their contents, directory tree and dates,
but lose owner, group, device files and Linux xattrs. Names differing
only in case overwrite each other, names longer than the filesystem
allows (30 characters on FFS) and names containing ":" fail, dates
before 1978 do not exist on the Amiga. To keep Linux files intact on an
Amiga, pack them first (tar) and transfer the archive.
Usage notes
-----------
- rsync's exit codes are often 10 or more (23 = some files not
transferred). AmigaDOS scripts stop at return codes from 10 on, so put
"FailAt 1000" before rsync in scripts that should continue.
- Replacing a file is not atomic on AmigaOS: Rename() never overwrites,
so rsync deletes the old file first. A crash right then leaves only
the temporary .name.XXXXXX file.
- Daemon on the Amiga: start it with
Run >NIL: rsync --daemon --no-detach --config=/S/rsyncd.conf
and use "use chroot = no" in rsyncd.conf. Paths in rsyncd.conf are Unix
style too (path = /Work/share).
Troubleshooting
---------------
- "Empty destination arg specified" for a correct command line: an old
ixemul.library is installed. "Version LIBS:ixemul.library" must say
80.1.
- "Function not implemented" on every network access: ixemul.library
and ixnet.library are not a matching pair (see Installation).
- "Please insert volume AmiTCP": assign AmiTCP:, see Installation.
- "Unknown module 'x'": the first part after rsync://host/ is the module
name from the server's rsyncd.conf. "rsync rsync://host/" lists them.
- "remote command not found" (code 127): an Amiga path like Work:src was
read as a remote host. Write /Work/src.
- To see what happens inside:
SetEnv RSYNC_AMIGA_TRACE RAM:rsync-trace.txt
makes every rsync process append to that file what the fork emulation
does, all of rsync's messages, and the address of a crash. Setting
RSYNC_AMIGA_KEEP as well keeps the record files in T:.
How it works
------------
ixemul has no fork(), only vfork() followed by exec. A local copy starts
the receiving side as a separate "rsync --server" process via vfork()
and exec, connected by pipes. The receiving side splits into a generator
and a receiver that need the same state (options, protocol, filter
rules, file list); rsync gets this by record and replay: until the split
the process records every byte it reads from a pipe or socket into T:,
then starts itself again, and the new process replays the recording with
its output suppressed and continues as the receiver on the inherited
connection. Each part is an ordinary AmigaDOS process; nothing shares
memory or DOS state. The recording is usually a few KB, but grows with
very large file lists.
History
-------
- amiga5: dates converted with the time zone (TZ), right on the server
all year round; rsync reads TZ and RSYNC_PASSWORD past ixemul's ENV:
cache; log time stamps fixed; rbackup.rexx. First Aminet release.
- amiga4: -H keeps hard links (ixemul reports a link count of 1); -X
with -l no longer stops at an "insert volume" requester.
- amiga3: files of 2 GB or more are refused with an error instead of
arriving empty.
- amiga2: file comments and protection bits with -X.
- amiga1: first release with local copies, daemon client and server.
Source and licence
------------------
rsync is free software under the GNU General Public License version 3
(see COPYING). The source of this port, with all Amiga changes, is at
https://github.com/OliverUrbann/amiga_rsync
in the branch amigaos.
The Amiga specific code is in amiga/ and in "#ifdef __amigaos__" blocks.
It is built with the m68k-amigaos cross toolchain from
https://github.com/AmigaPorts/amiga-gcc and amiga/build.sh.
rsync was written by Andrew Tridgell and Paul Mackerras and is
maintained by Wayne Davison; see https://rsync.samba.org/.
|
aminet net>