Hi All

i ask myself why there are no modern transfer protocols out there.
The X/Y/Z-Modem stuff was made long before TCP and IP where used by the
masses. But today there is no need for these old protocols. Modern terminal-programs work over tcp/ip and in most cases the filetransfer goes wrong. I tried to integrate a version of zmodem, writen in java, an it's a mess. There is so many error correction stuff in the protocol that i decided
to create a new protocol for my bbs ;-)

TCP/IP has all the beautiful corrections and sorting mechanisms that the filetransfer needed. In my case the protocol is written in to linux-shell-scripts. It doesn't ask for filenames, handles batch-downloads and is very simple.

At the moment i call it "XTP 1.0" (Xionum Transfer Protocol Version 1.0)

For the Download, the Communication comes only from the BBS:
BBS -> CLIENT: $02 (Start of Text)
BBS -> CLIENT: FILENAME + $0D
BBS -> CLIENT: FILELENGTH + $0D
BBS -> CLIENT: DATA (Base64 without Linebreaks) + $0D
BBS -> CLIENT: $04 (End of Transmission)

The TCP/IP-Connection handles all Errors ;-)

For the Upload:
BBS -> CLIENT: $05 (Enquiry)
BBS <- CLIENT: FILENAME + $0D
BBS <- CLIENT: DATA (Base64 without Linebreaks) + $0D
BBS -> CLIENT: $04 (End of Transmission)

The download can handle single/multi files and exported messages.

*** START OF SHELL - download.sh ***
#!/bin/sh
# this script is for downloading files from the bbs
# in mystic set the "Send Command" to ./download %7 %3

NODE=$1
FILENAME=$2
if [ -f temp$NODE/file.lst ]
then
for FILE in $(cat temp$NODE/file.lst)
do
FILENA=$(basename $FILE)
DATA=$(base64 -w 0 "$FILE")
LENGTH=$(echo $DATA | wc -m)

echo '$02' <- here you have to send the real byte
echo $FILENA
echo $LENGTH
echo $DATA
echo '$04' <- here you have to send the real byte
echo "z 0 19200 bps 9600 cps 0 errors 0 1024 $FILENA -1" >> $DSZLOG
done
else
DATA=$(base64 -w 0 "$2")
LENGTH=$(echo $DATA | wc -m)
FILENA=$(basename $FILENAME)

echo '$02' <- here you have to send the real byte
echo $FILENA
echo $LENGTH
echo $DATA
echo '$04' <- here you have to send the real byte
echo "z 0 19200 bps 9600 cps 0 errors 0 1024 $FILENA -1" >> $DSZLOG
fi
*** END OF SHELL ***

The upload can onyl handle single files!

*** START OF SHELL - upload.sh ***
#!/bin/sh
# this script is for uploading a file to the bbs
# in mystic set the "Recv Command" to /absolute/path/of/upload
# example: /home/bbs/mystic/upload

echo ''
stty -echo
read FILENAME
base64 -d -i > $FILENAME
stty echo
echo ''
sync
SIZE=$(wc -c <"$FILENAME")
if [ $SIZE -eq 0 ]
then
echo "ERROR"
else
echo "Z $SIZE 19200 bps 9600 cps 0 errors 0 1024 $FILENAME -1" >> $DSZLOG fi
*** END OF SHELL ***

With this simple (but surely not the best solution) i can up- and download files to my bbs without any error.

i'm very happy about this :-)

|11B|03est |11R|03egads|15, |11M|03artin
|01°±²|15|17 Ä͵XIONUM.COMÆÍÄ |16|01²±°

--- Mystic BBS v1.12 A45 2020/02/18 (Raspberry Pi/32)
* Origin: XIONUM-BBS (21:1/149)

On 27 Feb 2020 at 04:02p, dragonmaster pondered and said...

With this simple (but surely not the best solution) i can up- and
download files to my bbs without any error.

i'm very happy about this :-)

Why not just use something like HTTP to transfer files?

There are two potential issues with something like this.

1. The IP checksum isn't particularly strong with
respect to preventing corruption in flight. Most
of that is implicitly handled by running IP over
a link layer over ethernet or something that does
real CRCs and can detect a much broader range of
errors.
2. You run the risk of additional output written to
your terminal writing into your output file.
That may be unlikely, but it can happen.

--- Mystic BBS v1.12 A45 2020/02/18 (Windows/32)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

Hi tenser,

I use this "protocol" in my own terminalsoftware, written in java, since 2016. No transmission was failed in the last years. I connect over telnet and ssh
to my bbs and it always worked perfect.

But i had think about your notes.

1. The IP checksum isn't particularly strong with
respect to preventing corruption in flight. Most
of that is implicitly handled by running IP over
a link layer over ethernet or something that does
real CRCs and can detect a much broader range of
errors.

Do you mean it can fail if i run it over an "old" dail-in-connection?

2. You run the risk of additional output written to
your terminal writing into your output file.
That may be unlikely, but it can happen.

Yes, this can happen, but when it happens it is no longer important.

Mystic starts the "transferscript" in a new shell.
The only possible messages that can "squak" into this sessions on a
linuxsystem can come from a "kernelpanic", than it's all over, or from the system it self during shutdown (or other errors), than it's all over too.

In case of upload to the bbs: if the filesystem has no space left and the base64-command can't write the output to a file - it while pop out a message
to the terminal, but the transfer is finished at this moment. the rest is out of the users scope.

In case of download: the base64-command can print a "file not found" - but at the end of this line is linebreak which ends the transfer correct. The
terminal on the userside can't "decode" this message as a BASE64-Text and
will show an error, which one decides the programer of the terminal ;-)

I think this method is a "quick-and-dirty" helper when other protocols will fail.

|11B|03est |11R|03egads|15, |11M|03artin
|01°±²|15|17 Ä͵XIONUM.COMÆÍÄ |16|01²±°

--- Mystic BBS v1.12 A45 2020/02/18 (Raspberry Pi/32)
* Origin: XIONUM-BBS (21:1/149)

On 28 Feb 2020 at 07:35a, dragonmaster pondered and said...

I use this "protocol" in my own terminalsoftware, written in java, since 2016. No transmission was failed in the last years. I connect over
telnet and ssh to my bbs and it always worked perfect.

Hey, if it works for you, that's cool; it's just that
if you're going to do something new, there's a lot in
the design space.

1. The IP checksum isn't particularly strong with
respect to preventing corruption in flight. Most
of that is implicitly handled by running IP over
a link layer over ethernet or something that does
real CRCs and can detect a much broader range of
errors.

Do you mean it can fail if i run it over an "old" dail-in-connection?

What I mean is that some kinds of errors are undetectable
by the checksum, regardless of what kind of link layer
you run over. See e.g. http://noahdavids.org/self_published/CRC_and_checksum.html

As I mentioned, most of the time you don't notice these
because most link layers (well, those in common use today)
use stronger CRC, so which the IP and TCP checksums may
not notice those errors, something like Ethernet will, and
the resulting frame will be discarded (and presumably the
higher level protocol will retransmit, hopefully without
errors this time....). However, that's not perfect, either,
and there are some errors that could go undetected, but
it's much much more rare: https://www.evanjones.ca/tcp-and-ethernet-checksums-fail.html
but that links to this, that shows a general problem: https://codingrelic.geekhold.com/2009/11/ethernet-integrity-or-lack-thereof.htm l
The upshot of all of this is that it wouldn't hurt to
include a stronger checksum in your data. But if you're
going to do that, then consider using something like HTTP
over TLS.

Yes, this can happen, but when it happens it is no longer important. [snip]

Yeah, it's admittedly unlikely.

I think this method is a "quick-and-dirty" helper when other protocols will fail.

Hey, if it's workin', then good to go.

--- Mystic BBS v1.12 A45 2020/02/18 (Windows/32)
* Origin: Agency BBS | Dunedin, New Zealand | agency.bbs.nz (21:1/101)

Hi tenser,

The upshot of all of this is that it wouldn't hurt to
include a stronger checksum in your data. But if you're
going to do that, then consider using something like HTTP
over TLS.

That's right, it wouldn't hurt. But i think something like HTTP and TLS is to much in this context. If you use ssh to connect to a bbs than the security is already there. And with ssh it's easy to use SFTP for filetransfer.

have a nice weekend ;-)

|11B|03est |11R|03egads|15, |11M|03artin
|01°±²|15|17 Ä͵XIONUM.COMÆÍÄ |16|01²±°

--- Mystic BBS v1.12 A45 2020/02/18 (Raspberry Pi/32)
* Origin: XIONUM-BBS (21:1/149)