Alrighty in the end this is storage theory I went with, each server device
has its own drives jbod and shared, and the whole lot are merged before appearing on the network share. So the grand mess looks like this lot.

Filesystem Size Used Avail Use% Mounted on
mergerFS 4.9T 390G 4.4T 9% /local
/dev/sde1 917G 284G 588G 33% /jbod/1
/dev/sdc1 916G 86G 784G 10% /jbod/3
/dev/sdb1 293G 21G 258G 8% /jbod/4
/dev/sdd1 2.8T 101M 2.8T 1% /jbod/2
//192.168.1.9/data 5.8T 87G 5.4T 2% /sophos
//192.168.1.254/data 3.5T 3.2T 246G 93% /thecus
mergerFS 14T 3.7T 10T 27% /data

Only concern at this stage is just how many HD's are online. However being a jbod rather than LVM or RAID if something falls over it'll only take itself
out not the whole sharing arrangement. Also configured like this there is
no inherent backup, and most of my usb drives and in there too.

However as previously mentioned there's only some 150-200GB that I feel the need to keep on top of backups for. So my options for backup are either another USB Drive, one or two left, or fire up snaggletooth (G4 PowerMac)
and use rsync.

During this messing around, somehow I ended up with an upgraded nfs-kernel-server which stopped all the DOS clients for the BBS being able
to talk to brightmatter at all. A fair amount of head scratching and
attempted re-installs resulted in a purge of nfs-kernel-server and
nfs-common prior to everything just working again with the original distro version. This doesn't bode well going forward for XFS191, DOS NFS Client if its no longer able to talk to "production" installations.

Some file copying is still going on... I've been moving a heap of data out
onto the THECUS as it was large and empty of course. But a combination of a slow HD which has some kind of strange errors on it, and network copy its
only managing 6.6MB/s for close to 2TB of data :/ 3 days now... plus is
stops and waits everytime it decides it can't read.. skip or try again. Its presently promising 23 hrs to go, we'll see, we'll see.

So once all that is out the way, the server side of this network should be
all hiding behind brightmatter. So nic1 faces the client side of the
network and handles all the share traffic. nic2 only faces the other
storage devices so they have full bandwidth of their own to talk to each
other.

Limitations in this are some of the storage sections are still only fast ethernet while the front side will be Gb ethernet. So storage speed on some level will still be a limiting factor. Unless I can find some decent kind
of software caching that can be installed on brightmatter to help things
along. Preferably RAM based as its certainly not using everything its got available, and I don't happen to have a spare SSD for use..

And that is the current state of play... I'm happy with a 14TB slab of
shared space, somewhat concerned I have no seperate backups at the moment.
But thats an easy fix. And will now have to sit down and consolidate some of the saved in multiple locations stuff that doesn't need to be. Flashcache
looks promising, but I suppose I could just use tmpfs to create a large ramdrive and use that...

Progress continues getting that copy finished, and physical reorganisation
of the network.

Spec

- The Emperor Protects, but a loaded bolter doesn't hurt.

--- SuperBBS v1.17-3 (Eval)
* Origin: Scrawled in haste at The Lower Planes (21:3/101)

Impaired file system...

Well... I did mention I had to reinstall NFS from scratch... of course it took the exports file with it. It was missing the primary speed enhancer going... asyn . Re-enabling that appears to have gotten performance back where I was expecting it. With perhaps a few new other optimisations that I wouldn't have bothered with if it hadn't been behaving to slowly.

And on the other side, of course the first wall I decided to beat my head against was in fact samba.. not much you can do there, although I do have 3 new options in place for testing. Took a while for the realisation to sink in, the BBS isn't even using samba you donkey its NFS...

At a lower level poking about with iperf only served to confirm that the network physically is just fine. Although my home cabling isn't managing Gb speeds on the run to the daily drive PC. The cabling issue is one I've come across before. I'm unsure why, might be the length, this one is about 8M give or take. But I can't get hand made cat5e cabling to give me Gb negotiations..

Spec


--- SuperBBS v1.17-3 (Eval)
* Origin: (21:3/101)