Networking concept help

[erikr]

Hi,

 

I have just purchased two second hand servers building my own Funtoo server cluster here at home. This was actually easier and not particular more expensive than locating motherboards and CPUs with working support for ECC for my ZFS file server. But I also found a second hand server suitable for application server and as the optical fibre is on its way pretty soon, I moved the servers to the basement, and Funtoo is now more suitable for server usage than before, there was just no reason why not to purchase ?

I will come back to the specs in other posts. Right now it is about planning the network. Both servers have several (4+) network interfaces.

 

What I think I am trying to do (well I am open for proposals) is this:

Fileserver will be the ZFS file server. I will need this for the home network. Eth0 is connected to the home network.

Appserver will be the applications server. I will create a post for this as well, but plan is to run docker images for stuff. I see no reason to start with LXD containers but we save this for another thread.

I imagine that the Appserver shall to be on the DMZ-netowork and with a properly and stringent configured firewall.

The thing is that I suppose that the Appserver will need to access the Fileserver. How do I set this up in a secure manner and how do other do?

 

My plan that needs to be scrutinised:

Appserver:eth1 will be connected to the DMZ and accessible to the internet. I will most likely purchase a Firewall with DMZ from Zyxell dealing with the routing. Applications will run in containers. I would like to restrict eth0 for container usage (i.e. only open for the ports used by the containers and rout stuff directly there using nginx or Docker Registry (more study need to be done), no other access to the device but perhaps ssh, not decided yet. Ssh into a docker first?

Fileserver:eth0 will be connected to the home network.

Appserver:eth0 will be connected to the home network allowing me to access the server from home network and allow Appserver to access Fileserver.

I imagine this setup can be vastly improved. Please help me out here!

Also, if the recommendation is to not use the Fileserver but instead add some disks locally please met me know. Appserver does not have that many diskslots, while Fileserver have suffucient (12 or so). Also I prefere letting ZFS Arc on the Fileserver (with 32GB ram) consume most of the memory (this is the only task) while the Appserver (with 128GB ram) can use this for the containers instead.  I will most likely have some containers on the Appserver that need to interact with storage that is available on the local network.

Regards,

Erik

 

 

 

[lazlo.vii]

Hi, Erik.

As I am sure you know, how you set things up is largely a matter of choice. While there are a lot of guides and HowTo's for setting up most FLOS software on the internet I have found very little practical documentation for security and performance best practices.  While  I am certain this type of documentation exists I think it is most likely locked behind paywalls on sites like redhat.com and canonical.com and therefor requires very expensive subscription fees being paid to get access to them.  Of course a deep understanding of the source code would make most of that documentation unnessecary.  It's a shame really that the most needed and important information about Linux seems to be the least accessible to the layman.

That being said, I can offer you a bit of advice based on how I set up my own home server.  Keep in mind that this is only one way out of who knows how many to do what I want.

 

First I use ssh with 8192-bit RSA keys.  In /etc/ssh/sshd_config I have added the following to disable PAM password logins:

AuthenticationMethods publickey,keyboard-interactive

For accessing my server for outside networks I require 2FA with Google Authenticator.  Google Auth is a handy package that allows your cell phone to act as a key FOB for generating time-based one-time passcodes (TOTP) and all it takes to use is installing it, editing a few config files, and never losing your phone.  See https://wiki.gentoo.org/wiki/Google_Authenticator and https://serverfault.com/questions/518802/two-factor-ssh-authentication-on-external-address-only for info on setting it up.

For long term data storage I have 4x 3TB partitions formatted with a command something like this:

mkfs.btrfs --label DATA -m raid10 -d raid10 /dev/sda5 /dev/sdb5 /dev/sdc5 /dev/sdd5

I prefer btrfs over ZFS because a few years back I found myself trying to recover my system after I screwed it up and had to work hard "shoe horning" ZFS support into my recovery environment.  Never again.  Every Linux iso in my library has built in btrfs support so no matter what I want to do my data is accessible. 

When I want give NFS, lxd, or any other service access to my data array I only mount a subvolume at the desired location.  This helps me isolate my data and prevent directory traversal bugs/exploits from affecting my server.  To do this I first have to mount the array and create the new subvolume:

mount /dev/sda5 /mnt/working
	btrfs subv cre /mnt/working/nfs
	umount /mnt/working

Then I edit /etc/fstab to set up the new subvolume on a mount point:

echo "LABEL=DATA     /srv/share     btrfs     subvol=nfs,noatime,compress     1 2" >> /etc/fstab
	mkdir /srv/share
	mount /srv/share

Working with btrfs is nice because it allows abbreviated commands and understands your own personal shorthand.  As you may have noticed to mount the array I only have to tell it mount a single member partition and it will find and mount the rest of them.  For data storage I prefer raid 10 over other raid levels because it offers good I/O speeds and great data redundancy.

I tend to only use KVM for things that I either want and/or need to be truly isolated or for non-linux OS VMs. Everything else I run as lxd containers because the over head is lower and the performance better.  When I do use KVM I have an mdadm RAID0 array configured as an LVM2 volume group that does nothing but store the VM filesystems. 

As far as how you set up your networking, take a look at https://wiki.gentoo.org/wiki/Home_router because it might give you some ideas.  My server also has an Atheros 9271 usb wifi adapter plugged into it and uses hostapd, the ICS DHCP Server, iptables, and BIND to handle my routing and my wifi access.

I hope that helps and if I can think of anything else I'll post again.

EDIT:  For spelling, making links links and to add a P.S.

P.S.  You might like this HowTo I wrote on the Odroid forums.  I could give you some ideas for how to use your new toys:  https://forum.odroid.com/viewtopic.php?f=52&t=33529

[erikr]

Thanks for some good hands on tips lazlo.vii and the area is both as full of different advices as it is lacking good resources.

When it comes to files systems I do  BTRFS mostly but on the server I do ZFS. I once had problems with ZFS where I removal of a zfs from the pool messed things up. I solved it with a FreeBSD live-cd where I imported the pool, removed the zfs and exported the pool. It worked like a charm and since long the bug is fixed.  I still have a "thing" for ZFS though and decided to go for it some time ago (I participated in another forum post here about ZFS some time ago). I already have the pools setup on my current installation and will reuse them in the new server. 

The current network setup plan is as follows:

• AppServer on DMZ
• FileServer on HomeNetwork
• An intermediate physical network between AppServer and FileServer with fixed IPs and just a switch. This way I can make a really narrow IP-filter on the FileServer side for this particular network and I will rely on ZFS export mechanism to export the right stuff. I will try to block unexpected IPs from on the FileServer preventing the AppServer acquire access the nfs-exports meant for the home-network ( nfs will use the IP to determine export rules - faked IP meaning wrong nfs access).

Most likely I will look into SELinux, a new area for me. Containers are new for me (from hosting point of view) as well but I will learn eventually ?

Cheers,

Erik