Self Hosted

Self Hosted: Episode 8 - External Services

Michael Graves

• 14 min read

Software Used:

  • OpenBSD 7.7

Overview

Now that we have some services in our self hosted environment, we may want to allow access from outside our environment. The question is how do we allow this connectivity for ourselves (and anyone else we choose) but not allow the world at large?

To accomplish this we will setup a proxy server. But this will not be a standard proxy server. Our server will be setup to mutually authenticate incoming requests to the services we extend. More on the mutual authentication later.

First we will need to setup a proxy host that will accept requests from the Internet and tunnel them to our self hosted firewall. Here are some of the services we will want to have access to from the Internet.

  • Email access (both sending and receiving)
  • Web access to web based email client (proxied)
  • IMAPS and SMTPS to the email server (proxied)

Setup VM with external provider

There are many providers on the Internet that will provide virtual machines. For this article I will use Vultr (referral link attached). Your are free to use whatever hosting provider you deem best.

Since this VM will be hosted directly on the Internet it will be exposed to all kinds of attacks. My preference for an operating system for used in hostel environments is OpenBSD. Fortunately, Vultr has an option to deploy an OpenBSD VM directly from their console.

I cannot possibly get into tall the different nuances of hosting providers. The basic things to look for is:

a) A variety of build options. This setup will not require huge amounts of CPU or memory. Recommendation is to get at least 1G (preferable 2G of memory). Any CPU speed is fine.

b) Bandwidth. Are there caps. Do you pay for inbound and outbound usage? While email and general apps do not require lots of bandwidth, if you someday extend this self-hosted environment to host music or movies for yourself while you're on the road, bandwidth will spike.

c) ability to set the PTR record. This is possibly the number one requirement when hosting email.

After creating the VM and logging in for the first time there are several things we will want to change.

Create the local user

As with all the setups, I assume that there is a non-root user called 'alfred'. Good chance there is not a default user in the pre-canned OS install from the provider. If your build falls into this bucket create the alfred user.

useradd -g =uid -G wheel -k /etc/skel -s /bin/ksh -m alfred

create the 'alfred' user

Set Root and Local password

If you have logged in as a non-root user, change the users local password. Make it something that cannot be guessed. (We will disable password login via SSH, but the local password still needs to be a strong password.

$ passwd
Changin the password for alfred
New Password: <This should be something really hard to guess!>
Retype new password:
passwd: password updated successfully
$ su -
Password:<enter root password here>
# passwd root
Changin the password for root
New Password: This should be something really hard to guess!
Retype new password:
passwd: password updated successfully
#

password change example

Change the default banner. Blank it out, place a warning, whatever makes sense. 

cat <EOF>/etc/banner
This is a private system.
Go away!
EOF

script to update banner file.

SSH Daemon update

The following commands will set the banner sshd uses, change the port it listens on, deny root login, and disable password authentication(optional).

sed -i 's/^#?Banner.*/Banner /etc/banner/' /etc/ssh/sshd_config
sed -i 's/^#?Port.*/Port 3721/' /etc/ssh/sshd_config
sed -i 's/^#?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/^#?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config

Once the changes are made restart sshd. Note, do not exit your current SSH connection until you've tested it.

rcctl restart sshd

From another shell test the connection on a new port.

ssh -p 3721 [email protected]

If the SSH connection doesn't work, check the config, fix and restart sshd again.

Setup the firewall

Event though this device will be locked down and only run services we want, setting up the firewall to make sure that only the ports we want running are accessible is good practice.

ext=vio0
tun_port=5921

set skip on lo
set block-policy drop
set loginterface $ext
set syncookies adaptive ( start 25%, end 12% )
set limit states 400000
set limit table-entries 2000000

table <whitelist> persist file "/var/db/pf.whitelist"
table <banned> persist file "/var/db/pf.banned"
table <aggressive> persist file "/var/db/pf.aggressive"
table <sshguard> persist

block return
pass in quick on egress inet6 proto icmp6 all icmp6-type { routeradv neighbrsol neighbradv }
pass in quick on egress inet6 proto udp \
        from fe80::/10 port dhcpv6-server \
        to fe80::/10 port dhcpv6-client no state
pass in  quick on egress inet6 proto icmp6 from fe80::/10 to fe80::/10
pass in  quick on egress inet6 proto icmp6 from ff02::/16 to fe80::/10
pass out quick on egress inet6 proto icmp6 from fe80::/10 to ff02::/16
  
# allow traffic to/from wg
pass in on wg
pass out on wg nat-to (wg)

# allow ping
pass in  quick on egress inet6 proto icmp6 all icmp6-type echoreq
pass in  quick on egress inet  proto icmp all icmp-type echoreq

# pass in tunnel port
pass in proto udp to egress port $tun_port

# pass in ssh (remember port was changed to 3721)
pass in proto tcp to egress port 3721

# pass in http and https
pass in proto tcp to egress port { 80 443 }

# pass in email ports
pass in proto tcp to egress port { 25 465 }

# pass in imap ports
pass in proto tcp to egress port { 143 993 }

# allow us to send traffic out
pass out from self

/etc/pf.conf

Update Hosts file

We will need to connect to the real IP's of the internal hosts. To do this we will override the hostnames by placing the real IPs into the hosts file.

127.0.0.1 newserver.example.org
::1       newserver.example.org
192.168.30.14 sso sso.example.org
192.168.30.15 mail mail.example.org
192.168.30.11 certauth certauth.example.org

/etc/hosts

We will also need to tell the OS resolver to look at the hosts file for resolution. To do this add this like to the /etc/resolv.conf.

lookup file bind

partial /etc/resolv.conf

Update DNS with server name

Update the external DNS provider with the name and IP of this server. We will also need to add the following records.

example.org            A   add.server.ip.address
newserver.example.org  A   add.server.ip.address
mail.example.org   CNAME   newserver.example.org
smtp.example.org   CNAME   newserver.example.org
imap.example.org   CNAME   newserver.example.org
sso.example.org    CNAME   newserver.example.org
certauth.example.org CNAME newserver.example.org
autoconfig.example.org  CNAME newserver.example.org

Get certificate

This server will have many aliases. Since it will be proxying traffic for all internal systems we choose to expose. Since this will be public facing server we may want a "real" public certificate. We will also need an internal certificate for communicating with the rest of the self-hosted environment. acme-client(1) can be used to get both certificates.

# basic HTTP config for certificate generation
server "*" {
        listen on * port 80
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location * {
                block return 302 "https://$HTTP_HOST$REQUEST_URI"
        }
}

Next we will configure the acme-client.conf file to get the certificate from Lets Encrypt.

#
authority letsencrypt {
  api url "https://acme-v02.api.letsencrypt.org/directory"
  account key "/etc/acme/letsencrypt-privkey.pem"
}

domain newserver.example.org {
  alternative names {
    mail.example.org
    smtp.example.org
    imap.example.org
    sso.example.org
    certauth.example.org
    autoconfig.example.org
    example.org
  }
  domain key "/etc/ssl/private/newserver.key"
  domain full chain certificate "/etc/ssl/newserver.fullchain.pem"
  sign with letsencrypt
}

/etc/acme-client.conf

We will also need an internal certificate to mutually authenticate clients. We will do they by creating a second acme config file.

#
# example.org acme-internal.conf
#
authority example.org {
        api url "https://certauth.example.org/acme/acme/directory"
        account key "/etc/acme/example.org-privkey.pem"
}

domain newserver.example.org {
        domain key "/etc/ssl/private/newserver_internal.key"
        domain full chain certificate "/etc/ssl/newserver_internal.pem"
        sign with example.org
}

/etc/acme-internal.conf

Now start httpd and retrieve the certificates.

rcctl enable httpd
rcctl start httpd
acme-client -v newserver.example.org
acme-client -v -f /etc/acme-internal.conf newserver.example.org

command to start httpd and issue new certificates

We will need to periodically update the certificates. Update crontab with the following lines. Use the 'crontab -e' command to edit the crontab file. This will run the check/update every Sunday at a random time after midnight.

0 0 * * 0  sleep $((RANDOM \% 2048)) && acme-client newserver.example.org
0 0 * * 0  sleep $((RANDOM \% 2048)) && acme-client -f /etc/acme-internal.conf newserver.example.org

crontab to update the certificates as needed

Create Tunnel from firewall to newserver

To create a tunnel we will need to update the configs on both the newserver as well as the firewall server. We will assume that the firewall server does not have direct access to the Internet (because if it does, the article to this point is unnecessary). Due to the firewall having to pass through one or more NAT's we will setup the firewall to initiate the connection and the newserver to just listen for connections.

We will be configuring these systems in parallel as we will need information for each server to complete the configuration on the other.

First on both systems we will create a private key.

openssl rand -base64 32 > /etc/wg.pkey
chmod 640 /etc/wg.pkey

create and save wireguard private key.

New interfaces config

Now create a new /etc/hostname.wg0 file.

wgkey `cat /etc/wg.pkey`
wgport 5921
#wgpeer ### some public key ### wgdescr firewall wgaip 0.0.0.0/0
inet 192.0.2.1 255.255.255.0
!route add -net 192.168.30.0/24 192.0.2.10

newserver /etc/hostname.wg0

Do the same thing on the firewall

wgkey `cat /etc/wg.pkey`
# wgpeer ### some public key### wgendpoint newser.real.ip.address 5921 wgdescr wall wgaip 0.0.0.0/0
inet 192.0.2.10 255.255.255.0

firewall /etc/hostname.wg0

Now load the private keys. This will show us public keys.

/bin/sh /etc/netstart wg0

load the new wireguard interface config

Once the interfaces are loaded get the public key.

ifconfig wg0 | wgpubkey

command to get the public key

Now replace the ###some public key### part of the configs with the opposite public key (also remove the leading # sign). i.e. newserver public key goes into the firewall /etc/hostname.wg0 config file and visa versa. Once the configs are updated reload the configs. Test pinging from firewall server to the newserver.

firewall# ping 192.0.2.1
PING 192.0.2.1 (192.0.2.1): 56 data bytes
64 bytes from 192.0.2.1: icmp_seq=0 ttl=255 time=28.060 ms
64 bytes from 192.0.2.1: icmp_seq=1 ttl=255 time=29.061 ms
64 bytes from 192.0.2.1: icmp_seq=2 ttl=255 time=27.577 ms
^C
--- 192.0.2.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 27.577/28.232/29.061/0.618 ms

example of ping

Once the ping is successful from firewall to newserver, the reverse should also work. Since the tunnel will need to be established from the firewall to newserver we will use the ifstated(8) process to keep the tunnel active. This config will ping the new server over the tunnel every 60 seconds. Adjust as necessary if 60 seconds isn't enough to keep your connection alive.

#
# Keep the tunnel alive
#
chk_peer = '("ping -q -c 5 -w 1 192.0.2.1 >/dev/null 2>&1" every 60)'

init-state tunnel_down

state tunnel_up {
        init {
                run "logger -st ifstated 'state: UP'"
        }
        if ! $chk_peer {
                run "logger -st ifstated 'tunnel DOWN'"
                set-state tunnel_down
        }
}
state tunnel_down {
        init {
                run "logger -st ifstated 'state: DOWN'"
        }
        if $chk_peer {
                run "logger -st ifstated 'tunnel UP'"
                set-state tunnel_up
        }
}

/etc/ifstated.conf

Enable and start ifstated

rcctl enable ifstated
rcctl start ifstated

Setup relay service

Now that the tunnel is up we can setup the relay service. This will accept connection (on specified ports), perform TLS authentication of the client, and then forward to the internal servers. To start this, create the /etc/relayd.conf file.

ext_ip="newserver.ip.addr.ess"
ext_ip6="newserver:ipv6:ip::address"

sso_int="192.168.30.14"
mail_int="192.168.30.15"
certauth_int="192.168.30.11"

table <sso_table> { $sso_int }
table <mail_table> { $mail_int }
table <certauth_table> { $certauth_int }

# TLS PROXY for Internal services
tcp protocol tls_proxy {
        tcp nodelay
        tcp sack
        # validate the clients against internal CA
        tls client ca "/etc/ssl/example.org.pem"
        tls keypair "newserver_internal"
}
# HTTPS PROXY for Internal services
http protocol http_proxy {
        return error
        match header set "X-Forwarded-For" value "$REMOTE_ADDR"
        match header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
        match header set "Keep-Alive" value "$TIMEOUT"
        # match based on the Host header and set the table
        pass request header "Host" value "sso.example.org" forward to <sso_table>
        pass request header "Host" value "mail.example.org" forward to <mail_table>
        pass request header "Host" value "certauth.oexample.org" forward to <certauth_table>
        # only trust devices that have our cert
        tls client ca "/etc/ssl/example.org.pem"
        tls keypair wall
}
# setup the relays
relay proxy {
        listen on $ext_ip port 443 tls
        protocol http_proxy
        forward with tls to <sso_table> port https check icmp
        forward with tls to <mail_table> port https check icmp
        forward with tls to <certauth_table> port https check icmp
}
# remove if IPv6 is not used
relay proxy2 {
        listen on $ext_ip6 port 443 tls
        protocol http_proxy
        forward with tls to <sso_table> port https check icmp
        forward with tls to <mail_table> port https check icmp
        forward with tls to <certauth_table> port https check icmp
}
# setup mail and imap relays
relay mail_proxy {
        listen on $ext_ip port 465 tls
        protocol tls_proxy
        forward with tls to <mail_table> port 465 check icmp
}
relay imap_proxy {
        listen on $ext_ip port 993 tls
        protocol tls_proxy
        forward with tls to <mail_table> port 993 check icmp
}
# remove if IPv6 is not used
relay mail_proxy2 {
        listen on $ext_ip6 port 465 tls
        protocol tls_proxy
        forward with tls to <mail_table> port 465 check icmp
}
relay imap_proxy2 {
        listen on $ext_ip6 port 993 tls
        protocol tls_proxy
        forward with tls to <mail_table> port 993 check icmp
}

/etc/relayd.conf

Enable and start relayd(8)

rcctl enable relayd
rcctl start relayd

enable and start relayd

Now create a user/email certificate from our certauth server as described in Episode 4 - Host/Email certificates. Connect to the web email interface from a device that is not on the internal network. The web browser should prompt use to for a user defined certificate. If you ignore the browser request and continue the server should abort the connection. Try again, but this time provide the browser the certificate created and confirm that the web interface is now presented.

Setup MAIL server

Finally we stage where we will configure our ability to send and receive external email. As was discussed in Episode 7 we will need to setup SPF, DKIM and DMARC records. This will tell the world about our valid email service and provide a level of trust. While it's not guaranteed that our email will always be accepted, it goes a long way to indicate that our email server is run in a responsible fashion.

Add DNS records

As we did when we setup the mail server we will need to setup some external DNS records. These records will be used by external mail providers to make sure that our mail server is legitimate.

The first record is the MX record. This will tell the world where to send email to. The @ sign indicated the our root domain. Depending on the DNS provider that part may be able to be left blank.

@    IN   MX    10  newserver.example.org.

MX DNS record

SPF is the next record we need. Here we indicate what servers mail will come from for our domain. Since we will be receiving and sending mail from a single server we can just setup the SPF record to indicate the use of the MX values and to deny all else.

@   IN   TXT   "v=spf1 mx -all"

SPF DNS record

The DMARC record tells the world what we suggest be done when receiving email that does not comport to our DKIM and SPF settings

_dmarc  IN  TXT    "v=DMARC1;p=quarantine;pct=100;rua=mailto:[email protected];ruf=mailto:[email protected];"

DMARC DNS record

Finally we will create the DKIM record. This will be based on the private key we created on the mail server. On the mail server we can use the below command to generate the DNS record.

openssl rsa -in /etc/mail/dkim/example.org_priv.rsa.key -pubout 2>/dev/null | sed '1s/.*/v=DKIM1;p=/;:nl;${s/-----.*//;q;};N;s/\n//g;b nl;'

command to generate DKIM TXT record

The output should look something like this

v=DKIM1;p=MIIBIjANBgkqhki_LOTS_OF_MORE_BASE64_E1wIDAQAB

example of DKIM public key

Now add that to the DNS record.

selector01._domainkey    IN   TXT   "v=DKIM1;REST_OF_THE_LINE"

DKIM DNS record

Create SMTP config files

This smtpd.conf file will be a little different that the previous one we built for mail. This file will accept mail from any location, filter that mail based on some well known spamming techniques and then forward it onto the mail.example.org server in our self hosted environment.

#
# mail relay config
#
# 1) server will receive any mail destin for <domains> table and
#   forward to internal relay
# 2) server will receive mail from internal server for any and
#   relay
#
# DKIM signing will occur on the internal server
smtp max-message-size "100M"
ca  local_ca cert "/etc/ssl/example.org_root_ca.pem"
pki mail_cert cert "/etc/ssl/wall.fullchain.pem"
pki mail_cert key  "/etc/ssl/private/wall.key"
pki int_cert cert "/etc/ssl/wall_internal.pem"
pki int_cert key  "/etc/ssl/private/wall_internal.key"

table   aliases         file:/etc/mail/aliases
table   domains         file:/etc/mail/domains
table   bad-from        file:/etc/mail/blacklist
table   bad-to          file:/etc/mail/spamtrap
table   relayaccts      file:/etc/mail/relayaccts
table   relays          file:/etc/mail/relays
table   creds           file:/etc/mail/credentials

#check for dynamic dns sources
filter check_dyndns phase connect match rdns regex { \
  '.*\.dyn\..*', \
  '.*\.dsl\..*', \
  '.*dynamic.*', \
  '*\.vodafonedsl\.*', \
  '*\.vodafone\.*' \
  } disconnect "550 no residential connections"

# check for numeric IPv4 sources
filter check_ip_names phase connect match rdns regex { \
  '[0-9]{1,3}[-\.][0-9]{1,3}[-\.][0-9]{1,3}[-\.]]0-9]{1,3}[-\.].*' \
  } disconnect "550 no residential connections"

# confirm rdns is valid
filter check_rdns phase connect match !rdns \
  disconnect "550 no rDNS"

# confirm fcrdns is valid
filter check_fcrdns phase connect match !fcrdns \
  disconnect "550 no FCrDNS"

# check to see if mail is from a known bad sender
filter check_bad_from phase mail-from match mail-from regex <bad-from> \
  disconnect "550 go away spammer"

# check to see if mail is going to a bad receptient
filter check_bad_to phase rcpt-to match rcpt-to regex <bad-to> \
  disconnect "550 go away spammer"

# combine filters into a list
filter initial_checks chain { check_rdns, check_fcrdns, check_ip_names, check_dyndns, check_bad_from, check_bad_to }

# listen for incoming email and apply the filter list
listen on all tls pki mail_cert filter initial_checks
# listen for relayd mail on the tunnel interface
listen on wg port 9465 smtps pki int_cert auth <relayaccts> tag RELAY

action "outbound" relay
action "inbound" relay host smtps://[email protected]:9465 tls auth <creds>
# local mail, send inbound
match for local action "inbound"
# anything marked as relay, send outbound
match tag RELAY from src <relays> for any action "outbound"
# anything for one of our domains send inbound.
match from any for domain <domains> action "inbound"

# default deny

/etc/mail/smtpd.conf

# domains
#
# list domains to accept mail for
# to update smtpd use 'smtpctl update table domains'
#
example.org

/etc/mail/domains

#
# place sender names of domains (or whole domains here)
# to deny their acceptance.
#
#[email protected]

/etc/mail/blacklist

#
# List of servers that we will accpet relayed mail from
# use 'smtpctl update table relays' to update table
192.168.30.15

/etc/mail/relays

# spamtrap
#
# place email names of accounts that should never 
# receive email.
#[email protected]

/etc/mail/spamtrap

#
# relay credentials
# use 'smtpctl update table creds' to update
# relay user
ponyexpress     ponyexpress:MAKE_THIS_A_GOOD_PASSWORD

/etc/mail/credentials

#
# user names to be used for relays
#
# use 'smtpctl update table relayusers' to reload file
# use 'smtpctl encrypt "password"' to encrypt password
ponyexpress     $2b$08$19n8/YAQQxBxOVApmDxmHuQVqtbMZNSADb6LJaGvRW1NdW1oOO0y.

/etc/mail/relayaccts

With all those files in place check that smtpd(8) is happy then restart if successful.

smtpd -nvf /etc/mail/smtpd.conf
# if successful restart
rcctl restart smtpd

commands to check validity of smtpd.conf and associated files

Update forwarding on internal mail server

Now we can receive external email, we need to revisit the mail.example.org server to have it be able to forward email via newserver. On mail server un-comment the lines that were left commented.

table relayaccts        file:/etc/mail/relayaccts
table creds             file:/etc/mail/credentials

listen on 0.0.0.0 port 9465 smtps pki mail_cert auth <relayaccts> tag RELAY

action "relay" relay host smtps://[email protected]:9465 tls auth <creds> helo "mail.example.org"

match from auth for any action "relay"

lines in smtpd.conf to uncomment

Be sure to add the relayaccts and credentials files. These can be mirrors of the ones on newserver.

#
# relay credentials
# use 'smtpctl update table creds' to update
# relay user
ponyexpress     ponyexpress:MAKE_THIS_A_GOOD_PASSWORD

/etc/mail/credentials

#
# user names to be used for relays
#
# use 'smtpctl update table relayusers' to reload file
# use 'smtpctl encrypt "password"' to encrypt password
ponyexpress     $2b$08$19n8_ENCRYPTED_PASSWORD_W1NdW1oOO0y.

/etc/mail/relayaccts

Now check that the updates are valid and then restart the service.

smtpd -nvf /etc/mail/smtpd.conf

rcctl restart smtpd

Sending our first emails

Now that everything is in place, send your first email. Send it to an alternate account you have. If you receive it, respond to make sure that inbound access is working. If you run into problems, review the /var/log/maillog files on both newserver and mail servers for indications as to what failed.

Conclusion

Now that we have email access to and from our self hosted environment to the rest of the world... you need to decide what is next! Good Luck!