Obtuse Networking Blog

Running AI..

2026-10-21T00:00:00Z

AI is finally becoming useful... to me. When it was just a "chat" bot, I couldn't have cared less. And when it could spit out a python script that I had to copy and paste into an editor and test and debug myself... I still wasn't impressed. But now that AI can "use tools", it is now becoming useful to me.

I am not an AI pundit or an expert by any stretch of the imagination. I have not drunk the cool-aide and think AI will change everything. I'm just some guy on the Internet and I do think things will change. So take these tips for what their worth.

You do not need to follow or even understand everything here before starting with AI. You most likely will not. As you gain more experience revisit this page and what didn't make sense at first, may start to make more sense.

Here are some of my "discoveries" when running AI.

First things to always remember

Keep these items in mind when using AI.

YOU ARE RESPONSIBLE!!! And you will be held responsible. If you ask an AI to do a thing and it does it. Or you ask AI to do a thing and it does something totally different. You are responsible for any damage or outcome that happens. Never forget that.
AI is like an idiot savant or a stochastic parrot. What AI can do is amazing, but AI is just a tool. It doesn't "know" anything. It doesn't "understand" anything.
AI uses human language, so it can seem human. It is NOT. AI is, at its core, a large (really large) multidimensional array. The fact that you can put words in and get words out something that make sense and can do things is like magic... but its just technology. Remember what Arthur C Clark said "any sufficiently advanced technology is indistinguishable from magic". Right now, AI is like that.
Hardware prices are crazy for running models yourself (Oct '26). Start with a monthly subscriptions to one of the "frontier" companies if you have never used AI. Once you get some of the basics down, you'll probably want try a different harness than the "frontier" company harness. Then you will start with a site like ollama.com, huggingface.co, nousresearch.com, openrouter.ai, etc and be able to select from a bunch of different models.

General usage guides

Harness matters a lot more than the model. A good harness with a marginal model can give you good results. A bad harness with a great model can also give you good results. The difference is the great model probably costs 3x more than the marginal model. Result matter.
Don't get hung up on the 'latest and greatest' model released. Find a model that can do what you want for the price you want to pay.
You need to have an idea / goal first. Take as much time as you need to refine that goal. Ask AI to help refine that goal (which is, in itself, a goal). But get the goal for AI clearly defined first.
Remember "if you don't know where you are going, any road will get you there... lost". So it is with AI. AI will give you an answer. It will most likely not be 100% correct, but it may be close enough, but how can you tell? Always double check the answer.
Running multiple sub-agents may get things done faster, but will cost a many more tokens because every time sub-agent is spun up you will path the whole ingest tax. For long projects this can really add up. For most jobs, stick with one model.
For repeated activities (or ones you think will be repeated) have the model create scripts that both it and you can run. This will allow you to do the activity without the model.
Getting clever with prompts is usually not worth the effort. Todays models (as of Oct '26) are really good at interpreting your intent from your questions. With that said, you still need to be clear on what the goal of the task is and any parameters that the model should align to. Giving a 'one shot' prompt is a gimmick.
When prompting do:
a) Set a clear goal for the model. Define when the task is complete and the model is done.
b) Provide any parameters you want it to follow or adhere to.
c) Explicitly tell AI what you don't want it to do. Know that it may still ignore you and do it anyway. So make sure you have a backup way to prevent it.
d) Tell the model how you expect it to save its progress. This will also help with the model being able to "remember" where it left off.
e) Ask the model to repeat back to you or summarized its understanding of the ask before doing any work.
f) Have AI create a plan before any work is done so you can review the plan. And do the review. This will keep both you and the model aligned.
g) Explicitly tell the model to interview you or ask you questions for clarifications.
h) Tell the model to record the assumptions that it makes in its history. This way you can review it.
Don't chase parameters, quantization, test suites, etc. Find a model that works for you and what you want to do. Build your own series of "tests" and what "good" looks like. Compare models to that.

Application Tips

Basically all application development habits and process you use today are what we want the model to use.
Make sure the model creates a Makefile or other standard build file for the application. This way you can make changes and/or swap out the models and still receive consistent results.
Always tell the model to initialize a new git project in the local working directory. Even if you don't intend to have the project be replicated anywhere else. This provide history of the work and the models generally provide really good summaries of what each commit provides.
For each feature or step in the plan, have the model create a new branch to do the work and when complete and tested, merge that branch back into main.

Final thoughts...

None. I will continue to update this list for myself as time goes on. Hopefully this will be helpful for you.

All Aboard! The Upgrade Train

2026-10-14T13:57:44Z

(brought to you by AI... I mean the constant upgrades... not the article... that was all me... and my spell checker)

As IT professional(s), I (we) have been on the constant upgrade cycle. So much so, Microsoft created a monthly 'holiday' where we get 'gifts' from them.

The Problem

Different IT environments get different levels of tolerance for upgrades/patching. For Microsoft servers, there is usually an agreed upon window of time where 'upgrades' are done and the business knows to 'take a break'. However, additional IT environments are trying to join those Microsoft 'take a break' windows and a complete mess is ensuing. Storage patches need to go into place impacting the actual server storage the Microsoft servers rely on. And network upgrades are impacting both of them, leading to situation of:

No network,
No storage,
No servers...
No business

AI hasn't made that easier. Besides the flood of 'slop' patches, AI is actually finding real bugs and real vulnerabilities. But AI is doing it at a rate that neither the software developers nor the business users can handle.

Is there an Answer?

Honestly, not without a lot of work.

I've heard some people floated the idea that 'we just need to get thru this period and eventually all the bugs will be found and we will have perfect software'. In a static environment, that may be true. But who lives/works in a static environment? I think that there are several things that are working against this idea:

Developer fatigue: Researching and understanding a bug and then developing a patch and patching a software is not an instant process. It takes time. Time no one has. With the amount of reports coming in, we have a choice... drowned or fight AI with AI. Call in the life line of AI to evaluate, propose, and patch the bugs. How soon will it be before we just press the 'I believe' button and tell AI... just go do it? How soon will it be before we can no longer reason about the software?
The AI models keep getting better and finding more kinds of bugs. More reports, etc. etc. The key here is AI's ability isn't fixed.
Model availability: Or the lack there of. Not all models are available to everyone. Many models have been restricted to just some 'elite' companies for special purposes. Everyone else... you just get the scraps. Good Luck!
Model affordability: At some point in time the full cost of running models will come due. And those companies and/or individuals that cannot afford to pay, will not be allowed to play. At the risk of speaking something into existence... how soon we will start seeing 'Go Fund Me' campaigns for tokens?

Let's assume that none of those items manifests itself. Where does that leave us, the IT pro, who is trying to do the best we can, not having to upgrade the literal entire IT environment every... month, week, day? Here are my suggestions...

Short term: the business, as it is...

The business will need to have more 'downtime' in a month. If server patches are done on Sunday, then Storage patches are done on Saturday and Network (inclusive of firewall, WAFs, proxies, etc) patches are done Friday.
We will be introducing a lot of change. We will want to develop some good 'canary testing' methods to be reasonably sure when an upgrade is complete, everything is 'as we expect it'.
If you are in a business whose size makes a single patching day per area impractical, then that business will need to have more patching days per month. Maybe geographically related. (East coast on first week, West cost third week)

There will be pushback. But remind those that push back that patching systems doesn't prevent the business from making money... it protects the business so that it can continue to make money. Think of it like an insurance. Bad stuff will happen. This is you paying your premiums so that when it does happen, the impact can be recovered. If that is too nuanced, just think of it like this... It's just the cost of doing business.

Long term: the business, as it will need to become...

The business (and IT), will need to transform its thinking. Instead of relying on servers, and storage, and network we need to start thinking in terms of services. No business buys a server or storage or network gear just for that gears sake. (IT geeks excluded) That gear is purchased to provide a service. We need to begin thinking about those services. Service is important. Servers are not.

How do we do this?

Assume failure: Any server, storage, network device will fail. We have developed processes to try and weed out those failures early and to minimize them... but they do and will continue happen. Let us not 'hope they don't happen'. Let us assume they will and have a plan.
Stop worrying about the shortest path: What is the shortest path between your machine and Google? Microsoft? You don't know. And you don't need to know. So stop trying to optimize it. The only thing our users care about is they can get to the service they desire, when they desire them.
Re-architect applications: You will need to re-architect the application the business relies on. Just shoving the servers into a "cloud" and calling it "cloud enabled" is a lie. We can no longer lie to ourselves or allow ourselves to be lied to. What does re-architect mean? In a nutshell it means:
a) separating your data from its presentation layer (where your customers, employees, etc. interact with the data).
b) re-structuring how the data is organized so that it can be easily chunked up and replicated across physical locations.
c) and while we are at it, we need to plan in how we backup and restore the data when it is corrupted or lost.

"Does this mean I must move to the cloud and pay the cloud tax?"

No. You will need ~~some~~ multiple places to put servers, storage, network hardware in our environment. It also means you will need start thinking of your server rooms and data centers like the cloud providers do. Modular-ize. Container-ize.

"Now that I've done all this work what does my business get out of it. I still have to patch, upgrade, etc."

Correct. But now your application is setup to be "always on". Now your users can work from the office or their home or anywhere.

And now we're back to...

"How do I re-architect my application(s)?"

Well... AI. Can AI build a perfect application? No. AI can however, with good prompting, planning, validation and oversight, take an ~~ancient~~ historic application, and rebuild it in a modern, safe fashion using today's tools and methods. Slop is still possible. But maybe, a slop application is all that is needed for a specific service. Today's AI can build more than just slop. It can really customize an application for your specific business. Instead of what we all have done for the last 30 years which is change our business around the applications we could afford to rent.

Conclusion?!?

Instead of being stuck on the track and being run over by all the fallout of AI... can we use this time of change to help ourselves, our businesses, our families, achieve a better outcome?

Automatic Certificate Issuance with ACME

2026-07-13T11:47:22Z

In a previous article here, I described how to setup an ACME server using Step-CA software. While writing that article I got tired fighting with some of the Step-CA nuances and asked myself if there was a better way.

Not finding a better way to issue certificates using the ACME protocol, I created ACME.cgi. This is a CGI script that will utilize an external CA tool to issue/revoke certificates using the ACME protocol (and yes... it's a shell script!).

Overview

The ACME.cgi script will utilize a standard web sever and CGI environment to allow ACME clients to register and issue certificates for domains. The ACME.cgi script does not actually do the work of creating, signing, revoking certificates. Those operations are farmed out to external tools.

This article, and ACME.cgi in general, assume that the external tools is Personal Certificate Authority (PCA). However, this is not required.

Setup

Since servers and requirements vary greatly, the options below give the steps to deploy ACME.cgi into a Docker, non-chroot and chroot web environment. Please note, that I cannot know your specific server setup and you may need to make additional adjustments.

These setups all assume that the ACME.cgi service will utilize the Personal Certificate Authority (PCA) script to manage certificate functions. (If you want more information on what PCA is checkout this article.) If you choose to utilize another tool, update/replace the ACME_helper.* files with whatever makes sense for your setup.

For All Options

The following will need to be done for all deployment options.

Download the latest version of ACME.cgi from github.
Download the latest version of PCA from github.
1. Copy the pca script to a location in the PATH. i.e. /usr/local/bin/. If you will setup ACME.cgi in a CHROOT environment, then this path must be callable from that environment.
2. Since these instruction assume that PCA will be called from the ACME_helper script, set the PCA_ROOT environment variable to the location where the ACME.cgi service is expected to store the certificate environment. i.e. /var/www/etc/pca.
Install a FastCGI server. e.g. fcgiwrap, slowcgi. This program is probably already in the package manager of the server.
Setup web server to send traffic to CGI file. Below is a quick config for nginx. This configlet assumes the following (adjust for your setup):
1. The path ACME.cgi will be called from is https://ISSUER_DOMAIN/acme/ (where ISSUER_DOMAIN is the FQDN of the server hosting the application)
2. The web root is /var/www/html
3. Web server certificates are kept in /etc/certs
4. FastCGI program is listening on localhost port 9000

server {
  listen 443 ssl;
  listen [::]:443 ssl;

  ssl_certificate /etc/certs/server.fullchain.pem;
  ssl_certificate_key /etc/certs/server.key;
# use if the server key is encrypted
#  ssl_password_file /etc/certs/server.pass;
  ssl_protocols   TLSv1.2 TLSv1.3;
  ssl_ciphers     HIGH:!aNULL:!MD5;

  index index.html;
  root /var/www/html;

  location /acme/ACME.cgi {
    include /etc/nginx/fastcgi_params;
    fastcgi_pass 127.0.0.1:9000;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param SCRIPT_NAME nfquery.cgi;
  }
}

nginx server configlet

Deployment Options

On Docker system

This is the simplest deployment method. There is a Makefile created to assist in creation of the docker container. Three volumes are used, /config, /data and /logs. Config is used to store the configuration files for ACME.cgi and the web server. Data is where PCA certificates are stored as well as the ACME working files. Logs as the name implies, is where the logs are stored.

Change to the package/Docker path.
Build the docker image by running make build.

make build

Copy the env.example file to .env. Configure the .env file as necessary. Note: Set SERVER_NAME, CONTACT_EMAIL and SERVER_SECRET for the site.
Start the docker container. Use option A or B
1. Start docker via make run.
2. Start docker-container via docker-container up -d

The docker container will create a new CA, create and export a CA directory, create certificate and keys for the container and enable all cron scripts.

On non-chroot systems

This is only slightly more complicated that the docker image. In this option, all configuration steps will need to be performed manually.

Copy the ACME.cgi ACME_helper.sh and ACME_cleaner.sh files to the CGI-BIN directory.
Copy the ACME.conf example to the /etc directory.
1. Update the ACME.conf file as necessary. Note that the ISSUER_DOMAIN and ISSUER_EMAIL do not have default values and are required to be filled out.
Copy the ACME_helper.conf file to /etc directory.
1. Set the CA_NAME to be the name to initialize PCA with (or the name that PCA was initialized with, if done in another step.)
2. Optional: If initializing PCA here run the PCA init command. These commands will:
  1. Initialize the PCA environment using the CA_NAME that was described above.
  2. Create a certificate chain
  3. Create a CRL file (nothing will be in it yet)
  4. Export an html directory with those files in them.
  5. Extract that html export into a web root under a /ca directory.

PCA_ROOT=/etc/pca pca <ca_name> init
PCA_ROOT=/etc/pca pca <ca_name> create chain
PCA_ROOT=/etc/pca pca <ca_name> create crl
PCA_Root=/etc/pca pca <ca_name> export html -file /tmp/ca_html.tgz -overwrite
tar -C /var/www/html -zxf /tmp/ca_hml.tgz

commands to create the CA HTML files on the CA server

Verify that everything is working by connecting to https://ISSUER_DOMAIN/acme/directory. This will return a directory listing of ACME functions and confirm the script is installed.
Setup the ACME_cleaner.sh script to run daily.
Create a script and set it to run daily that will create a new CRL, HTML export and deploy that export.

#!/bin/sh
export PCA_ROOT=/etc/pca
export CANAME=<ca_name>
pca ${CANAME} create crl
pca ${CANAME} export html -file /tmp/ca_html.tgz -overwrite
tar -C /var/www/html -zxf /tmp/ca_html.tgz

script to maintain the CRL on the CA server

On CHROOT systems

This is perhaps the most complicated setup. If the system is running on OpenBSD, check out the packages/OpenBSD directory. There is a port that can be run to create an OpenBSD port to deploy. In addition read packages/OpenBSD/pkg/README. That file has all command necessary to copy the helper programs into the CHROOT environment.

Copy the ACME.cgi, ACME_helper.sh and ACME_cleaner.sh files to the CGI-BIN directory.
Copy the required helper programs into the CHROOT directory (Assume the directory /var/www/etc)
1. The following directories are used and will need to be created in the CHROOT environment: /tmp, /var/run
2. The following programs are required to be accessible by the CHROOT environment, as well as any libraries that the programs call: cat, sleep, rm, cut, mktemp, host, curl, fold, od, openssl, printf, wc, jq
3. The following device is required to be available in the CHROOT environment: /dev/null
Copy the ACME.conf into a config directory located in the CHROOT path. i.e /var/www/etc
1. Update the ACME.conf file as necessary. Note that the ISSUER_DOMAIN and ISSUER_EMAIL do not have default values and are required to be filled out.
Copy the ACME.conf into a config directory located in the CHROOT path. i.e /var/www/etc
1. Update the ACME.conf file as necessary. Note that the ISSUER_DOMAIN and ISSUER_EMAIL do not have default values and are required to be filled out.
Copy the AMCE_helper.conf into a config directory located in the CHROOT path. i.e. /var/www/etc
1. Set the CA_NAME to be the name to initialize PCA with (or the name that PCA was initialized with, if done in another step.)
2. Optional: If initialized PCA here run the PCA init command. Make sure to initialize from the CHROOT path.
  1. Initialize the PCA environment using the CA_NAME that was described above.
  2. Create a certificate chain
  3. Create a CRL file (nothing will be in it yet)
  4. Export an html directory with those files in them.
  5. Extract that html export into a web root under a /ca directory. (see example below)
Verify that everything is working by connecting to https://ISSUER_DOMAIN/acme/directory. This will return a directory listing of ACME functions and confirm the script is installed.
Setup the ACME_cleaner.sh script to run daily.
Create a script and set it to run daily that will create a new CRL, HTML export and deploy that export.

PCA_ROOT=/etc/pca chroot /var/www pca <ca_name> init
PCA_ROOT=/etc/pca chroot /var/www pca <ca_name> create chain
PCA_ROOT=/etc/pca chroot /var/www pca <ca_name> create crl
PCA_Root=/etc/pca chroot /var/www pca <ca_name> export html -file /tmp/ca_html.tgz -overwrite
tar -C /var/www/html -zxf /var/www/tmp/ca_hml.tgz

example initializing PCA in CHROOT path

Clients

Before you begin selecting your favorite ACME client to pull a certificate, you will need to download and install PCA root certificate.

Browse to the https://ISSUER_DOMAIN/ca/ site. Accept any certificate errors for "unknown authority".
Download the Certificate Chain. It should be in a PEM format.
Install that chain into the trusted certificate store on the client. (The process will vary depending on client)

Now use your preferred ACME client to request and download a certificate.

Conclusions

This is the first step in gaining independence from public certificate authorities.

Notes on Roundcube Mail

2026-07-08T15:15:01Z

These are some notes for my future self on how to get data out of the roundcube mail database. This assume sqlite3 database but the SQL should also work with MySQL or PostgreSQL databases as well.

View schema

sqlite3 rc_sqlite.db ".schema"

cli for showing roundcube database schema

List users

sqlite3 rc_sqlite.db "select user_id,username from users;"

cli for listing roundcube users from database

Export contacts in vcard format

sqlite3 rc_sqlite.db "select vcard from contacts where user_id = NN;" > username_contacts.vcf

# where the NN is the user_id number for the list users output

cli for extracting vcard contact information

Be your own CA...

2026-04-01T13:42:03Z

I have been running my own Certificate Authority (CA) for my systems for some time now. While running my CA is nothing like running a large commercial CA, running a personal CA is surprisingly easy.

Background

When I first got started with becoming my own CA, it was in the early 2000's and the tools out there were nonexistent. There was basically just openssl and the command line.

This was well before things like Lets Encrypt existed and the only place to get certificates was to pay for them. I recognize the landscape is different now (in 2026) but some of my reasons are even more important now than they were then.

My main reason for running my own CA was I didn't want to be forced to trust an external authority for my systems. Have you looked at the list of Certificate Authorities you browser and/or OSes trust by default? Fully trust. By default. Have a look, read through the names. I'm sure that 95% of these CAs you have never heard of, yet they are fully trusted. Any certificate issue by them for ANY domain will be fully trusted. I had a problem with this.

My original goal was to be able to create a certificate authority and be able to sign certificate requests (CSR) using a minimum of applications. Fortunately, openssl was extremely versatile, if poorly documented.

In order to preform all the CA related functions I started out with a set of Makefiles. These files would do almost everything, but were getting difficult to maintain as I added more features. Eventually I created a script to do the work. The script is a simple shell script and should run anywhere. This script can be found here.

Setup

Download the script from my github.
Copy to any directory in the path.

That's it! The script is self contained. By default it will create a $HOME/.pca directory. This can be overridden by specifying PCA_ROOT=/some/path in your environment.

NOTE: A quick note about security. This program predates the availability and use of hardware security modules. It assumes that the filesystem that the CA files are stored in is secure. Maybe a future improvement will be to add the ability to use hardware security module capabilities.

Initialize CA

To start, initialize the CA. This will setup structures and configs openssl will need to become a CA.

pca CANAME init

The setup will only need to be done once per CA. If you would like to have a second CA... just change CANAME to anything else... maybe 2ndCA... and you have a second CA.

To view all the commands and options, just run the pca command.

$ pca
pca 1.6 - Personal Certificate Authroity
pca <ca> <command> [options]
Commands:
  <ca> init
  <ca> config list <mac[ros] | pol[icy] | ext[ension] | prog[ram]>
  <ca> config get macro <-key key> <-value>
  <ca> config set macro [-a|-d] <-key key> <-value val>
  <ca> config get pol|ext|prog <-name name> <-key key> <-value>
  <ca> config set pol|ext|prog [-a|-d] <-name name> <-key key> <-value val>
  <ca> config sign <-name name>
  <ca> create root [-days #] [-bits #] [-cn name] [-ct x] [-sp x] [-co x] [-or x] [-ou x] [-em x] [ -san val ... ]
  <ca> create crl
  <ca> create key <-name name> [-bits #]
  <ca> create chain
  <ca> create req[uest] <-name name> <-key file | -newkey [#bits]> [-days #] [-cn name] [-co x] [-sp x] [-ct x] [-or x] [-ou x] [-em x] [ -san val ... ]
  <ca> show crl [-issuer] [-hash] [-fingerprint] [-num] [-last] [-next]
  <ca> show req [-name name]
  <ca> show root [-serial] [-expire] [-subject] [-subjecthash] [-issuer] [-issuerhash] [-fingerprint]
  <ca> show cert [-name name] [-serial] [-expire] [-subject] [-subjecthash] [-issuer] [-issuerhash] [-fingerprint] [-client] [-ca]
  <ca> sign <-name name> [-sign] [-days] [-san val ...] [-ext alt_extension]
  <ca> resign <-name name> [-sign] [-days]
  <ca> revoke <-name name | -serial #> [-crl]
  <ca> delete req|key|cert <-name name>
  <ca> export pkcs12 <-name name> <-file name> <-pass -|file:filename|password> [-chain] [-overwrite]
  <ca> export html <-file name> [-overwrite]
  <ca> export cert <-name name> <-file name> [-overwrite] [-chain]
  <ca> import pkcs12 <-name name> <-file name> <-pass -|file:filename|password> [-overwrite]
  <ca> import request <-name name> <-file name> [-overwrite]

Basics Uses

Below are the basics of creating and maintaining a CA.

Create a CSR

To create a CSR and associated key, run the create req command.

pca CANAME create req -name CERT1 -newkey -cn CERT1 -san DNS=CERT1

This will create a CSR and KEY setting the subject to CN=CERT1 and the Subject Alternative Name (SAN) to CERT1.

Show unsigned CSRs

To show all unsigned CSRs run the show req command.

pca CANAME show req

Sign a CSR

To sign a CSR run the sign command.

pca CANAME sign -name CERT1

This will sign the CSR and create the certificate.

Show signed Certificates

To show signed certificates run the show cert command. It's important to note that unless a certificate is deleted, it will continue to show in this list regardless of its state (valid, expired or revoked)

pca CANAME show cert

Revoke a certificate

If a certificate needs to be revoked because it expired or for any other reason, use the revoke command.

pca CANAME revoke -name CERT1

Resign a certificate

It may not be desirable to just revoke a certificate and create a new request and then sign that request. That is a lot of steps! Fortunately for everyone I got tired of that dance and created a resign command to resign an existing certificate and give it a new expiration date.

pca CANAME resign -name CERT1

Import a CSR

There may come times when a CSR is created outside of the PCA program. In those cases, the CSR can be imported and then signed. Use the import request command.

pca CANAME import request -name EXTCERT2 -file EXTCERT2.csr

Now the certificate can be signed, resigned, revoked, etc like any other certificate/request.

Export certificate and key

Now that we can create a certificate (and key) we need a way to export them from the CA so they can be used on a device, server, etc. To do this the standard was of exporting a certificates and keys securely is to use a PKCS12 file. To do this use the export pkcs12 command.

pca CANAME export pkcs12 -name CERT1 -file CERT1.p12 -pass a_better_password_should_be_choosen -chain

This will create a CERT1.p12 file with the certificate, key and certificate chain. These can be transferred to any system and exported using the openssl pkcs12 commands.

Export the certificate

openssl pkcs12 -in CERT1.p12 -nokeys -clcert -out CERT1.crt

Export the key

openssl pkcs12 -in CERT1.p12 -nocerts -nodes -out CERT1.key

Export the chain

openssl pkcs12 -in CERT1.p12 -nokeys -cacert -out CANAME_chain.pem

Depending on what system the PKCS12 files was created and what system the cert/key/chain is being extracted to the -legacy flag may need to be specified.

Export Certificate (only)

In cases where the CSR was imported or where only the certificate needs to be exported, the PKCS12 format may not be necessary. To export only the certificate use the export cert command.

pca CANAME export cert -name EXTCERT2 -file EXTCERT2.crt

Advanced Uses

Openssl uses various files and configurations to run the CA. The init command above created the defaults of these files. But for reasons of your own, you may need to update/change these default files. There are 4 types of files that get used: Macros, Policy, Extension, and Program files. To view the defaults for these files run the config list command.

pca CANAME config list prog

The files that you will probable change the most often are the policy and extension files as these control how openssl behaves based on different types of inputs.

To make a change to one of the files use the config set command.

pca CANAME config set pol -name POLICY_NAME -key KEY_NAME -value VALUE

If a key needs to be added or deleted use the -a or -d flags when calling the set command.

Conclusion

Hopefully this article helps you consider running your own certificate authority and shows you how easy it can be.

Feel free to provide me any comments, questions, or feedback.

Notes on Doveadm

2026-01-09T14:18:53Z

Some quick notes for myself about using doveadm and managing mailboxes. It should be no surprise, that if you are an admin with a server, you can read any email/files on that server.

List users

doveadm user '*'

List mailboxes for user

doveadmm mailbox list -u <user>

Show messages in trash for user older than 30 days

doveadm search -u <user> mailbox Trash NOT SINCE 30days

Delete messages in trash for all users older than 30 days

doveadm expunge -A mailbox Trash NOT SINCE 30days

Delete messages from users mailbox older than 6 months and not flagged

doveadm expunge -u <user> mailbox <mailbox> NOT SINCE 26weeks NOT FLAGGED

Output and users email

doveadm fetch -u <user> "text" MAILBOX <mailbox> UID <email_uid>

Fix index on a mailbox

doveadm index -u <user> <mailbox>

Sync users mailboxes with backup server

doveadm -v sync -u <user> -d

Self Hosted: Episode 8 - External Services

2025-10-25T05:00:30Z

Software Used:

OpenBSD 7.7

Overview

Now that we have some services in our self hosted environment, we may want to allow access from outside our environment. The question is how do we allow this connectivity for ourselves (and anyone else we choose) but not allow the world at large?

To accomplish this we will setup a proxy server. But this will not be a standard proxy server. Our server will be setup to mutually authenticate incoming requests to the services we extend. More on the mutual authentication later.

First we will need to setup a proxy host that will accept requests from the Internet and tunnel them to our self hosted firewall. Here are some of the services we will want to have access to from the Internet.

Email access (both sending and receiving)
Web access to web based email client (proxied)
IMAPS and SMTPS to the email server (proxied)

Setup VM with external provider

There are many providers on the Internet that will provide virtual machines. For this article I will use Vultr (referral link attached). Your are free to use whatever hosting provider you deem best.

Since this VM will be hosted directly on the Internet it will be exposed to all kinds of attacks. My preference for an operating system for used in hostel environments is OpenBSD. Fortunately, Vultr has an option to deploy an OpenBSD VM directly from their console.

I cannot possibly get into tall the different nuances of hosting providers. The basic things to look for is:

a) A variety of build options. This setup will not require huge amounts of CPU or memory. Recommendation is to get at least 1G (preferable 2G of memory). Any CPU speed is fine.

b) Bandwidth. Are there caps. Do you pay for inbound and outbound usage? While email and general apps do not require lots of bandwidth, if you someday extend this self-hosted environment to host music or movies for yourself while you're on the road, bandwidth will spike.

c) ability to set the PTR record. This is possibly the number one requirement when hosting email.

After creating the VM and logging in for the first time there are several things we will want to change.

Create the local user

As with all the setups, I assume that there is a non-root user called 'alfred'. Good chance there is not a default user in the pre-canned OS install from the provider. If your build falls into this bucket create the alfred user.

useradd -g =uid -G wheel -k /etc/skel -s /bin/ksh -m alfred

create the 'alfred' user

Set Root and Local password

If you have logged in as a non-root user, change the users local password. Make it something that cannot be guessed. (We will disable password login via SSH, but the local password still needs to be a strong password.

$ passwd
Changin the password for alfred
New Password: <This should be something really hard to guess!>
Retype new password:
passwd: password updated successfully
$ su -
Password:<enter root password here>
# passwd root
Changin the password for root
New Password: This should be something really hard to guess!
Retype new password:
passwd: password updated successfully
#

password change example

Change the default banner. Blank it out, place a warning, whatever makes sense.

cat <EOF>/etc/banner
This is a private system.
Go away!
EOF

script to update banner file.

SSH Daemon update

The following commands will set the banner sshd uses, change the port it listens on, deny root login, and disable password authentication(optional).

sed -i 's/^#?Banner.*/Banner /etc/banner/' /etc/ssh/sshd_config
sed -i 's/^#?Port.*/Port 3721/' /etc/ssh/sshd_config
sed -i 's/^#?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/^#?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config

Once the changes are made restart sshd. Note, do not exit your current SSH connection until you've tested it.

rcctl restart sshd

From another shell test the connection on a new port.

ssh -p 3721 alfred@newserver.example.org

If the SSH connection doesn't work, check the config, fix and restart sshd again.

Setup the firewall

Event though this device will be locked down and only run services we want, setting up the firewall to make sure that only the ports we want running are accessible is good practice.

ext=vio0
tun_port=5921

set skip on lo
set block-policy drop
set loginterface $ext
set syncookies adaptive ( start 25%, end 12% )
set limit states 400000
set limit table-entries 2000000

table <whitelist> persist file "/var/db/pf.whitelist"
table <banned> persist file "/var/db/pf.banned"
table <aggressive> persist file "/var/db/pf.aggressive"
table <sshguard> persist

block return
pass in quick on egress inet6 proto icmp6 all icmp6-type { routeradv neighbrsol neighbradv }
pass in quick on egress inet6 proto udp \
        from fe80::/10 port dhcpv6-server \
        to fe80::/10 port dhcpv6-client no state
pass in  quick on egress inet6 proto icmp6 from fe80::/10 to fe80::/10
pass in  quick on egress inet6 proto icmp6 from ff02::/16 to fe80::/10
pass out quick on egress inet6 proto icmp6 from fe80::/10 to ff02::/16
  
# allow traffic to/from wg
pass in on wg
pass out on wg nat-to (wg)

# allow ping
pass in  quick on egress inet6 proto icmp6 all icmp6-type echoreq
pass in  quick on egress inet  proto icmp all icmp-type echoreq

# pass in tunnel port
pass in proto udp to egress port $tun_port

# pass in ssh (remember port was changed to 3721)
pass in proto tcp to egress port 3721

# pass in http and https
pass in proto tcp to egress port { 80 443 }

# pass in email ports
pass in proto tcp to egress port { 25 465 }

# pass in imap ports
pass in proto tcp to egress port { 143 993 }

# allow us to send traffic out
pass out from self

/etc/pf.conf

Update Hosts file

We will need to connect to the real IP's of the internal hosts. To do this we will override the hostnames by placing the real IPs into the hosts file.

127.0.0.1 newserver.example.org
::1       newserver.example.org
192.168.30.14 sso sso.example.org
192.168.30.15 mail mail.example.org
192.168.30.11 certauth certauth.example.org

/etc/hosts

We will also need to tell the OS resolver to look at the hosts file for resolution. To do this add this like to the /etc/resolv.conf.

lookup file bind

partial /etc/resolv.conf

Update DNS with server name

Update the external DNS provider with the name and IP of this server. We will also need to add the following records.

example.org            A   add.server.ip.address
newserver.example.org  A   add.server.ip.address
mail.example.org   CNAME   newserver.example.org
smtp.example.org   CNAME   newserver.example.org
imap.example.org   CNAME   newserver.example.org
sso.example.org    CNAME   newserver.example.org
certauth.example.org CNAME newserver.example.org
autoconfig.example.org  CNAME newserver.example.org

Get certificate

This server will have many aliases. Since it will be proxying traffic for all internal systems we choose to expose. Since this will be public facing server we may want a "real" public certificate. We will also need an internal certificate for communicating with the rest of the self-hosted environment. acme-client(1) can be used to get both certificates.

# basic HTTP config for certificate generation
server "*" {
        listen on * port 80
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location * {
                block return 302 "https://$HTTP_HOST$REQUEST_URI"
        }
}

Next we will configure the acme-client.conf file to get the certificate from Lets Encrypt.

#
authority letsencrypt {
  api url "https://acme-v02.api.letsencrypt.org/directory"
  account key "/etc/acme/letsencrypt-privkey.pem"
}

domain newserver.example.org {
  alternative names {
    mail.example.org
    smtp.example.org
    imap.example.org
    sso.example.org
    certauth.example.org
    autoconfig.example.org
    example.org
  }
  domain key "/etc/ssl/private/newserver.key"
  domain full chain certificate "/etc/ssl/newserver.fullchain.pem"
  sign with letsencrypt
}

/etc/acme-client.conf

We will also need an internal certificate to mutually authenticate clients. We will do they by creating a second acme config file.

#
# example.org acme-internal.conf
#
authority example.org {
        api url "https://certauth.example.org/acme/acme/directory"
        account key "/etc/acme/example.org-privkey.pem"
}

domain newserver.example.org {
        domain key "/etc/ssl/private/newserver_internal.key"
        domain full chain certificate "/etc/ssl/newserver_internal.pem"
        sign with example.org
}

/etc/acme-internal.conf

Now start httpd and retrieve the certificates.

rcctl enable httpd
rcctl start httpd
acme-client -v newserver.example.org
acme-client -v -f /etc/acme-internal.conf newserver.example.org

command to start httpd and issue new certificates

We will need to periodically update the certificates. Update crontab with the following lines. Use the 'crontab -e' command to edit the crontab file. This will run the check/update every Sunday at a random time after midnight.

0 0 * * 0  sleep $((RANDOM \% 2048)) && acme-client newserver.example.org
0 0 * * 0  sleep $((RANDOM \% 2048)) && acme-client -f /etc/acme-internal.conf newserver.example.org

crontab to update the certificates as needed

Create Tunnel from firewall to newserver

To create a tunnel we will need to update the configs on both the newserver as well as the firewall server. We will assume that the firewall server does not have direct access to the Internet (because if it does, the article to this point is unnecessary). Due to the firewall having to pass through one or more NAT's we will setup the firewall to initiate the connection and the newserver to just listen for connections.

We will be configuring these systems in parallel as we will need information for each server to complete the configuration on the other.

First on both systems we will create a private key.

openssl rand -base64 32 > /etc/wg.pkey
chmod 640 /etc/wg.pkey

create and save wireguard private key.

New interfaces config

Now create a new /etc/hostname.wg0 file.

wgkey `cat /etc/wg.pkey`
wgport 5921
#wgpeer ### some public key ### wgdescr firewall wgaip 0.0.0.0/0
inet 192.0.2.1 255.255.255.0
!route add -net 192.168.30.0/24 192.0.2.10

newserver /etc/hostname.wg0

Do the same thing on the firewall

wgkey `cat /etc/wg.pkey`
# wgpeer ### some public key### wgendpoint newser.real.ip.address 5921 wgdescr wall wgaip 0.0.0.0/0
inet 192.0.2.10 255.255.255.0

firewall /etc/hostname.wg0

Now load the private keys. This will show us public keys.

/bin/sh /etc/netstart wg0

load the new wireguard interface config

Once the interfaces are loaded get the public key.

ifconfig wg0 | wgpubkey

command to get the public key

Now replace the ###some public key### part of the configs with the opposite public key (also remove the leading # sign). i.e. newserver public key goes into the firewall /etc/hostname.wg0 config file and visa versa. Once the configs are updated reload the configs. Test pinging from firewall server to the newserver.

firewall# ping 192.0.2.1
PING 192.0.2.1 (192.0.2.1): 56 data bytes
64 bytes from 192.0.2.1: icmp_seq=0 ttl=255 time=28.060 ms
64 bytes from 192.0.2.1: icmp_seq=1 ttl=255 time=29.061 ms
64 bytes from 192.0.2.1: icmp_seq=2 ttl=255 time=27.577 ms
^C
--- 192.0.2.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 27.577/28.232/29.061/0.618 ms

example of ping

Once the ping is successful from firewall to newserver, the reverse should also work. Since the tunnel will need to be established from the firewall to newserver we will use the ifstated(8) process to keep the tunnel active. This config will ping the new server over the tunnel every 60 seconds. Adjust as necessary if 60 seconds isn't enough to keep your connection alive.

#
# Keep the tunnel alive
#
chk_peer = '("ping -q -c 5 -w 1 192.0.2.1 >/dev/null 2>&1" every 60)'

init-state tunnel_down

state tunnel_up {
        init {
                run "logger -st ifstated 'state: UP'"
        }
        if ! $chk_peer {
                run "logger -st ifstated 'tunnel DOWN'"
                set-state tunnel_down
        }
}
state tunnel_down {
        init {
                run "logger -st ifstated 'state: DOWN'"
        }
        if $chk_peer {
                run "logger -st ifstated 'tunnel UP'"
                set-state tunnel_up
        }
}

/etc/ifstated.conf

Enable and start ifstated

rcctl enable ifstated
rcctl start ifstated

Setup relay service

Now that the tunnel is up we can setup the relay service. This will accept connection (on specified ports), perform TLS authentication of the client, and then forward to the internal servers. To start this, create the /etc/relayd.conf file.

ext_ip="newserver.ip.addr.ess"
ext_ip6="newserver:ipv6:ip::address"

sso_int="192.168.30.14"
mail_int="192.168.30.15"
certauth_int="192.168.30.11"

table <sso_table> { $sso_int }
table <mail_table> { $mail_int }
table <certauth_table> { $certauth_int }

# TLS PROXY for Internal services
tcp protocol tls_proxy {
        tcp nodelay
        tcp sack
        # validate the clients against internal CA
        tls client ca "/etc/ssl/example.org.pem"
        tls keypair "newserver_internal"
}
# HTTPS PROXY for Internal services
http protocol http_proxy {
        return error
        match header set "X-Forwarded-For" value "$REMOTE_ADDR"
        match header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
        match header set "Keep-Alive" value "$TIMEOUT"
        # match based on the Host header and set the table
        pass request header "Host" value "sso.example.org" forward to <sso_table>
        pass request header "Host" value "mail.example.org" forward to <mail_table>
        pass request header "Host" value "certauth.oexample.org" forward to <certauth_table>
        # only trust devices that have our cert
        tls client ca "/etc/ssl/example.org.pem"
        tls keypair wall
}
# setup the relays
relay proxy {
        listen on $ext_ip port 443 tls
        protocol http_proxy
        forward with tls to <sso_table> port https check icmp
        forward with tls to <mail_table> port https check icmp
        forward with tls to <certauth_table> port https check icmp
}
# remove if IPv6 is not used
relay proxy2 {
        listen on $ext_ip6 port 443 tls
        protocol http_proxy
        forward with tls to <sso_table> port https check icmp
        forward with tls to <mail_table> port https check icmp
        forward with tls to <certauth_table> port https check icmp
}
# setup mail and imap relays
relay mail_proxy {
        listen on $ext_ip port 465 tls
        protocol tls_proxy
        forward with tls to <mail_table> port 465 check icmp
}
relay imap_proxy {
        listen on $ext_ip port 993 tls
        protocol tls_proxy
        forward with tls to <mail_table> port 993 check icmp
}
# remove if IPv6 is not used
relay mail_proxy2 {
        listen on $ext_ip6 port 465 tls
        protocol tls_proxy
        forward with tls to <mail_table> port 465 check icmp
}
relay imap_proxy2 {
        listen on $ext_ip6 port 993 tls
        protocol tls_proxy
        forward with tls to <mail_table> port 993 check icmp
}

/etc/relayd.conf

Enable and start relayd(8)

rcctl enable relayd
rcctl start relayd

enable and start relayd

Now create a user/email certificate from our certauth server as described in Episode 4 - Host/Email certificates. Connect to the web email interface from a device that is not on the internal network. The web browser should prompt use to for a user defined certificate. If you ignore the browser request and continue the server should abort the connection. Try again, but this time provide the browser the certificate created and confirm that the web interface is now presented.

Setup MAIL server

Finally we stage where we will configure our ability to send and receive external email. As was discussed in Episode 7 we will need to setup SPF, DKIM and DMARC records. This will tell the world about our valid email service and provide a level of trust. While it's not guaranteed that our email will always be accepted, it goes a long way to indicate that our email server is run in a responsible fashion.

Add DNS records

As we did when we setup the mail server we will need to setup some external DNS records. These records will be used by external mail providers to make sure that our mail server is legitimate.

The first record is the MX record. This will tell the world where to send email to. The @ sign indicated the our root domain. Depending on the DNS provider that part may be able to be left blank.

@    IN   MX    10  newserver.example.org.

MX DNS record

SPF is the next record we need. Here we indicate what servers mail will come from for our domain. Since we will be receiving and sending mail from a single server we can just setup the SPF record to indicate the use of the MX values and to deny all else.

@   IN   TXT   "v=spf1 mx -all"

SPF DNS record

The DMARC record tells the world what we suggest be done when receiving email that does not comport to our DKIM and SPF settings

_dmarc  IN  TXT    "v=DMARC1;p=quarantine;pct=100;rua=mailto:dmarcreports@example.org;ruf=mailto:dmarcreports@example.org;"

DMARC DNS record

Finally we will create the DKIM record. This will be based on the private key we created on the mail server. On the mail server we can use the below command to generate the DNS record.

openssl rsa -in /etc/mail/dkim/example.org_priv.rsa.key -pubout 2>/dev/null | sed '1s/.*/v=DKIM1;p=/;:nl;${s/-----.*//;q;};N;s/\n//g;b nl;'

command to generate DKIM TXT record

The output should look something like this

v=DKIM1;p=MIIBIjANBgkqhki_LOTS_OF_MORE_BASE64_E1wIDAQAB

example of DKIM public key

Now add that to the DNS record.

selector01._domainkey    IN   TXT   "v=DKIM1;REST_OF_THE_LINE"

DKIM DNS record

Create SMTP config files

This smtpd.conf file will be a little different that the previous one we built for mail. This file will accept mail from any location, filter that mail based on some well known spamming techniques and then forward it onto the mail.example.org server in our self hosted environment.

#
# mail relay config
#
# 1) server will receive any mail destin for <domains> table and
#   forward to internal relay
# 2) server will receive mail from internal server for any and
#   relay
#
# DKIM signing will occur on the internal server
smtp max-message-size "100M"
ca  local_ca cert "/etc/ssl/example.org_root_ca.pem"
pki mail_cert cert "/etc/ssl/wall.fullchain.pem"
pki mail_cert key  "/etc/ssl/private/wall.key"
pki int_cert cert "/etc/ssl/wall_internal.pem"
pki int_cert key  "/etc/ssl/private/wall_internal.key"

table   aliases         file:/etc/mail/aliases
table   domains         file:/etc/mail/domains
table   bad-from        file:/etc/mail/blacklist
table   bad-to          file:/etc/mail/spamtrap
table   relayaccts      file:/etc/mail/relayaccts
table   relays          file:/etc/mail/relays
table   creds           file:/etc/mail/credentials

#check for dynamic dns sources
filter check_dyndns phase connect match rdns regex { \
  '.*\.dyn\..*', \
  '.*\.dsl\..*', \
  '.*dynamic.*', \
  '*\.vodafonedsl\.*', \
  '*\.vodafone\.*' \
  } disconnect "550 no residential connections"

# check for numeric IPv4 sources
filter check_ip_names phase connect match rdns regex { \
  '[0-9]{1,3}[-\.][0-9]{1,3}[-\.][0-9]{1,3}[-\.]]0-9]{1,3}[-\.].*' \
  } disconnect "550 no residential connections"

# confirm rdns is valid
filter check_rdns phase connect match !rdns \
  disconnect "550 no rDNS"

# confirm fcrdns is valid
filter check_fcrdns phase connect match !fcrdns \
  disconnect "550 no FCrDNS"

# check to see if mail is from a known bad sender
filter check_bad_from phase mail-from match mail-from regex <bad-from> \
  disconnect "550 go away spammer"

# check to see if mail is going to a bad receptient
filter check_bad_to phase rcpt-to match rcpt-to regex <bad-to> \
  disconnect "550 go away spammer"

# combine filters into a list
filter initial_checks chain { check_rdns, check_fcrdns, check_ip_names, check_dyndns, check_bad_from, check_bad_to }

# listen for incoming email and apply the filter list
listen on all tls pki mail_cert filter initial_checks
# listen for relayd mail on the tunnel interface
listen on wg port 9465 smtps pki int_cert auth <relayaccts> tag RELAY

action "outbound" relay
action "inbound" relay host smtps://ponyexpress@mail.example.org:9465 tls auth <creds>
# local mail, send inbound
match for local action "inbound"
# anything marked as relay, send outbound
match tag RELAY from src <relays> for any action "outbound"
# anything for one of our domains send inbound.
match from any for domain <domains> action "inbound"

# default deny

/etc/mail/smtpd.conf

# domains
#
# list domains to accept mail for
# to update smtpd use 'smtpctl update table domains'
#
example.org

/etc/mail/domains

#
# place sender names of domains (or whole domains here)
# to deny their acceptance.
#
#somename@somedom.ain

/etc/mail/blacklist

#
# List of servers that we will accpet relayed mail from
# use 'smtpctl update table relays' to update table
192.168.30.15

/etc/mail/relays

# spamtrap
#
# place email names of accounts that should never 
# receive email.
#spamtrap@example.org

/etc/mail/spamtrap

#
# relay credentials
# use 'smtpctl update table creds' to update
# relay user
ponyexpress     ponyexpress:MAKE_THIS_A_GOOD_PASSWORD

/etc/mail/credentials

#
# user names to be used for relays
#
# use 'smtpctl update table relayusers' to reload file
# use 'smtpctl encrypt "password"' to encrypt password
ponyexpress     $2b$08$19n8/YAQQxBxOVApmDxmHuQVqtbMZNSADb6LJaGvRW1NdW1oOO0y.

/etc/mail/relayaccts

With all those files in place check that smtpd(8) is happy then restart if successful.

smtpd -nvf /etc/mail/smtpd.conf
# if successful restart
rcctl restart smtpd

commands to check validity of smtpd.conf and associated files

Update forwarding on internal mail server

Now we can receive external email, we need to revisit the mail.example.org server to have it be able to forward email via newserver. On mail server un-comment the lines that were left commented.

table relayaccts        file:/etc/mail/relayaccts
table creds             file:/etc/mail/credentials

listen on 0.0.0.0 port 9465 smtps pki mail_cert auth <relayaccts> tag RELAY

action "relay" relay host smtps://ponyexpress@newserver.example.org:9465 tls auth <creds> helo "mail.example.org"

match from auth for any action "relay"

lines in smtpd.conf to uncomment

Be sure to add the relayaccts and credentials files. These can be mirrors of the ones on newserver.

#
# relay credentials
# use 'smtpctl update table creds' to update
# relay user
ponyexpress     ponyexpress:MAKE_THIS_A_GOOD_PASSWORD

/etc/mail/credentials

#
# user names to be used for relays
#
# use 'smtpctl update table relayusers' to reload file
# use 'smtpctl encrypt "password"' to encrypt password
ponyexpress     $2b$08$19n8_ENCRYPTED_PASSWORD_W1NdW1oOO0y.

/etc/mail/relayaccts

Now check that the updates are valid and then restart the service.

smtpd -nvf /etc/mail/smtpd.conf

rcctl restart smtpd

Sending our first emails

Now that everything is in place, send your first email. Send it to an alternate account you have. If you receive it, respond to make sure that inbound access is working. If you run into problems, review the /var/log/maillog files on both newserver and mail servers for indications as to what failed.

Conclusion

Now that we have email access to and from our self hosted environment to the rest of the world... you need to decide what is next! Good Luck!

Self Hosted: Episode 7 - Email

2025-10-18T05:00:48Z

Software Used:

OpenBSD 7.7 (as VM)
OpenSMTPD (part of OS)
Dovecot 2.3.21.1 (from ports)
RoundcubeMail 1.6.11 (from ports)
twofactor_webauthn 1.3.0

Overview

Email is a surprisingly vast and charged topic. It is one of the first services that someone/anyone can deploy on the Internet and begin communicating. However, email has gotten this reputation that it is "hard". I disagree. Just about anything can be messed up if you don't understand what you are doing. Email is no different.

I hope to dispel the myth of email being "hard" and provide a guide that will get you started on the right foot. I will touch on some topics only lightly and try to provide some links where you can get more information.

A little background

Before we begin creating services and sending email it is always good to understand the protocols and interactions that are involved.

To begin, email is transmitted over the internet using SMTP. Email access from an client can occur over several protocols. The protocol we will use to access email is called IMAP. When sending email to others, and not be mistaken for SPAM, the email needs to be seen as legitimate. In order to prove the legitimacy of email being sent a couple of protocols a are used: SPF and DKIM. SPF allows a receiving email server to verify that the sending server is allowed to send email for that domain. DKIM provides a cryptographic way that the email being received actually originated under to control of that domain. DKIM and SPF are wrapped up into an additional protocol called DMARC. DMARC is a way for a domain owner to publish how receiving servers should treat email that does not pass the SPF and DKIM tests. There is no way to force the receiving servers to act per the DMARC policy but DMARC is a way to signal to the Internet that servers falsely claiming to operate under a domain should be suspect or better yet ignored.

SPF, DKIM and DMARC are all controlled using DNS records. In addition to the above protocols simply publishing proper A and PTR records for the email servers will also help in proving legitimacy. In fact in the next episode when we deploy an external email forwarder, we will use the fact that improper A and PTR records are indicators of illegitimate mail servers.

With the background out of the way and to keep the things simple, I have broken down this guide into 4 parts. There is 1) creating the VM, 2) the mail storage, 3) the mail exchanger, and 4) mail access sections.

Create the mail VM...

First this we need to do is create the VM. Similar to building the firewall.

vm create -t openbsd77 -s 50G -m 4G -c 2 mail

Next add the DMZ network to the VM.

vm add -t network -s net-dmz mail

The image is was already downloaded so we can skip this step. Now install the OS.

vm install mail install77.iso

and connect to the console

vm console mail

Finish installing the OS and shut it down. When the installation is complete break out of the console using ~. escape. This will take you back to a the hypervisor. From the hypervisor make sure the VM is halted using the 'vm list' command. The state should be "Stopped"

root@hyperv01:~ # vm list
NAME      DATASTORE  LOADER  CPU  MEMORY  VNC  AUTO  STATE
mail      default    uefi    2    4G      -    No    Stopped

Start the VM using the 'vm start mail' command. Next we can install the software.

Mail Storage

There needs to be a place to store the mail that we receive. This is the mail servers job. The mail server will need to do several things.

Authenticate against the LDAP directory.
To be able to "sort" the mail as it comes into the server. This capability is referred to as Sieve.
It will need to store the mail (obviously). But we have options in now we store the mail. We can use single mailboxes (called the mbox) format. Which is about as old a email itself. However there is a better options. If the mail is stored in in Maildir format then each email is kept separate and we have options for folders. This is the options that will be show here.

Install mail server

For the mail server we will use an application called Dovecot.

First install the packages

pkg_add dovecot-- dovecot-ldap-- dovecot-pigeonhole--

Next configure the packages. Dovecot installs a lot of files under the /etc/dovecot directory. We will only need to update some of those files.

First we will enable LDAP access.

# LDAP connectivity setting
uris = ldaps://ldap.example.org:636
uris = ldap://ldap.example.org
dn = cn=ldapadmin,dc=example,dc=org
dnpass = letmein
tls = no
tls_ca_cert_file = /etc/ssl/example.org_root_ca.pem
tls_cert_file = /etc/ssl/mail.fullchain.pem
tls_key_file = /etc/ssl/private/mail.key

debug_level = 0
auth_bind = no

base = dc=example,dc=org
user_attrs = homeDirectory=home,uidNumber=uid,gidNumber=gid
user_filter = (&(objectClass=posixAccount)(|(mail=%u)(uid=%u)))
pass_attrs = uid=user,userPassword=password
pass_filter = (&(objectClass=posixAccount)(mail=%u))
iterate_attrs = mail=user
iterate_filter = (objectClass=posixAccount)
default_pass_scheme = SSHA

/etc/dovecot/dovecot-ldap.conf.ext

#
#
passdb {
        driver = ldap
        args = /etc/dovecot/dovecot-ldap.conf.ext
}

# override all ldap settings
userdb {
        driver = ldap
        args = /etc/dovecot/dovecot-ldap.conf.ext
        # override ldap values
        override_fields = uid=vmail gid=vmail home=/var/mail/vmail/%n
}

/etc/dovecot/conf.d/auth-ldap.conf.ext

#
# for mutual auth
#auth_ssl_require_client_cert = yes
#auth_ssl_username_from_cert = yes

#
auth_mechanisms = plain

# use ldap to auth users
!include auth-ldap.conf.ext

/etc/dovecot/conf.d/10-auth.conf

Enable the server to support TLS connections.

#
ssl = yes
ssl_cert = </etc/ssl/mail.fullchain.pem
ssl_key  = </etc/ssl/private/mail.key
ssl_ca = </etc/ssl/example.org_root_ca.pem
#ssl_key_password =

# only validate certs the we generated
#ssl_client_ca_dir =
ssl_client_ca_file = </etc/ssl/example.org_root_ca.pem
ssl_client_cert = </etc/ssl/mail.fullchain.pem
ssl_client_key = </etc/ssl/private/mail.key

#ssl_cert_username_field = commonName
ssl_client_require_valid_cert = no
ssl_require_crl = no
ssl_verify_client_cert = no

/etc/dovecot/conf.d/10-ssl.conf

Now set the remaining variables into the local.conf file

# local changes to dovecot setting
# This file is processed last and overrides the conf.d/*.conf 
# configs.

# only support imap, lmtp and sieve protocols.  disable pop3
protocols = imap lmtp sieve

# set the mail location
mail_location = maildir:~/Maildir

# set the perms on the auth listener (used by smtpd)
service auth {
  unix_listener auth-userdb {
    mode = 0660
    user = vmail
    group = vmail
  }
}

# change the authworker to run as the same user (not root)
service auth-worker {
  user = $default_internal_user
}

# set the metrics
metric auth_success {
  filter = event=auth_request_finished AND success=yes
}
metric auth_failures {
  filter = event=auth_request_finished AND NOT success=yes
}
metric imap_command {
  filter = event=imap_command_finished
  group_by = cmd_name tagged_reply_state
}
metric smtp_command {
  filter = event=smtp_server_command_finished
  group_by = cmd_name status_code duration:exponential:1:5:10
}
metric mail_delivery {
  filter = event=mail_delivery_finished
  group_by = duration:exponential:1:5:10
}

### conf.d/15-lda.conf
# temp fail if user over quote... do not just bounce email
quota_full_tempfail = yes

# enable SIEVE for pre-sorting email
protocol lda {
  mail_plugins = $mail_plugins sieve
}

### conf.d/15-mailboxes.conf
# add other common mailboxes that are used by different clients and point them to consisent destinations
namespace inbox {
  mailbox Spam {
    special_use = \Junk
  }
  mailbox "Deleted Items" {
    special_use = \Trash
  }
}

### conf.d/20-imap.conf
# add sieve to imap
protocol imap {
  mail_plugins = $mail_plugins imap_sieve
}
### conf.d/20-lmtp.conf
# disable verify quota before replying rcpt to
lmtp_rcpt_check_quota = no

# add sieve to lmtp
protocol lmtp {
  mail_plugins = $mail_plugins sieve
}

# mail process statistics tracking
service stats {
  unix_listener stats-writer {
    user = 
    group = $default_internal_group
    mode = 0660
  }
}

### conf.d/90-plugin.conf
# automate the reporting of SPAM and HAM 
plugin {
  sieve_plugins = sieve_imapsieve sieve_extprograms
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment

# uncomment if spam/ham processing is configured
#  imapsieve_mailbox1_name = Junk
#  imapsieve_mailbox1_causes = COPY
#  imapsieve_mailbox1_before = file:/etc/dovecot/sieve/report-spam.sieve
#  imapsieve_mailbox2_name = *
#  imapsieve_mailbox2_from = Junk
#  imapsieve_mailbox2_causes = COPY
#  imapsieve_mailbox2_before = file:/etc/dovecot/sieve/report-ham.sieve

  sieve_before = /etc/dovecot/sieve/default.sieve
}

### conf.d/90-sieve-extprograms.conf
plugin {
  sieve_pipe_bin_dir = /etc/dovecot/sieve
}

### conf.d/90-sieve.conf
plugin {
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment
  sieve_plugins = sieve_imapsieve sieve_extprograms
}

# enable extended logging to troubleshoot
auth_debug = yes
auth_verbose = yes
verbose_ssl = yes

/etc/dovecot/local.conf

We will add the default.sieve file. This file will be run on all incoming emails.

require ["fileinto"];
# rule:[SPAM]
if header :contains "X-Spam" "yes" {
  fileinto "Junk";
}

if header :contains "X-Spam-Flag" "YES" {
  fileinto "Junk";
}

/etc/dovecot/sieve/default.sieve

We can test the configuration for errors using the doveconf(1) command.

doveconf -n

The output will show non-default config options. Use the '-a' to show all the config options. If there are errors in the config they will be indicated. Fix the errors before continuing.

Finally enable and start the mail storage server

rcctl enable dovecot
rcctl start dovecot

To verify that the server is running and LDAP connection is functioning we can lookup the users by using the doveadm(1) command.

doveadm user '*'

This command will return a list of all valid users in LDAP.

Mail Exchanger

The mail exchanger is the server that is responsible for communicating with other mail exchangers either locally or in the Internet. This device uses the SMTP protocol to exchange emails. We will be using OpenSMTPd as our mail exchanger software. OpenSMTPd is built into the OpenBSD operating system. We will install a few extensions to tie the service into the LDAP server for user accounts.

pkg_add opensmtpd-table-ldap opensmtpd-filter-dkimsign

First configure the LDAP access.

# Connection information
url: ldap://ldap.example.org
username: cn=ldapadmin,dc=example,dc=org
password: letmein
basedn: dc=example,dc=org

# filter for aliases
# used in aliases and virtual tables
# attribute is the destination user or another mailaddress
alias_filter: (&(objectclass=posixaccount)(mail=%s))
alias_attributes: uid

# filter for username
# attributes are username and hashed password
# the account need to be allowed to read the hashed password
credentials_filter: (&(objectclass=posixaccount)(mail=%s))
credentials_attributes: uid,description

## domain table
## used for: match for domain <ldap>
#domain_filter (&(objectclass=dnsdomain)(cn=%s))
#domain_attributes: cn

# user info
# extra user info
userinfo_filter: (&(objectclass=posixaccount)(uid=%s))
userinfo_attributes: uidNumber,gidNumber,homeDirectory

## filter for mailaddr info
## list of mailaddresses for match statements
## attribute will be ignored but is needed for ldap
#mailaddr_filter: (&(objectclass=posixaccount)(mail=%s))
#mailaddr_attributes: mail

## filter for mailaddrmap info
## used for: liste on ... sender <senders>
#mailaddrmap_filter: (&(objectclass=posixaccount)(uid=%s))
#mailaddrmap_attributes: mail

## filter for netaddr
## used for: match from src <ldap>
## not real useful, because match is on IP not CIDR
#netaddr_filter: (&(objectclass=host)(ipaddr=%s))
#netaddr_attributes: ipaddr

/etc/mail/ldap.conf

Next configure the SMTPd configuration file. Here we will specify that LDAP should be used for various table lookups as well as a static file for a list of domains the email server supports. We will leave some of the configuration commented out until we have setup the relay server.

#

smtp max-message-size "100M"
ca  local_ca cert "/etc/ssl/example.org_root_ca.pem"
pki mail_cert cert "/etc/ssl/mail.fullchain.pem"
pki mail_cert key  "/etc/ssl/private/mail.key"

table ldap      ldap:/etc/mail/ldap.conf
table domains   file:/etc/mail/domains
#table relayaccts        file:/etc/mail/relayaccts
#table creds             file:/etc/mail/credentials

# create filter for signing emails for supported domains.
filter dkimsign_rsa proc-exec "filter-dkimsign -d example.org -s selector01 -k /etc/mail/dkim/example.org_priv.rsa.key" user _dkimsign group _dkimsign

# create a filter chain to allow more than one filter.
filter dkimsign chain { dkimsign_rsa  }

listen on 0.0.0.0 port submission tls-require pki mail_cert auth <ldap> filter dkimsign
listen on 0.0.0.0 port smtps smtps pki mail_cert auth <ldap> filter dkimsign
listen on 0.0.0.0 tls pki mail_cert
#listen on 0.0.0.0 port 9465 smtps pki mail_cert auth <relayaccts> tag RELAY

# setup the action to handle local email.
# leave external email commented out for now.
action "local" lmtp "/var/dovecot/lmtp" rcpt-to user "vmail" userbase <ldap> virtual <ldap>
#action "relay" relay host smtps://ponyexpress@newserver.example.org:9465 tls auth <creds> helo "mail.example.org"

# Create a match rule for email that has been received.
match for local action "local"
match for domain <domains> action "local"
match from auth for domain <domains> action "local"
#match from auth for any action "relay"

# by default any email that doesnt match a rule is dropped

/etc/mail/smtpd.conf

Create a file for domains that we will support. This will allow us to add additional email domains as things grow/change.

# domains supported by this mail server
example.org

/etc/mail/domains

DKIM keys are like other cryptographic keys. There is a private key and a public key. To create the private key use the following command.

[[ ! -d /etc/mail/dkim ]] && mkdir -p /etc/mail/dkim
openssl genrsa -out /etc/mail/dkim/example.org_priv.rsa.key 2048

Generate RSA DKIM key

Later we will generate the public key record for deploying to DNS.

Confirm that the configuration are correct. Fix any errors.

smtpd -nvf /etc/mail/smtpd.conf

Check validity of smtpd config

If all configuration changes succeed, restart the smtpd service.

rcctl restart smtpd

Restart smtpd service

Mail Access

Now that we can receive email and store email. We need to be able to access the emails. This can be done in two ways. First will be over the web and second will be using a native email client that can talk directly to the email server. Both options will be covered. However in the second option only the basics will be touched on as each email client is a little different.

Web email client

Perhaps the most widely used web based email client is Roundcube. Do the following to install roundcube and its dependencies.

pkg_add roundcube php-pdo_sqlite8.2 php-curl8.2 php-gd8.2 php-ldap8.2

Also download and extract the webauthn two factor plugin.

ftp -o /tmp/twofactor_webauthn.tar.gz "https://github.com/bartnv/twofactor_webauthn/archive/refs/tags/v1.3.0.tar.gz"
tar -C /var/www/roundcubemail/plugins -xzf /tmp/twofactor_webauthn.tar.gz

Initialize the roundcube database.

sqlite3 -init /var/www/roundcubemail/SQL/sqlite-initial.sql /var/www/roundcubemail/db/sqlite.db

Initialize the main configuration.

<?php
$config = [];
// seth the database location
$config['db_dsnw'] = 'sqlite:///db/sqlite.db?mode=0660';
// set the IMAP host and port used for authentication
$config['imap_host'] = 'mail.example.org:143';
// set the SMTP host.  used for sending emaik
$config['smtp_host'] = 'ssl://mail.example.org:465';
$config['smtp_user'] = '%u';
$config['smtp_pass'] = '%p';
$config['smtp_conn_options'] = [
        'ssl' => [
                'verify_peer'   => false,
                'cafile'        => '/etc/ssl/example.org_root_ca.pem',
                'security_level'=> 4,
        ],
];
$config['support_url'] = '';
// set our name
$config['product_name'] = 'example.orgt Webmail';
// change the des_key to something random.
$config['des_key'] = 'CHANGE_ME_PLEASE!!!';
// enable some useful plugins
$config['plugins'] = [
    'archive',
    'zipdownload',
    'managesieve',
    'markasjunk',
    'jqueryui',
    'twofactor_webauthn',
];
$config['skin'] = 'elastic';
// disable the installer
$config['enable_installer'] = false;
// setup logging of some functions
$config['log_logins'] = true;
$config['log_session'] = true;
$config['smtp_debug'] = false;
$config['imap_debug'] = false;

/var/www/roundcubemail/config/config.inc.php

Configure and enable the httpd server

server "*" {
        listen on * port 80
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location * {
                block return 302 "https://$HTTP_HOST$REQUEST_URI"
        }
}
server "*" {
        log style combined
        connection max request body 102400000
        listen on * tls port 443
        tls {
                certificate "/etc/ssl/mail.fullchain.pem"
                key "/etc/ssl/private/mail.key"
        }
        # used for autodiscovery
        location "/mail/*" {
                root "/htdocs"
                request strip 1
                directory no index
        }
        root "roundcubemail/public_html"
        directory index index.php
        # block direct access to locations that shouldn't be accessable from HTTPD
        location "/*.git" { block }
        location "/*.tx" { block }
        location "/*SQL*" { block }
        location "/*bin*" { block }
        location "/*config*" { block }
        location "/*logs*" { block }
        location "/*temp*" { block }
        location "/*tests*" { block }
        location "/*install*" { block }
        # all other php request sent to FPM socket
        location "*.php*" {
                fastcgi socket "/run/php-fpm.sock"
        }
}

/etc/httpd.conf

Link the PHP libraries.

for f in `ls /etc/php-8.2.sample`; do ln -s /etc/php-8.2.sample/${f} /etc/php-8.2/${f}; done

Add a custom.ini file for some tweaks

; Custom PHP settings
; override the defaults
file_uploads = On
upload_max_filesize = 100M
max_file_uploads = 20
post_max_size = 100M

/etc/php-8.2/custom.ini

Finally enable web email

rcctl enable php82_fpm httpd
rcctl start php82_fpm httpd

To confirm things are working connect to https://mail.example.org/ and login with the LDAP user mail address and password.

Native email client

Similar to the web email client, dedicate email clients use SMTP and IMAP to send and access email.

Establishing trust in our CA

Recall we are using our own certificate authority to secure our servers. We will need to load the CA certificate into the clients / systems so that our certificate authority is trusted. The trusted certificate can be download from https://certauth.example.org/roots.pem. Once downloaded, follow instructions of the device / client to trust the certificate.

Setting up Autodiscover for easy configuration

There are several things we can do to assist the different mail clients to find our services and auto-configure themselves properly.

Add a DNS alias for autoconfig.example.org

autoconfig IN CNAME mail.example.org.
autodiscover IN CNAME mail.example.org.
_autodiscover._tcp IN SRV 0 0 443 mail.example.org.
_imap.tcp IN SRV 10 1 143 mail.example.org.
_imaps.tcp IN SRV 0 1 993 mail.example.org.
_pop3.tcp IN SRV 0 0 0 .
_pop3s.tcp IN SRV 0 0 0 .
_submission._tcp IN SRV 0 1 587 mail.example.org.
_submissions.tcp IN SRV 0 1 465 mail.example.org.

Create an autodiscovery config file that can be fetched from the email server.

<?xml version="1.0" encoding="UTF-8"?>
<clientConfig version="1.1">
      <emailProvider id="example.org">  
         <domain>example.org</domain>
         <incomingServer type="imap">
            <hostname>mail.example.org</hostname>
            <port>993</port>
            <socketType>SSL</socketType>
            <username>%EMAILADDRESS%</username>
            <authentication>password-cleartext</authentication>
         </incomingServer>
         <outgoingServer type="smtp">
            <hostname>mail.example.org</hostname>
            <port>465</port>
            <socketType>SSL</socketType>
            <username>%EMAILADDRESS%</username>
            <authentication>password-cleartext</authentication>  
         </outgoingServer>  
      </emailProvider>
</clientConfig>

Thunderbird client

When the Thunderbird client first starts it will ask to create an account. Normally this is exactly what we want to happen. However, since Thunderbird does not use the system certificate store, we need to add our CA certificate to the trusted store on Thunderbird.

To do this exit the account creation dialogue.

Open the Setting dialogue

Select Privacy & Security and scroll down to find the Manage Certificates button and click it.

In the Certificate Manager dialog select the Authorities tab then Import button.

Select the certificate PEM file that contains the our CA certificate. Then click Open.

Select the check boxes to trust this certificate for web sites and email users. Then click OK button.

With that complete, click on the gear icon again and select Add Mail Account to start the account setup process again.

Enter your username email address and password into the boxes and click Continue. Thunderbird will try to guess your sites settings.

The settings should come back with something similar. However the username does not have the domain. To correct this click the Configure Manually link.

Retype the the email in the username field. Then click the Done button.

Finally you should receive the Account successfully created window. Click Finish button

You should now be able to send and receive emails... to yourself. In the next episode we will build the proxy server that will allow us to send and receive emails with the outside world.

Conclusion

If you made it to here, congratulations! You now have a working email server free from spying and corporate policies that are aggressively against you.

Self Hosted: Episode 6 - Authentication

2025-10-11T05:00:43Z

Software used:

FreeBSD 14.2 (jail)
OpenLDAP 2.6 (from ports)

It has been said: "Three people can keep a secret, if two are dead." Fortunately, it doesn't need to come to that.

Authentication

As with many things in computing there needs to be a source of truth. This is true for user accounts. Where we have one account, we will need more. Good, bad, or indifferent, the defacto way of keeping user and group information is using LDAP.

Update DNS with new address

To begin with we will need to be able to resolve the new server. To do this update the example.org and 30.168.192.in-addr.arpa zone files on the DNS servers respectively.

ldap     IN      A      192.168.30.13

record for example.org zone file

13      IN      PTR      ldap.example.org.

record for the 30.168.192.in-addr.arpa zone file

Don't forget to update the zone serial number and restart the named service.

Setup LDAP server

Just like with the CA server, the LDAP server will be setup using a jail. If you have not already setup the Jail infrastructure review Episode 3 - Setting up FreeBSD Jail. For this jail we will be using the following file.

# LDAP server jail file

ldap {
	# STARTUP / LOGGING
	exec.start = "/bin/sh /etc/rc";
	exec.stop = "/bin/sh /etc/rc.shutdown";
	exec.consolelog = "/var/log/jail_console_${name}.log";

	# PERMISSIONS
	allow.raw_sockets;
	exec.clean;
	mount.devfs;
	allow.reserved_ports = true;

	# HOSTNAME / PATH
	host.hostname = "${name}";
	path = "/usr/local/jails/containers/${name}";

	# NETWORK
	vnet;
	vnet.interface = "${epair}b";

	# NETWORKS/INTERFACES
	$id = "12";
	$ip = "192.168.30.${id}";
	$mask = "255.255.255.0";
	$gateway = "192.168.30.1";
	$bridge = "net-dmz";
	$epair = "epair${id}";

	exec.prestart += "ifconfig ${epair} create up";
	exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
	exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
	exec.start    += "ifconfig ${epair}b ${ip} netmask ${mask} up";
	exec.start    += "route add default ${gateway}";
	exec.poststop += "ifconfig ${bridge} deletem ${epair}a";
	exec.poststop += "ifconfig ${epair}a destroy";
}

Start the LDAP jail

service jail start ldap

Install the LDAP server packages

pkg -j ldap update
pkg -j ldap install -y openldap-server openldap-client

Connect to the new jail

jexec ldap /bin/sh

Edit ldap config file

The LDAP server has a server configuration files called slapd.conf located in the /usr/local/etc/openldap directory. This file consists of several parts defined in the config file. Each part has been commented.

# Import Schemas to support
include         /usr/local/etc/openldap/schema/core.schema
include         /usr/local/etc/openldap/schema/cosine.schema
include         /usr/local/etc/openldap/schema/inetorgperson.schema
include         /usr/local/etc/openldap/schema/nis.schema

# specify paths for restarting
pidfile         /var/run/openldap/slapd.pid
argsfile        /var/run/openldap/slapd.args

# load the DB module for storate
modulepath      /usr/local/libexec/openldap
moduleload      back_mdb

# Set access controls on different keys
# NOTE: rootdn can always read and write EVERYTHING!
# Access Control
access to attrs=userPassword
        by group/groupOfNames/member="ou=services,dc=example,dc=org" read
        by self write
        by * none
access to attrs=description
        by group/groupOfNames/member="ou=services,dc=example,dc=org" read
        by self write
        by * none

# TLS
TLSCACertificateFile /usr/local/share/certs/example.org_root_ca.pem
TLSCertificateFile /usr/local/etc/ssl/ldap_fullchain.pem
TLSCertificateKeyFile /usr/local/etc/ssl/ldap.key
#TLSVerifyClient never

# enable database config definitions
database config
# configure storage DB settings
database        mdb
maxsize         1073741824
suffix          "dc=example,dc=org"
rootdn          "cn=ldapadmin,dc=example,dc=org"
rootpw          change_me_please!
directory       /var/db/openldap-data

# Indices to maintain
index   objectClass     eq
index   cn,sn,mail      pres,eq,approx,sub
index   uid             eq

# enable DB monitoring
database monitor

/usr/local/etc/openldap/slapd.conf

Get LDAP certificate

Get a new certificate from the CA for the LDAP server

Add and trust the root cert

test -d /usr/local/share/certs || mkdir -p /usr/local/share/certs
fetch --no-verify-peer --no-verify-hostname -o /usr/local/share/certs/root_ca.crt https://certauth.example.org/roots.pem
certctl rehash

commands to install and trust the CA certificate

Install the acme package

pkg install -y acme.sh

command to install the acme.sh package

Generate the certificate.

export CRTP=/usr/local/etc/ssl
test -f $CRTP || mkdir -p $CRTP
acme.sh --issue --standalone -d $(hostname) --cert-file $CRTP/$(hostname -s).crt --key-file $CRTP/$(hostname -s).key --fullchain-file $CRTP/$(hostname -s)_fullchain.pem --server https://certauth.example.org/acme/acme/directory

commands to generate new certifiate

Update LDAP server args

To allow the LDAP server to listen on both ldap:/// and ldaps:/// update the service flags

sysrc slapd_flags="-h 'ldapi://%2fvar%2frun%2fopenldap%2fldapi/ ldap:/// ldaps:///'"

update service flags before start

Start the LDAP server

service slapd start

command to start the ldap server

Forward logs to logging server

Create the /usr/local/etc/syslog.d directory if it doesn't exist.

[ -f /usr/local/etc/syslog.d ] || mkdir -p /usr/local/etc/syslog.d

command to create the syslog.d directory

Next, create the syslog config file

# Send log messages to log server
*.*                                          @192.168.30.12

/usr/local/etc/syslog.d/remote.conf

Finally, restart syslogd

service syslogd restart

command to restart syslogd

LDAP Setup

Now that the server is running we need to setup the structure and add some user accounts.

Setup structure

#
# Simple LDAP Schema
#
dn: dc=example,dc=org
objectclass: dcObject
objectclass: organization
dc: example
o: example.org LDAP Server
description: Root entry for example.org

# First level
dn: ou=users,dc=example,dc=org
objectclass: organizationalUnit
ou: users
description: All people in organization

dn: ou=groups,dc=example,dc=org
objectclass: organizationalUnit
ou: groups
description: All groups in organization

dn: ou=domains,dc=example,dc=org
objectclass: organizationalUnit
ou: domains
description: All domains in organization

dn: ou=serviceaccounts,dc=example,dc=org
objectclass: organizationalUnit
ou: serviceaccounts
description: All sevice accounts in organization

init_schema.ldif

Load the ldif data to the openldap server

ldapadd -W -H ldap://127.0.0.1:389 -D "cn=ldapadmin,dc=example,dc=org" -f init_schema.ldif

Add a user account

Now the server is up and running let's add a user account. Updates to the LDAP server are made using LDIF files. Create a LDIF file from the example below with the appropriate user information.

Create a SSHA password

export PASS=letmein
export RND_SALT=`openssl rand -base64 6`
export PASS_HASH=`echo -n "$PASS$RND_SALT" | openssl dgst -sha1 -binary | openssl enc -base64 -A`
export LDAP_PASS_HASH=`(echo -n "$PASS_HASH" | openssl base64 -d -A; echo -n "$RND_SALT";) | openssl enc -base64 -A | awk '{print "{SSHA}"$0 }'`
echo $LDAP_PASS_HASH
unset RND_SALT
unset PASS_HASH
unset LDAP_PASS_HASH
unset PASS

shell to create a SSHA password

Edit the ldif as needed.

#
# Create User alice
#
dn: uid=2e86409ff65ff189,ou=users,dc=example,dc=org
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
uid: 2e86409ff65ff189
cn: alice
sn: user
uidNumber: 5000
gidNumber: 5000
homeDirectory: /home/alice
givenName: alice
displayName: alice user
mail: alice@example.org
mail: @example.org
userPassword:: e1NTSEF9WmdwcCt6ZE90UC9KcEhXNjIxUGd5Wm5MZlV0RGExUjZNVzk2V0E9PQ=
 =
description: $2b$10$JxGxv8v2hGw4/dytUMRjbOlHrzg7wyCRqZX6YtFdAEaruH.OsIvWm

add_user.ldif

Load the user account.

ldapadd -W -H ldap://127.0.0.1:389 -D "cn=ldapadmin,dc=example,dc=org" -f add_user.ldif

Modify a user account

From time to time updates/changes are going to need to be made to a user account. Updates to LDAP also use LDIF files to preform updates. Use the follow script to create an update LDIF.

dn: uid=juser,ou=users,dc=example,dc=org
changetype: modify
replace: userPassword
userPassword: {SSHA}adsfasdfasdfadfasdfadfas
-

update_user.ldif

Load the updated password

ldapmodify -W -H ldap://127.0.0.1:389 -D "cn=ldapadmin,dc=example,dc=org" -f update_user.ldif

Delete a user account

User accounts that are no longer needed should be deleted. Use the following script to help delete an old user account.

uid=juser,ou=users,dc=example,dc=org

delete_user.ldif

Load the delete ldif

ldapdelete -W -H ldap://127.0.0.1:389 -D "cn=ldapadmin,dc=dc=example,dc=org" -f delete_user.ldif

An easier way...

While we can manipulate the LDAP users and attributes by hand I have created a few scripts that will make the daily maintenance of users much easier. These scripts can be fetched from mgraves00/misc-scripts/ldap.

Conclusion

While this seems like a lot of work to setup a self hosted environment for just one person (and it is), once we have the infrastructure setup, adding new self hosted service will be much easier.

Self Hosted: Episode 5 - Logging

2025-10-04T05:00:01Z

Software Used:

FreeBSD 14.2 (jail)

Introduction

If you can't observe what is happening then you can't find and fix what is broken

Setting up a logging server at this point may seem premature. But as we move forward, things will get more complicated and more interdependent. Having a logging server that can handle logs from all services will aide is troubleshooting what is going wrong.

Logging Discussion

What is it we want to get out of the logging server? At this point we just need a place to store logs from other systems. There are many logging servers that have nice graphical interfaces. We may get to self hosting one of those in the future, but for now we want a simple, syslog like logging server. Each log should be timestamped with the time it arrived and the hostname of the server sending the log. If no server hostname is available, we should us the IP address. In addition the syslog server should support TLS encrypted logs.

Update DNS with new address

To begin with we will need to be able to resolve the new log server. To do this update the example.org and 30.168.192.in-addr.arpa zone files on the DNS servers respectively.

logger     IN      A      192.168.30.12

record for example.org zone file

12      IN      PTR      logger.example.org.

record for the 30.168.192.in-addr.arpa zone file

Don't forget to update the zone serial number and restart the named service.

Setting up the syslog jail

The syslog server will be created using a FreeBSD jail. If you have not already setup the Jail infrastructure review Episode 3 - Setting up FreeBSD Jail. For this jail we will be using the following file.

# Log server jail file

logger {
	# STARTUP / LOGGING
	exec.start = "/bin/sh /etc/rc";
	exec.stop = "/bin/sh /etc/rc.shutdown";
	exec.consolelog = "/var/log/jail_console_${name}.log";

	# PERMISSIONS
	allow.raw_sockets;
	exec.clean;
	mount.devfs;
	allow.reserved_ports = true;

	# HOSTNAME / PATH
	host.hostname = "${name}";
	path = "/usr/local/jails/containers/${name}";

	# NETWORK
	vnet;
	vnet.interface = "${epair}b";

	# NETWORKS/INTERFACES
	$id = "12";
	$ip = "192.168.30.${id}";
	$mask = "255.255.255.0";
	$gateway = "192.168.30.1";
	$bridge = "net-dmz";
	$epair = "epair${id}";

	exec.prestart += "ifconfig ${epair} create up";
	exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
	exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
	exec.start    += "ifconfig ${epair}b ${ip} netmask ${mask} up";
	exec.start    += "route add default ${gateway}";
	exec.poststop += "ifconfig ${bridge} deletem ${epair}a";
	exec.poststop += "ifconfig ${epair}a destroy";
}

/etc/jail.conf.d/logger.conf

Start the log server jail

service jail start logger

Configure the Log server

First we will create the /usr/local/etc/syslog.d directory if not already created.

[ -f /usr/local/etc/syslog.d ] || mkdir -p /usr/local/etc/syslog.d

command to create the syslog.d directory if it doesn't exist

Next create the directory to keep the logs in

[ -f /var/log/remote ] || mkdir -p /var/log/remote

command to create the remote directory if it doesn't exist

Next create the config file for receiving the remote logs. The config file will be kept in the /usr/local/etc/syslog.d directory

# Send any remote logs to a the remote file
:hostname, startswith, "192.168"
*.*                               /var/log/remote/remote.log

/usr/local/etc/syslog.d/remote.conf

Finally we will update the syslogd flags

sysrc syslogd_flags="-n -O rcf3164 "

Now we do not want the log file growing infinitely, so we need to rotate the log every so often. Everyday seems right. Let's also keep a week of old logs. Adjust the time frame and retention to suit your needs.

First we create a directory for our newsyslog config file.

[ -f /usr/local/etc/newsyslog.conf.d ] || mkdir -p /usr/local/etc/newsyslog.conf.d

command to create the newsyslog.d directory

Finally we create the config file. The newsyslog command gets run every hour so we do not need to make any additional changes.

/var/log/remote/remote.log     640 7 * $D0 CNTZ

/usr/local/etc/newsyslog.conf.d/remote.conf

Allocate the certificate

Review Episode 4 section Deploy Root cert and pull new cert to add a certificate to the logging server.

Enable and start the log server

Finally let's start the log server. Since syslog may already be started by default jail we will call the restart command to restart if necessary.

Then we will create the config file

service syslogd restart

command to restart syslogd

Update Logging in existing servers

Firewall

First login to the firewall and su(1) to root. Update the /etc/syslog.conf file and add the following line to the bottom of the file.

# Send log messages to log server
*.*                                              @192.168.30.12

append to /etc/syslog.conf

Finally, restart the syslogd service

rcctl restart syslogd

command to restart syslogd

DNS/DHCP server

First, login to the DNS/DHCP server and su(1) to root. Create the /usr/local/etc/syslog.d directory if it doesn't exist.

[ -f /usr/local/etc/syslog.d ] || mkdir -p /usr/local/etc/syslog.d

command to create syslog.d directory

Next create the syslog config file

Send log messages to log server
*.*                                          @192.168.30.12

/usr/local/etc/syslog.d/remote.conf

Finally restart syslogd

service syslogd restart

command to restart syslogd

Certificate Authority server

First, login to the CA server and su(1) to root. Create the /usr/local/etc/syslog.d directory if it doesn't exist.

[ -f /usr/local/etc/syslog.d ] || mkdir -p /usr/local/etc/syslog.d

command to create syslog.d directory

Next create the syslog config file.

# Send log messages to log server
*.*                                          @192.168.30.12

/usr/local/etc/syslog.d/remote.conf

Finally restart syslogd

service syslogd restart

command to restart syslogd

Conclusion

Now there is an issue with something, there is only one place to go to review the logs for all servers.

Self Hosted: Episode 4 - Certificates

2025-09-27T05:00:30Z

Software used:

FreeBSD 14.2 (jail)
Step CA 0.28.1 (from ports)

Updated: 2026-02-14

Introduction

If it isn't DNS... then it's certificates!

Certificates are simple... but it's complicated!

Certificates themselves are very simple... ignoring the all the crypto algorithms, hashes, etc. They are just two complementary long numbers that when put through those algorithms are able encrypt and decrypt messages. They are stored in two different files called the "certificate"/"public certificate"/"public cert"/"pem file", etc and the "private cert"/"private key"/"key file".

That's it! Simple!

Here's where things get complicated. There is this notion of "chain of trust". This chain is made up of one or more links. Each link must be trusted in order to trust the cert at the end of the chain. (example to follow) The head of the chain is called the "root". Root(s) (and there are many) "sign" a specific certificate file. This signing says to anyone receiving this certificate file, that if they "trust the root" they can then "trust the certificate file". This trust can go many layers deep.

Example: You trust Jim. (I'm not trying to figure out why you trust Jim, but you do. I mean, Jim is a good guy. Has good hair and a winning smile... so why not). Now Jim has given Sally a card that says that she can be trusted too. In addition Sally's card says that she can issue cards to others that she trusts. Sally gives Larry a card that she trusts him. Now when Larry shows you his card, should you trust Larry? Well you see that Larrys card is signed by Sally and Sallys card is signed by Jim. So if you trust Jim, then you trust Sally and thus you can trust Larry.

Now what happens if you don't trust Jim? Pretty simple, you don't trust anyone who Jim has extended his trust to. Or what happens if suddenly Sally becomes untrustworthy? Now anyone she extended her trust too is not untrust worthy.

The real question is how do you know who to trust. Fortunately?!, device and OS manufactures have created a list of trusted sources (referred to as Certificate Authorities or CAs)... and it's a long list. If you had to choose, you probably wouldn't have as long a list as this. Because anyone in this list is considered totally trust worthy and anyone they say is trustworthy is also totally trust worthy. (most probably are... but not all!)

So who can we really trust?!?! Ourselves! Now, let's create our own Certificate Authority!!!

Wait! What? Why would we do this?

Three reasons. First is that all the "default" CAs are public and only issue certificates to public reachable entities. (We will eventually make use of these public CAs). And our Self Hosted environment will not all be accessible to the greater Internet. (If we setup our firewall correctly!) Second, certificates from these public CAs can cost money. Sometime lots of money. Third, eventually we will want to do what is called mutual authentication (but more on that later).

Now in order to make running our own certificate authority we will want to automate as much of the certificate functions as possible. The issuing, expiration, revocation, etc should be automated. Fortunately, the ACME protocol, made popular by, Lets Encrypt can be used on our systems to automatically allocated and update certificates on our internal servers as long as our CA supports the ACME protocol (and it does).

Update DNS with new address

To begin with we will need to be able to resolve the new certificate authority. To do this update the example.org and 30.168.192.in-addr.arpa zone files on the DNS servers respectively.

certauth     IN      A      192.168.30.11

record for example.org zone file

11      IN      PTR      certauth.example.org.

record for the 30.168.192.in-addr.arpa zone file

Don't forget to update the zone serial number and restart the named service.

Setup the CA

The CA server will be created using a FreeBSD jail. If you have not already setup the Jail infrastructure review Episode 3 - Setting up FreeBSD Jail. For this jail we will be using the following file.

# Certificate server jail file

certauth {
	# STARTUP / LOGGING
	exec.start = "/bin/sh /etc/rc";
	exec.stop = "/bin/sh /etc/rc.shutdown";
	exec.consolelog = "/var/log/jail_console_${name}.log";

	# PERMISSIONS
	allow.raw_sockets;
	exec.clean;
	mount.devfs;
    allow.reserved_ports;

	# HOSTNAME / PATH
	host.hostname = "${name}";
	path = "/usr/local/jails/containers/${name}";

	# NETWORK
	vnet;
	vnet.interface = "${epair}b";

	# NETWORKS/INTERFACES
	$id = "11";
	$ip = "192.168.30.${id}";
	$mask = "255.255.255.0";
	$gateway = "192.168.30.1";
	$bridge = "net-dmz";
	$epair = "epair${id}";

	exec.prestart += "ifconfig ${epair} create up";
	exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
	exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
	exec.start    += "ifconfig ${epair}b ${ip} netmask ${mask} up";
	exec.start    += "route add default ${gateway}";
	exec.poststop += "ifconfig ${bridge} deletem ${epair}a";
	exec.poststop += "ifconfig ${epair}a destroy";
}

/etc/jail.conf.d/certauth.conf

Start the Jail

Start the certauth jail.

service jail start certauth

command to start the jail

Install the CA service in the jail

Once the jail has been started we need to update the FreeBSD package system and install the CA packages into the jail.

pkg -j certauth update
pkg -j certauth install -y step-certificates step-cli

commands to install the CA software

Connect to the new jail

jexec certauth /bin/sh

command to enter the jail

Generate the configuration

First export some common used vars and check if directories exist

export SDIR=/usr/local/etc/step
export STEPPATH=$SDIR/ca
export FQDN=$(hostname)
test -d $SDIR || mkdir -p $SDIR

commands to create the CA directories

Next generate the password key. This is the key that will be used to encrypt all cert passwords. (KEEP IT SAFE!) It is not out of the realm to have this key be stored on some sort of hardware token. The process for doing that is outside of this article.

openssl rand -base64 20 > $SDIR/password.txt
chown root:wheel $SDIR/password.txt
chmod 400 $SDIR/password.txt

commands to create the CA password

Next initialize the the certificate authority. This will create the root and intermediate certificates.

test -f $SDIR/ca/config/ca.json || /usr/local/bin/step ca init --address :8443 --deployment-type standalone --dns $FQDN --name "Root CA" --with-ca-url https://$FQDN --password-file $SDIR/password.txt --provisioner certificate_authority --provisioner-password-file $SDIR/password.txt

commands to create and initialize the CA

Setup the ACME provisioner.

/usr/local/bin/step ca provisioner add acme --type ACME

commands to enable an ACME provisioner in the CA

Now setup nginx to act as a proxy.

#user  nobody;
worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    server {
        listen       80;
        server_name  certauth.example.org;
        return 302 https://$host$request_uri;
    }

    # HTTPS server
    server {
        listen       443 ssl;
        server_name  cetauth.example.org;

        ssl_certificate      /usr/local/etc/ssl/certauth.example.org.crt;
        ssl_certificate_key  /usr/local/etc/ssl/certauth.example.org.key;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

        location / {
            proxy_pass   https://127.0.0.1:8443;
            proxy_set_header Host "certauth.example.org";
        }
    }
}

/usr/local/etc/nginx/nginx.conf

Start the CA service

service step_ca enable
service step_ca start

commands to enable and start the CA

Create a web server certificate

test -f /usr/local/etc/ssl/certauth.example.org.crt || STEPPATH=/usr/local/etc/step/ca /usr/local/bin/step ca certificate ${FQDN} /usr/local/etc/ssl/${FQDN}.crt /usr/local/etc/ssl/${FQDN}.key --ca-config /usr/local/etc/step/ca/config/ca.json --offline --password-file /usr/local/etc/step/password.txt --san ${FQDN} --provisioner certificate_authority

command to create web certificate

Now start the web server

service nginx enable
service nginx start

commands to start nginx web server

Deploy the new root certificate

To deploy the new root certificate to the certificate host (can confirm things are working with nginx proxy) first fetch the cert into the /usr/local/share/certs directory.

test -d /usr/local/share/certs || mkdir -p /usr/local/share/certs
fetch --no-verify-peer --no-verify-hostname -o /usr/local/share/certs/root_ca.crt https://certauth.example.org/roots.pem

commands to fetch the root CA certificate

Finally update the system certificate store

certctl rehash

command to install and trust the new CA certificate

Deploy root cert and pull new cert

DNS server

Add and trust the root cert

test -d /usr/local/share/certs || mkdir -p /usr/local/share/certs
fetch --no-verify-peer --no-verify-hostname -o /usr/local/share/certs/root_ca.crt https://certauth.example.org/roots.pem
certctl rehash

commands to install and trust the new CA certificate

Install the acme package

pkg install -y acme.sh

command to install the acme.sh package

Generate the certificate.

export CRTP=/usr/local/etc/ssl
test -f $CRTP || mkdir -p $CRTP
acme.sh --issue --standalone -d $(hostname) --cert-file $CRTP/$(hostname -s).crt --key-file $CRTP/$(hostname -s).key --fullchain-file $CRTP/$(hostname -s)_fullchain.pem --server https://certauth.example.org/acme/acme/directory

Commands to get a new certificate issued.

Update cron to renew the certificate

echo '0 0 * * * "/usr/local/sbin/acme.sh --cron --home /root/.acme.sh" > /dev/null' > /etc/cron.d/acme_sh

command to deploy a cron job to update certificate when it expires.

Firewall

Add and trust the root cert.

ftp -S dont -o /etc/ssl/example.org_root_ca.pem https://192.168.30.11/roots.pem
cat /etc/ssl/example.org_root_ca.pem >> /etc/ssl/cert.pem

command to install and trust the new CA root certificate.

Create the /etc/acme-client.conf

#
authority example.org {
        api url "https://certauth.example.org/acme/acme/directory"
        account key "/etc/acme/example.org-privkey.pem"
}

domain firewall.example.org {
        domain key "/etc/ssl/private/firewall.key"
        domain full chain certificate "/etc/ssl/firewall.fullchain.pem"
        sign with example.org
}

/etc/acme-client.conf

Create the /etc/httpd.conf

# 
server "*" {
        listen on * port 80
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location * {
                block 
        }
}

/etc/httpd.conf

Start the web server

rcctl enable httpd
rcctl start httpd

command to start the web server

Generate the certificate

acme-client $(hostname)

command to install a new device certificate

Update daily cron file to check / renew certificate.

echo '/usr/sbin/acme-client $(hostname)' >> /etc/daily.local

command to setup cron jobe to renew certificate automatically

Create a manual certificates

Host Certificates

To manually create a host certificate the following command can be used on the certauth server.

step certificate create blah.example.org blah.crt blah.key \
  --force --profile=leaf \
  --ca=/usr/local/etc/step/ca/certs/intermediate_ca.crt \
  --ca-key=/usr/local/etc/step/ca/secrets/intermediate_ca_key \
  --ca-password-file=/usr/local/etc/step/password.txt \
  --no-password --insecure --not-after=$((24*390))h

step certificate p12 blah.p12 blah.crt blah.key

User/Email Certificates

At some time we may want to create a certificate for an email address and not a host. To do this create the following file as a template 'email.tpl'.

{
 "sans": {{ toJson .SANs }},
 "keyUsage": ["digitalSignature" ],
 "extKeyUsage": ["clientAuth", "emailProtection" ]
}

email.tpl

Then use this command to create the certificate.

step certificate create oscar@example.org oscar.crt oscar.key \
  --force --ca=/usr/local/etc/step/ca/certs/intermediate_ca.crt \
  --ca-key=/usr/local/etc/step/ca/secrets/intermediate_ca_key \
  --ca-password-file=/usr/local/etc/step/password.txt \
  --no-password --insecure --not-after=$((24*390))h \
  --template=email.tpl

step certificate p12 oscar.p12 oscar.crt oscar.key

Conclusion

Now that we have DNS, DHCP, Certificates and the firewall working, can we begin to start thinking about adding something that is actually useful? Well not quite yet... :/

Self Hosted: Episode 3 - DNS/DHCP/NTP

2025-09-20T05:00:06Z

Used Software:

FreeBSD 14.2 (as jail)
ISC Bind 9.2.0 (from ports)
ISC DHCP 4.4 (from ports)

Introduction

It all begins with DNS...

Building the Jail environment

The DNS/DHCP server will be built using a FreeBSD jail. This is done for several reason. First, FreeBSD Jails come up nearly as quick as the hypervisor itself, meaning if everything is (re)started at the same time the DNS server should be the first service up an running. Second, is that FreeBSD jails provide isolation from other processes on the same system. Third, they are light weight.

What is a FreeBSD Jail (referred to as just a 'jail' from now on)? A jail is an isolated virtual space to run processes in that is separate from the host OS, and all other jails. From the Linux perspective jails are very close to Docker containers (although they pre-date Docker by a decade).

Setup the Jail sever

To setup the jail server first we need a space to store all the jails. Jails consist of the base FreeBSD OS version of our choice. We will be making use of ZFS snapshots to make deploying and updating jails quickly.

Create Jail from template

To create a jail from the template use the ZFS clone command.

zfs clone data01/jails/templates/14.2-RELEASE@base data01/jails/dns

command to clone a jail drive

Setup the DNS/DHCP server jail

Next we will create a jail configuration file. These files will be stored in the /etc/jails.conf.d directory. Each jail will have its own file. The file is used to setup the properties of the jail, what it is allowed to do within the host OS and any pre/port commands that need to be run in the host OS when the jail is started/stopped.

First we need to add/update the /etc/devfs.conf file. This file controls what resources are available to the jail. In this case we will need to expose the BPF devices to listen for DHCP requests. The following lines need to be added to the /etc/devfs.conf file

[devfsrules_jail_bpf=10]
add include $devfsrules_jail
add path bpf* unhide

partial /etc/devfs.conf

Now create the /etc/jail.conf.d/dns.conf file for defining and starting the jail.

# DNS/DHCP server jail file

dns {
	# STARTUP / LOGGING
	exec.stop = "/bin/sh /etc/rc.shutdown";
	exec.consolelog = "/var/log/jail_console_${name}.log";

	# PERMISSIONS
	allow.raw_sockets;
	exec.clean;
	mount.devfs;
    devfs_ruleset = 10;
    allow.reserved_ports = true;

	# HOSTNAME / PATH
	host.hostname = "${name}";
	path = "/data01/jails/${name}";

	# NETWORK
	vnet;
	vnet.interface = "${epair}b";

	# NETWORKS/INTERFACES
	$id = "10";
	$ip = "192.168.30.${id}";
	$mask = "255.255.255.0";
	$gateway = "192.168.30.1";
	$bridge = "net-dmz";
	$epair = "epair${id}";

	exec.prestart += "ifconfig ${epair} create up";
	exec.prestart += "ifconfig ${epair}a up descr jail:${name}";
	exec.prestart += "ifconfig ${bridge} addm ${epair}a up";
	exec.start    += "ifconfig ${epair}b ${ip} netmask ${mask} up";
	exec.start    += "route add default ${gateway}";
  	exec.start    += "/bin/sh /etc/rc";
	exec.poststop += "ifconfig ${bridge} deletem ${epair}a";
	exec.poststop += "ifconfig ${epair}a destroy";
}

/etc/jails.conf.d/dns.conf

Start the jail

Once the configuration file has been complete the jail should be started.

service jail start dns

command to start the jail

Install the DNS/DHCP service in the jail

Once the jail has been started we need to update the FreeBSD package system and install the DNS and DHCP packages into the jail.

pkg -j dns update

command to update the packages in the jail

Setup the new jail

To enter the jail from the host OS use the jexec(8) command and specify the command to run inside the jail.

jexec -l dns /bin/sh

command to connect to the jail

Once in the jail you will be the root user. The jail itself will have no service running by default, not even SSH. Let's enable the SSH service... but first we need to create a user account because we don't want to SSH into the server as root... (or at least we shouldn't do this... it is up to you).

jexec -l dns pw user add alfred -m -s /bin/sh -u 1001 -G wheel
jexec -l dns passwd alfred
jexec -l dns passwd root

commands to create a jail user and set the passwords

Fix sshd to disallow root to login via SSH. Edit the /etc/ssh/sshd_config and explicitly disable root login.

jexec -l dns sed -i -r -e 's/^#?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config

command to edit sshd_config to prevent root logins

Now lets enable and start the SSH service within the jail

service -j dns sshd enable
service -j dns sshd start

command to enable and start sshd

Now we test. SSH to 192.168.30.10 (remember no DNS yet). Once that is working we can move on to the DNS configuration.

DNS server setup discussion...

Before we configure the DNS settings lets create a list of items we need to accomplish to be successful.

Resolve internal names from internal sources
Resolve external names from internal sources
Resolve DMZ names from external sources (future episode)
Filter external names from internal sources (bad domains, advertisements, etc) (future episode)
Provide Dynamic DNS updates from internal hosts

We can accomplish this is three different ways (there are probably more... but only three I'll mention). Each way has it's own complications.

The first way is to use DNS views (or DNS split view). Based on where the client request comes from the client will be presented with one view or another view of the domain. This can be tricky to troubleshoot.

The second method, that is easier to troubleshoot, is to use two different domains. One domain is used for internal applications and the other domain is used for external applications. This is easier to troubleshoot, but become complicated when we need to expose an internal resource externally.

The third method is to use a single domain but two authoritative name servers, and internal and external name server. This method requires the most work of all since we will need to maintain two separate servers, but from the client standpoint, it is very easy to understand. External clients connect to the external name server, Internal clients use the internal name server.

In series I will present the third option. While the goal is to "self host" everything, we will require internet facing services at predictable addresses. This will allow us to utilize one of the many Internet DNS providers out there while maintaining maximum control.

Configuring DNS server

(this section assumes that we are ssh'd into the jail)

On FreeBSD the configuration files are generally stored in the /usr/local/etc/namedb directory.

First we need to create the rndc.conf file.

rndc-confgen -a -k DNS_UPDATE

command to create the rndc.conf file

This will create a file in the /usr/local/etc/namedb/ directory.

Next we will crate the named.conf file.

//
options {
        // All file and path names are relative to the chroot directory,
        // if any, and should be fully qualified.
        directory       "/usr/local/etc/namedb/working";
        pid-file        "/var/run/named/pid";
        dump-file       "/var/dump/named_dump.db";
        statistics-file "/var/stats/named.stats";
        listen-on       {
                                127.0.0.1;
                                192.168.30.10;
                         };
// These zones are already covered by the empty zones listed below.
// If you remove the related empty zones below, comment these lines out.
        disable-empty-zone "255.255.255.255.IN-ADDR.ARPA";
        disable-empty-zone "0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA";
        disable-empty-zone "1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.IP6.ARPA";

        # allow recursion
        allow-recursion { 0.0.0.0/0; };
};

include "/usr/local/etc/namedb/rndc.key";
controls {
        inet 127.0.0.1 port 953 allow { 127.0.0.1; } keys { "DNS_UPDATE"; };
};

zone "." { type hint; file "/usr/local/etc/namedb/named.root"; };

zone "example.org" {
        type primary;
        file "/usr/local/etc/namedb/primary/example.org";
        allow-update { key DNS_UPDATE; };
};

zone "10.168.192.in-addr.arpa" {
        type primary;
        file "/usr/local/etc/namedb/primary/10.168.192.in-addr.arpa";
        allow-update { key DNS_UPDATE; };
};
zone "20.168.192.in-addr.arpa" {
        type primary;
        file "/usr/local/etc/namedb/primary/20.168.192.in-addr.arpa";
        allow-update { key DNS_UPDATE; };
};
zone "30.168.192.in-addr.arpa" {
        type primary;
        file "/usr/local/etc/namedb/primay/30.168.192.in-addr.arpa";
        allow-update { key DNS_UPDATE; };
};
zone "40.168.192.in-addr.arpa" {
        type primary;
        file "/usr/local/etc/namedb/primary/40.168.192.in-addr.arpa";
        allow-update { key DNS_UPDATE; };
};

/usr/local/etc/namedb/named.conf

Finally we create the zone files stored in the /usr/local/etc/namedb/primary/ directory.

; 
$TTL 60
$ORIGIN example.org.
@       IN      SOA     dns.example.org. noone.example.org. (
                10
                1h
                3h
                2h
                4h
)
;
; NS records
        IN      NS      dns.example.org.
; MX records
; A records
justatest       IN      A       192.168.10.254
dns     IN      A       192.168.30.10
firewall-dmz      IN      A      192.168.30.1
firewall-iot      IN      A      192.168.20.1
firewall      IN      A      192.168.10.1
; CNAME records

/usr/local/etc/namedb/primary/example.org

; 
$TTL 60
$ORIGIN 10.168.192.in-addr.arpa.
@       IN      SOA     dns.example.org. noone.example.org. (
        10
        1h
        3h
        2h
        4h
)
; NS records
        IN      NS      dns.example.org.
; PTR records
254     IN      PTR     justatest.example.org.
1       IN      PTR     firewall.example.org.

/usr/local/etc/namedb/primary/10.168.192.in-addr.arpa

; 
$TTL 60
$ORIGIN 20.168.192.in-addr.arpa.
@       IN      SOA     dns.example.org. noone.example.org. (
        10
        1h
        3h
        2h
        4h
)
; NS records
        IN      NS      dns.example.org.
; PTR records
1       IN      PTR     firewall-iot.example.org.

/usr/local/etc/namedb/primary/20.168.192.in-addr.arpa

; 
$TTL 60
$ORIGIN 30.168.192.in-addr.arpa.
@       IN      SOA     dns.example.org. noone.example.org. (
        10
        1h
        3h
        2h
        4h
)
; NS records
        IN      NS      dns.example.org.
; PTR records
10      IN      PTR     dns.example.org.
1       IN      PTR     firewall-dmz.example.org.

/usr/local/etc/namedb/primary/30.168.192.in-addr.arpa

; 
$TTL 60
$ORIGIN 40.168.192.in-addr.arpa.
@       IN      SOA     dns.example.org. noone.example.org. (
        10
        1h
        3h
        2h
        4h
)
; NS records
        IN      NS      dns.example.org.

/usr/local/etc/namedb/primary/40.168.192.in-addr.arpa

Once bind has been configured we need to enable it so that it starts automatically.

service named enable
service named start

command to enable and start named service

Confirm DNS server is working

To confirm that DNS is working the following commands can be run.

dig @127.0.0.1 google.com
dig @127.0.0.1 justatest.example.org

commands to test recursive and authoritative resolution

Both commands should succeed. Assuming it is working, update the /etc/resolv.conf file

nameserver 127.0.0.1

/etc/resolv.conf

DHCP server discussion

As with DNS server, let's take a moment to discuss what we want to accomplish with the DHCP server.

Provide DHCP for the COMMON segment.
Provide DHCP for the IOT segment.
Dynamically update DNS records for the assigned IPs

One important thing to note is that DHCP clients use a broadcast to find a DHCP server. This broadcast does not extend past the local subnet (without help).

Install DHCP server software

First step is to install the DHCP server software

pkg install isc-dhcp44-server

command to install the dhcp server software

Configuring DHCP server

Use the following config for the DHCP server

#

# option definitions common to all supported networks...
option domain-name "example.org";

default-lease-time 86400;
max-lease-time 86400;

ddns-update-style standard;
deny client-updates;
ddns-updates on;

authoritative;

log-facility local7;

include "/usr/local/etc/namedb/rndc.key";

zone example.org. {
        primary 127.0.0.1;
        key DNS_UPDATE;
}

zone 10.168.192.in-addr.arpa. {
        primary 127.0.0.1;
        key DNS_UPDATE;
}
zone 20.168.192.in-addr.arpa. {
        primary 127.0.0.1;
        key DNS_UPDATE;
}
zone 30.168.192.in-addr.arpa. {
        primary 127.0.0.1;
        key DNS_UPDATE;
}
zone 40.168.192.in-addr.arpa. {
        primary 127.0.0.1;
        key DNS_UPDATE;
}

subnet 192.168.10.0 netmask 255.255.255.0 {
        range 192.168.10.10 192.168.10.254;
        option routers 192.168.10.1;
        option domain-name-servers 192.168.30.10;
}
subnet 192.168.20.0 netmask 255.255.255.0 {
        range 192.168.20.10 192.168.20.254;
        option routers 192.168.20.1;
        option domain-name-servers 192.168.30.10;
}
subnet 192.168.30.0 netmask 255.255.255.0 {
        range 192.168.30.10 192.168.30.254;
        option routers 192.168.30.1;
        option domain-name-servers 192.168.30.10;
}
subnet 192.168.40.0 netmask 255.255.255.0 {
        range 192.168.40.10 192.168.40.254;
        option routers 192.168.40.1;
        option domain-name-servers 192.168.30.10;
}

/usr/local/etc/dhcpd.conf

Next we enable the DHCP service

service isc-dhcpd enable
service isc-dhcpd start

commands to enable and start the dhcp server

Setup DHCP relay on Firewall

As mentioned above DHCP uses a link local broadcast to find a DHCP server. In order to provide the help necessary to forward the broadcasts from different LAN segments we will setup the firewall to DHCP relaying.

On the firewall we will enable and start the DHCP relay service.

ln -s /etc/rc.d/dhcrelay /etc/rc.d/dhcrelay_iot
ln -s /etc/rc.d/dhcrelay /etc/rc.d/dhcrelay_common
rcctl enable dhcrelay_common dhcrelay_iot
rcctl set dhcrelay_common flags -i vio2 192.168.30.10
rcctl set dhcrelay_iot flags -i vio1 192.168.30.10
rcctl start dhcrelay_common dhcrelay_iot

Conclusion

This setups up the initial DNS and DHCP services. However, as this series progresses we will add DNS records.

Self Hosting: Episode 2 - Firewall

2025-09-14T17:00:31Z

Software Used:

OpenBSD 7.7

Introduction

The firewall will separate the segments and provide protection for services that must cross the segment boundaries. To that end firewall will have several interfaces. To keep rules as manageable as possible we will treat all hosts behind each segment/interface the same. This way we should only have to really worry about the exceptions.

Build the firewall OS

The firewall will run on OpenBSD.

Hypervisor setup

We will create a VM on the hypervisor for OpenBSD. We will add it to each virtual network so it will have an interface in each one. In addition we will connect it to the external segment.

To install OpenBSD in the bhyve hypervisor first create the VM.

vm create -t openbsd77 -s 10G -m 1G -c 2 firewall

Next add the different networks to the VM. This will create interfaces in the VM on the specific segments specified. Note: the order because that is the order the interfaces will show up within the guest OS.

vm add -t network -s net-external
vm add -t network -s net-iot
vm add -t network -s net-common
vm add -t network -s net-dmz

Now download the install image

vm iso https://cdn.openbsd.org/pub/OpenBSD/7.7/amd64/install77.iso

Once the download is complete. Start the VM install.

vm install firewall install77.iso

and connect to the console

vm console firewall

Now install OpenBSD like normal. One change that some may not be used to, since the VM does not have a "screen" we will need to tell the boot loader that use the TTY (com0) as the console. To do this on the boot prompt "boot>" type "set tty com0"

Now type "boot" to boot the installer. Once the boot is finished you'll be asked what you want to do. Select "I" for install and follow the prompts. OpenBSD has sensible defaults, so when in doubt, accept the default.

boot> boot
cannot open cd0a:/etc/random.seed: No such file or directory
booting cd0a:/7.7/amd64/bsd.rd: 4232111+1745920+3887480+0+716800 [109+483816+334631]=0xae2100
entry point at 0x1001000
Copyright (c) 1982, 1986, 1989, 1991, 1993
        The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2025 OpenBSD. All rights reserved.  https://www.OpenBSD.org

OpenBSD 7.7 (RAMDISK_CD) #613: Sun Apr 13 08:35:08 MDT 2025
    deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/RAMDISK_CD
real mem = 1032982528 (985MB)
avail mem = 995553280 (949MB)
random: good seed from bootblocks
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0x3fbcf000 (11 entries)
bios0: vendor BHYVE version "14.0" date 10/17/2021
bios0: FreeBSD BHYVE
efi0 at bios0: UEFI 2.7
efi0: BHYVE rev 0x10000
acpi0 at bios0: ACPI 5.1
acpi0: tables DSDT FACP APIC HPET MCFG SPCR
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz, 1597.73 MHz, 06-3f-02
cpu0: cpuid 1 edx=1f8bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT> ecx=f6da3203<SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV>
cpu0: cpuid 6 eax=4<ARAT>
cpu0: cpuid 7.0 ebx=7a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID> edx=400<MD_CLEAR>
cpu0: cpuid d.1 eax=1<XSAVEOPT>
cpu0: cpuid 80000001 edx=2c100800<NXE,PAGE1GB,RDTSCP,LONG> ecx=21<LAHF,ABM>
cpu0: cpuid 80000007 edx=100<ITSC>
cpu0: MELTDOWN
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache
cpu0: apic clock running at 134MHz
cpu at mainbus0: not configured
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 32 pins
acpihpet0 at acpi0: 16777216 Hz
acpiprt0 at acpi0: bus 0 (PC00)
acpipci0 at acpi0 PC00
"PNP0303" at acpi0 not configured
"PNP0F13" at acpi0 not configured
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifocom0: console
com1 at acpi0 COM2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
com2 at acpi0 COM3 addr 0x3e8/0x8 irq 4: ns16550a, 16 byte fifo
com3 at acpi0 COM4 addr 0x2e8/0x8 irq 3: ns16550a, 16 byte fifo
acpicmos0 at acpi0
"Bhyve_V_Gen_Counter_V1" at acpi0 not configured
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: bhyve
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 unknown vendor 0x1275 product 0x1275 rev 0x00
ahci0 at pci0 dev 3 function 0 "Intel 82801H AHCI" rev 0x00: msi, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus0 at ahci0: 32 targets
cd0 at scsibus0 targ 0 lun 0: <BHYVE, BHYVE DVD-ROM, 001> removable
virtio0 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio0
virtio0: msix per-VQ
scsibus1 at vioblk0: 1 targets
sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 0MB, 512 bytes/sector, 32 sectors
virtio1 at pci0 dev 5 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio1: 1 queue, address 58:9c:fc:00:73:1c
virtio1: msix per-VQ
virtio2 at pci0 dev 6 function 0 "Qumranet Virtio RNG" rev 0x00
viornd0 at virtio2
virtio2: msix per-VQ
"Intel 82371SB ISA" rev 0x00 at pci0 dev 31 function 0 not configured
isa0 at mainbus0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0 mux 1
efifb at mainbus0 not configured
softraid0 at root
scsibus2 at softraid0: 256 targets
root on rd0a swap on rd0b dump on rd0b
WARNING: CHECK AND RESET THE DATE!
erase ^?, werase ^W, kill ^U, intr ^C, status ^T

Welcome to the OpenBSD/amd64 7.7 installation program.
(I)nstall, (U)pgrade, (A)utoinstall or (S)hell? i
At any prompt except password prompts you can escape to a shell by
typing '!'. Default answers are shown in []'s and are selected by
pressing RETURN.  You can exit this program at any time by pressing
Control-C, but this can leave your system in an inconsistent state.

Terminal type? [vt220] 
System hostname? (short form, e.g. 'foo') firewall

Available network interfaces are: vio0 vlan0.
Network interface to configure? (name, lladdr, '?', or 'done') [vio0] 
IPv4 address for vio0? (or 'autoconf' or 'none') [autoconf] 
IPv6 address for vio0? (or 'autoconf' or 'none') [none] 
Available network interfaces are: vio0 vlan0.
Network interface to configure? (name, lladdr, '?', or 'done') [done] 

Password for root account? (will not echo) 
Password for root account? (again) 
Start sshd(8) by default? [yes] 
Change the default console to com0? [yes] 
Available speeds are: 9600 19200 38400 57600 115200.
Which speed should com0 use? (or 'done') [9600] 115200
Setup a user? (enter a lower-case loginname, or 'no') [no] alfred
Full name for user alfred? [alfred] 
Password for user alfred? (will not echo) 
Password for user alfred? (again) 
WARNING: root is targeted by password guessing attacks, pubkeys are safer.
Allow root ssh login? (yes, no, prohibit-password) [no] 
What timezone are you in? ('?' for list) [UTC] 

Available disks are: sd0.
Which disk is the root disk? ('?' for details) [sd0] 
Encrypt the root disk with a (p)assphrase or (k)eydisk? [no] 
No valid MBR or GPT.
Use (W)hole disk MBR, whole disk (G)PT or (E)dit? [gpt] 
Setting OpenBSD GPT partition to whole sd0...done.
The auto-allocated layout for sd0 is:
#                size           offset  fstype [fsize bsize   cpg]
  a:          1177.2M           532544  4.2BSD   2048 16384     1 # /
  b:           256.0M          2943424    swap                    
  c:         10240.0M                0  unused                    
  d:          3072.0M          3467712  4.2BSD   2048 16384     1 # /usr
  e:          2048.0M          9759168  4.2BSD   2048 16384     1 # /home
  i:           260.0M               64   MSDOS                    
Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a] 
/dev/rsd0a: 1177.2MB in 2410880 sectors of 512 bytes
6 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
/dev/rsd0e: 2048.0MB in 4194304 sectors of 512 bytes
11 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
/dev/rsd0d: 3072.0MB in 6291456 sectors of 512 bytes
16 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
/dev/sd0a (bc82b06ecfc64241.a) on /mnt type ffs (rw, asynchronous, local)
/dev/sd0e (bc82b06ecfc64241.e) on /mnt/home type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd0d (bc82b06ecfc64241.d) on /mnt/usr type ffs (rw, asynchronous, local, nodev)

Let's install the sets!
Location of sets? (cd0 disk http nfs or 'done') [cd0] 
Pathname to the sets? (or 'done') [7.7/amd64] 

Select sets by entering a set name, a file name pattern or 'all'. De-select
sets by prepending a '-', e.g.: '-game*'. Selected sets are labelled '[X]'.
    [X] bsd           [X] base77.tgz    [X] game77.tgz    [X] xfont77.tgz
    [X] bsd.mp        [X] comp77.tgz    [X] xbase77.tgz   [X] xserv77.tgz
    [X] bsd.rd        [X] man77.tgz     [X] xshare77.tgz
Set name(s)? (or 'abort' or 'done') [done] 
Directory does not contain SHA256.sig. Continue without verification? [no] yes
Installing bsd          100% |**************************| 31152 KB    00:09    
Installing bsd.mp       100% |**************************| 31257 KB    00:09    
Installing bsd.rd       100% |**************************|  4687 KB    00:01    
Installing base77.tgz   100% |**************************|   483 MB    03:01    
Extracting etc.tgz      100% |**************************|   275 KB    00:00    
Installing comp77.tgz   100% |**************************| 81731 KB    00:38    
Installing man77.tgz    100% |**************************|  8265 KB    00:04    
Installing game77.tgz   100% |**************************|  2746 KB    00:01    
Installing xbase77.tgz  100% |**************************| 60706 KB    00:26    
Extracting xetc.tgz     100% |**************************|  8474       00:00    
Installing xshare77.tgz 100% |**************************|  4554 KB    00:03    
Installing xfont77.tgz  100% |**************************| 23021 KB    00:08    
Installing xserv77.tgz  100% |**************************| 20074 KB    00:08    
Installing BUILDINFO    100% |**************************|    54       00:00    
Location of sets? (cd0 disk http nfs or 'done') [done] 
Saving configuration files... done.
Making all device nodes... done.
Multiprocessor machine; using bsd.mp instead of bsd.
fw_update: add intel; update none
Relinking to create unique kernel... done.

CONGRATULATIONS! Your OpenBSD install has been successfully completed!

When you login to your new system the first time, please read your mail
using the 'mail' command.

Exit to (S)hell, (H)alt or (R)eboot? [reboot] halt
syncing disks... done

The operating system has halted.
Please press any key to reboot.

Once the VM has been installed break out of the console using ~. escape. This will take you back to a the hypervisor. From the hypervisor make sure the VM is halted using the 'vm list' command. The state should be "Stopped"

root@hyperv01:/data01/vms # vm list
NAME      DATASTORE  LOADER  CPU  MEMORY  VNC  AUTO  STATE
firewall  default    uefi    2    1G      -    No    Stopped

If not, then use the 'vm poweroff firewall' command to stop the VM.

Now that OpenBSD has been installed on the VM, restart the VM using the 'vm start firewall' command.

Firewall Setup

The firewall needs to be able to route packets. Edit the /etc/sysctl.conf file to enable IP forwarding.

net.ip.forwarding=1
net.ip6.forwarding=1

/etc/sysctl.conf

Next we will setup the /etc/pf.conf and other firewall config files. First we will create a directory to keep the files organized.

mkdir /etc/pf.d

Since we want to treat each segment/interface the same we will make use of tags to help organize the traffic. The way tags work is traffic is matched and tagged on input. The tag can then be used to allow/deny the traffic on exit.

One thing to note with PF rules, they are processed in what is called "last match". Meaning that the last matching rule for a packet is the action that is taken. With this in mind we will specify the general actions at the top of the rule file and then get more specific at the end of the rule file.

The following files will be used to start. We will update the files as required.

# IOT Segment
anchor iot on IOT {
  match in tag IOT
}

/etc/pf.d/pf.anchor.iot

# Common Segment
anchor common on COMMON {
  match in tag COMMON
  deny out tagged IOT
  deny out tagged DMZ
}

/etc/pf.d/pf.anchor.common

# DMZ Segment
anchor dmz on DMZ {
  match in tag DMZ
  deny out tagged IOT
}

/etc/pf.d/pf.anchor.dmz

# External Segment
anchor external on EXTERNAL {
  deny in drop
  deny out drop
  deny out tagged IOT return
  # NAT outbound traffic to external interfaces
  pass out tagged { DMZ, COMMON } nat-to ($extif)
}

/etc/pf.d/pf.anchor.extranet

And finally create the /etc/pf.conf file.

# pf.conf
#

####
#### MACROS
####
ext_if="vio0"

# skip internal interfaces
set skip on lo
set skip on enc
set state-defaults pflow

table <all_nets> { 192.168.0.0/16 }
table <all6_nets> { 2001:db8::/32 }

####
#### Common RULES
####
# default block.  Remove log once all rules are sorted
block log
# capture NTP from any internal sources and respond for them
pass in proto udp from <all_nets> to port 123 rdr-to 127.0.0.1 port 123
pass in inet6 proto udp from <all6_nets> to port 123 rdr-to ::1 port 123
# Allow Neighbor solicitations
pass in quick on $ext_if inet6 proto icmp6 all icmp6-type { routeradv neighbrsol neighbradv }
# Allow DHCPv6
pass in quick on $ext_if inet6 proto udp \
	from fe80::/10 port dhcpv6-server \
	to fe80::/10 port dhcpv6-client no state
# Allow ping????
#pass in  quick on $ext_if inet6 proto icmp6 all icmp6-type echoreq
# Allow Link Local communications
pass in  quick on $ext_if inet6 proto icmp6 from fe80::/10 to fe80::/10
pass in  quick on $ext_if inet6 proto icmp6 from ff02::/16 to fe80::/10
pass out quick on $ext_if inet6 proto icmp6 from fe80::/10 to ff02::/16

# Include the anchor rules
include "/etc/pf.d/pf.anchor.common"
include "/etc/pf.d/pf.anchor.iot"
include "/etc/pf.d/pf.anchor.dmz"
include "/etc/pf.d/pf.anchor.external"

/etc/pf.conf

Once all the rules are in place we can test them using the pfctl(8) command.

pfctl -nf /etc/pf.conf

Any errors, typo, etc will be displayed. They will need to be fixed before the rules can be in place.

Adding the -v flag will show an output of all the rules.

Finally...

Once the rules are loading successfully, the firewall can be rebooted to make sure the rules are applied automatically.

reboot

Conclusion

This will get the basic firewall up and running. We will be revisiting the firewall config as we build out more of the environment.

Self Hosting: Episode 1 - Hypervisor

2025-09-07T16:22:33Z

Software Used:

Introduction

Before we can deploy any services we need two things. First a place to store those services and two a place to run those services from (i.e storage and compute). These two requirements do not need to run on the same system and for larger self hosted sites or businesses, they probably wont. But for these examples I will present them using a single system.

In these examples I will be utilizing FreeBSD as my compute and storage system. There are other systems out there: Proxmox, XCP-NG, etc. Please feel free to use any/all of these. My preference is FreeBSD because it gives me all the underlying services I need in a rock solid operating system.

Installing Base OS

Installing FreeBSD is straight forward. Chapter 2 of the FreeBSD handbook will walk you through installing FreeBSD on a bare system.

Network Prep

Within the hypervisor we will create separate virtual networks. Each virtual network will be on a separate VLAN. The hypervisor is not required to have an interface on any of the virtual networks. In fact, we don't want it to.

Create Bridge for Virtual Networks

On FreeBSD (as in Linux) a bridge network interface is used to create the virtual network. These commands will create the bridge and rename it to something more meaningful.

ifconfig bridge1 create
ifconfig bridge1 name net-common
ifconfig bridge2 create
ifconfig bridge2 name net-iot
ifconfig bridge3 create
ifconfig bridge3 name net-dmz
ifconfig bridge4 create
ifconfig bridge4 name net-external

Attach each bridge to the appropriate VLAN. Creating VLANs allows this system to communicate with other systems while maintaining the segments. For this example I will assume that the physical interface of the hypervisor is "vmx0". Replace this with the actual physical interface name.

ifconfig vlan10 create
ifconfig vlan10 vlan 10 vlandev vmx0
ifconfig net-common addm vlan10
ifconfig vlan20 create
ifconfig vlan20 vlan 20 vlandev vmx0
ifconfig net-iot addm vlan20
ifconfig vlan30 create
ifconfig vlan30 vlan 30 vlandev vmx0
ifconfig net-dmz addm vlan30
ifconfig vlan90 create
ifconfig vlan90 vlan 90 vlandev vmx0
ifconfig net-external addm vlan90

Next we will update /etc/rc.conf to make sure the changes will take affect on next reboot.

sysrc ifconfig_vlan10="vlan 10 vlandev vmx0
sysrc ifconfig_bridge1="name net-common addm vlan10"
sysrc ifconfig_vlan20="vlan 20 vlandev vmx0
sysrc ifconfig_bridge2="name net-iot addm vlan20"
sysrc ifconfig_vlan30="vlan 30 vlandev vmx0
sysrc ifconfig_bridge3="name net-dmz addm vlan30"
sysrc ifconfig_vlan90="vlan 90 vlandev vmx0
sysrc ifconfig_bridge4="name net-external addm vlan90"

sysrc cloned_interfaces+=" bridge1 vlan10"
sysrc cloned_interfaces+=" bridge2 vlan20"
sysrc cloned_interfaces+=" bridge3 vlan30"
sysrc cloned_interfaces+=" bridge4 vlan90"

Storage Prep

FreeBSD comes with ZFS. If you are not familiar with it see here, here and here for explanations. ZFS provides the many features we will need to hosting our environment.

Create data_dirs

This environment will have data that we want to protect. While ZFS will do everything reasonable to keep the data safe that isn't a 100% guarantee. Setting up our data_dirs will help us determine what data we consider important.

zpool create data01 da2

Now we create a dataset for virtual machine images

zfs create data01/vms

Now we create a dataset for jail'd processes.

zfs create data01/jails

Create a dataset for shared user data.

zfs create data01/user_data

Finally we create a dataset for backups

zfs create data01/backup_data

Virtual machines

Virtual machines can take several different flavors. They can be fully emulated hardware (type 1/2) or just isolated processes in containers (type 3). In this environment both will be used. To start with we will make use of FreeBSD Jails.

Prepare for Jails

To prepare the server for jails add a /etc/jail.conf file

#
.include "/etc/jail.conf.d/*.conf";

/etc/jail.conf

And then enable the jails to start on boot.

service jails enable

One nice feature of ZFS is that it can do snapshots. A snapshot is a exact copy of a dataset at the moment the snapshot is taken. Any updates to the dataset will not update the snapshot. We will use this feature of ZFS to create a snapshot of a jail. Then we will use that jail snapshot when we create new jails. By doing this we make adding new jail very quick.

To prepare the jail template first we create a new dataset.

zfs create -p data01/jails/templates/14.2-RELEASE

Next we will download the FreeBSD base archive and extract into the new dataset.

fetch -o - https://download.freebsd.org/ftp/releases/amd64/amd64/14.2-RELEASE/base.txz | tar -xf - -C /data01/jailes/templates/14.2-RELEASE --unlink

Update the resolv.conf and localtime settings.

cp /etc/resolv.conf /data01/jails/templates/14.2-RELEASE/etc/resolv.conf
cp /etc/localtime /data01/jails/templates/14.2-RELEASE/etc/localtime

Optimally we can perform any OS updates between now and when the base.txz was originally created.

freebsd-update -b /data01/jails/templates/14.2-RELEASE fetch install

Now take a snapshot of the dataset

zfs snapshot data01/jails/templats/14.2-RELEASE@base

An when we are ready to deploy a new jail we will "clone" the dataset for our jail.

zfs clone data01/jails/templates/14.2-RELEASE@base data01/jails/containers/<jailname>

Prepare for Virtual Machines

The virtual machines using Bhyve (the FreeBSD hypervisor) takes a few more steps. First install some packages

pkg install -y vm-bhyve bhyve-firmware edk2-bhyve grub-bhyve

After the packages are installed update the rc.conf with the following variables

sysrc vm_delay=5
sysrc vm_dir=zfs:/data01/vms

Enable the VMs to start on boot

service vm enable

Initialize the VM configuration

/usr/local/sbin/vm init

And finally update the VM config file adding the different segments so they were be available to any VMs.

vm switch create -t manual -b net-iot net-iot
vm switch create -t manual -b net-dmz net-dmz
vm switch create -t manual -b net-common net-common
vm switch create -t manual -b net-external net-external

Now the hypervisor is ready for VMs

Backups

While ZFS is incredibly stable and will do everything it can to protect your data, problems can happen outside of ZFS's control. For those case we need to create a backup. (See discussion on 3-2-1 backup scheme: here, here, here). Perhaps the best tool I've found to automate these backups is Sanoid/Syncoid.

Sanoid

Sanoid is a script that can be called from cron that will create and manage snapshots automatically. This will allow us to protect the files on the filesystem from accidental deletion. However we are not fully protected. If something were to happen to the hose or to enough hard drives all data could be lost. So we need to copy the data off of this system to someplace else.

First install sanoid on the system

pkg install sanoid

Next configure the sanoid.conf file located in /usr/local/etc/sanoid directory

#
[ data01/vms ]
        use_template = production
        recursive = yes

[ data01/jails ]
        use_template = production
        recursive = yes

[ data01/data ]
        use_template = production

#############################
# templates below this line #
#############################

# name your templates template_templatename. you can create your own, and use them in your module definitions above.

[template_demo]
        daily = 60

[template_production]
        frequently = 0
        hourly = 36
        daily = 30
        monthly = 3
        yearly = 0
        autosnap = yes
        autoprune = yes

/usr/local/etc/sanoid/sanoid.conf

Finally update crontab using 'crontab -e' command add the sanoid command.

*/15 * * * * TZ=UTC /usr/local/bin/sanoid --cron

Syncoid

Syncoid is also a script run from cron. It will take the snapshots created by Sanoid and ZFS send the snapshots to another system running ZFS. This systems does not need to be FreeBSD.

Syncoid uses the sanoid.conf file to determine what to sync so we only need to specify the destination server. Update cron and specify bkupuser, bkuphost, and bkuppath to tell syncoid where to sync the snapshots.

0 1 * * * TZ=UTC /usr/local/bin/syncoid --no-privilege-elivation --create-bookmark data01/vms <bkupuser>@<bkuphost>:<backuppath>
15 1 * * * TZ=UTC /usr/local/bin/syncoid --no-privilege-elivation --create-bookmark data01/jails <bkupuser>@<bkuphost>:<backuppath>
30 1 * * * TZ=UTC /usr/local/bin/syncoid --no-privilege-elivation --create-bookmark data01/data <bkupuser>@<bkuphost>:<backuppath>

Conclusion

With the physical systems out of the way, we can now begin creating the different services that will be needed to setup our self hosted environment.

Self Hosting: Episode 0

2025-09-07T15:59:02Z

As with all endeavors, success is not guaranteed. However, without a plan... failure is.

In this series I hope to provide instruction on how to setup and self host an environment that can provide everything necessary to function in todays connected world. While I will be simplifying setups and configurations, it is my goal that these instructions can be extended to businesses.

As with all things on the Internet, this is my opinion. I may not do things exactly as you would. That is fine. Please take this as a starting point and extend in whatever way you see fit. Now to the planning....

Segmentation Plan

Before we start creating servers, VMs, containers, etc. we need to have a quick think about segmentation. This environment can and will have all sorts of devices on it. These devices will be used for different services and have different levels of trust. For this series the following Places In the Network (PIN) will be used as a way to segment different devices.

External Segment

This segment represents the outside untrusted world. Only the firewall(s) should ever be connected to this segment. There should be a default deny on all inbound traffic.

Wither that default deny is a DROP (all traffic is just thrown away. No return traffic is ever sent) or a RETURN (ICMP return traffic is sent on bad packets) is a topic of much debate. With the DROP setup the firewall appears to be non-existent from the Internet. Only "known" communications will ever connect. However, this might slow down some services that will try to "test" for the presence of a service. If you do not know you need the RETURN response then do DROP for the external interface.

DMZ Segment

This segment will be where devices that can be accessed from outside our environment will be placed. These devices can/should be proxied in some form or fashion but since direct access to them will be provided we should separate these devices. Devices in this environment should not be able to reach back to Common Segment. Reachback to the IOT segment should require specific rules.

IOT Segment

This segment will be where devices that are IOT in nature are placed. (Cameras, outlets, doorbells, etc) Devices in this environment will not be able to reach out to the greater Internet or any other segment without explicit rules.

Common Segment

This segment will be where all other devices will be placed. Access from this segment to DMZ or IOT is generally allowed.

Other Segments

Other segments can be created as necessary. Word of caution, each segment carries with it a new IP space, possibly a new domain, and additional firewall rules and considerations on how this segment should interact with all other segments. In some cases this interaction is simple and in others it can be complex. Adding additional segments has an exponential cost. I am definitely not saying "Don't add any more segments.", I am only stating, understand the cost.

Here is a quick table that will be referenced by other episodes.

Segment	Vlan	Bridge	Address/Mask
Common	Vlan 10	Bridge 1	192.168.10.0/24
IOT	Vlan 20	Bridge 2	192.168.20.0/24
DMZ	Vlan 30	Bridge 3	192.168.30.0/24
External	Vlan 90	Bridge 4	10.0.0.0/24

Iterative approach

With this series we will touch on many topics. I will purposely limit my discussion on various topics to just the item at hand and when the time if right my come back to the topic to extend with additional items as required. As I know that people will skip around when reading the articles, I will do my best to refer back to previous articles where a similar topic was covered.

Services

Finally, the good stuff. The following services will be required to build the self hosted environment. These services can run on whatever hardware you have available. (Or virtual hardware of the cloud). How you choose to deploy these services is up to you and your budget. In these examples I will assume that all hardware an services are "local" unless otherwise specified.

List of services

Storage - We will need some place to store our OS's, files etc.
Virtual Machines - We will need a place to create virtual machines and services to run within the environment.
Firewall - Perhaps the most important service (only slightly ahead of DNS). This will allow communication between the different segments.
DNS/DHCP - It all starts with DNS. We will create a DNS environment that we can build and add onto.
Certificates - Right behind DNS we will create our own Certificate Authority and manage our own certificates.
Accounts - Even if this environment is only used for a single person, different types of accounts will be necessary.
SSO/MFA - With all these accounts, being forced to login to every machine/service every time gets annoying. Setting up a SSO/MFA solution will help consolidate the authentication.
Logging - What happens with things go wrong... and they will. We need logs.
Email - This is what it is all about? Well not totally, but email still plays an important role with interacting with the outside world.

This is not the complete of services that we can host, but this is my list of required services that we must host in order to really host anything else.

Conclusion

Throughout this series I will be running different operating systems and software. I will endeavor to list the version used.

Now we have a plan. Success still isn't guaranteed... but we can now measure how close we came.

Password sensibilities...

2024-10-11T18:27:26Z

Photo by David Gomez / Unsplash

Finally, some password sense! NIST has released some updates for password verifiers (SP 800-63B). While this isn't everyones cup of tea, it is nice when things like this come out to with discussions around security best practices.

Highlights

Some highlights from section 3.1.1.2 and my thoughts...

1. Verifiers and CSPs SHALL require passwords to be a minimum 
of eight characters in length and SHOULD require passwords 
to be a minimum of 15 characters in length.

This should already be in place.

2. Verifiers and CSPs SHOULD permit a maximum password
length of at least 64 characters.

This seems slightly silly. Passwords should not be stored in plaintext. If required, their hashes should be stored and those will use a fixed amount of space. Now, if the max length needs to be limited to prevent some denial of service, that would make sense, but set the length to 1000 characters.

3. Verifiers and CSPs SHOULD accept all printing ASCII
[RFC20] characters and the space character in passwords.

Yes! Once again, passwords should not be store plain text. So if the passwords are being hashed, then the characters shouldn't matter!

5. Verifiers and CSPs SHALL NOT impose other composition
rules (e.g., requiring mixtures of different character 
types) for passwords.

100 x Yes! Making passwords harder for people to type and impossible for them to remember doesn't make more secure passwords.

6. Verifiers and CSPs SHALL NOT require users to change 
passwords periodically. However, verifiers SHALL force a 
change if there is evidence of compromise of the 
authenticator.

1000 x Yes! Forcing periodic password changes forces people into bad habits.. (e.g. $uperP@ss01, $uperP@ss02, etc). Only change when required!

7. Verifiers and CSPs SHALL NOT permit the subscriber to 
store a hint that is accessible to an unauthenticated 
claimant.

Why was this even a thing? If I have to use this, just send me through the password reset process.

8. Verifiers and CSPs SHALL NOT prompt subscribers to use
knowledge-based authentication (KBA) (e.g., “What was the 
name of your first pet?”) or security questions when 
choosing passwords.

Awesome! I no longer have to make up random crap and store that crap. Once again, if I can't remember what I entered, I'm headed for the password reset process.

9. Verifiers SHALL verify the entire submitted password
(i.e., not truncate it).

This should really go without saying. Once again, hash the password and verify the hash!

Verifiers SHALL allow the use of password managers. 
Verifiers SHOULD permit claimants to use the “paste” 
functionality when entering a password to facilitate 
their use.

I was shocked when a company I worked for disallowed browser integrated password managers, forcing a person to copy the password to the clipboard (which is visible to all applications) and then to the password field. Now I know this doesn't exactly say "browser integrated password managers", but if it's not easy... people will not use it.

Email SHALL NOT be used for out-of-band authentication because
it may be vulnerable to:

 -  Accessibility using only a password
 -  Interception in transit or at intermediate mail servers
 -  Rerouting attacks, such as those caused by DNS spoofing

Email is easy to use and easy to abuse. While hopefully, an email user is using MFA / passkey / OTP to login, but there is no guarantee. And since DNSSEC and SMTPS are not ubiquitous, interception of the out-of-band authentication is possible. Although allowing some to login via a link in email, is easy... no tech experience required. :/

Section 5.1.1 even covers how browser cookies should be used for session maintenance.

Conclusion

Now I don't expect the password silliness to change over night. But now there is a government document that companies can refer to as "best practices".

Notes on DPI...

2024-09-06T13:40:03Z

Photo by Christian Kaindl / Unsplash

Some quick notes on setting DPI for High Density Displays and X11.

Background

On "normal" displays within X11 it is assumed that there are 96 pixels/dots per inch (DPI). This is fine for most cases. However, with "newer" displays, including on laptops, having double or more DPI this can make text really hard to see. Here are some tips that I've collected to help bring the fonts back in line.

Setting the correct DPI...

My daily window manager is cwm(1). within my ~/.xsession script I have the following.

DPI=`xdpyinfo | grep resolution | awk '{ print $2 }' | cut -f1 -dx`
DPI=${DPI:-96}
echo "DPI=${DPI}"
xrandr --dpi ${DPI}

~/.xsession snippet

This uses xdpyinfo(1) to fetch the DPI identified by X11. If not there or something goes wrong with the parse, assume 96 DPI and then output the setting (which is captured in ~/.xsession-errors) and use xrandr(1) to set the DPI.

Setting the pointer...

This will make the mouse pointers really small. To fix this I export the following settings (from the .xsession script).

export XCURSOR_PATH="~/.icons:/usr/local/lib/X11/icons:$XCURSOR_PATH"
export XCURSOR_SIZE="64"

~/.xsession snippet

The first export adds some new paths to look for new cursors. Use pkg_add(1) script to fine cursors that can be installed.

pkg_add -Q cursor

I also set the size to 64. (This should be multiples of 8). But on my display 64 seems to be a good size.

Setting the Xresources...

Updating the ~/.Xresources file will also help for programs that use this.

! set cursor size
Xcursor.theme: dmz-aa
Xcursor.size: 64

! default face name to use
*faceName: CodeNewRoman Nerd Font:style=Bold:size=20

~/.Xresources snippet

This also sets the cursor theme to use and size (duplicates the exports, but doesn't hurt). This also sets the default face name, style and size to use if not overridden by more specific settings.

Browsers...

Naturally the browsers don't seem to obey any of the X11 settings. Here are some specific settings that seem to help.

Firefox

Open about:config in the firefox multibar. Find layout.css.devPixelsPerPx and set to an appropriate number. Less than 1.0 makes things smaller and greater than 1.0 make thing larger. Updates take place immediately so it's nice that changes can be seen without restarting.

Chrome

Chrome is better/worse depending on your point of view. There doesn't seem to be any config setting to set. Instead a CLI option --force-device-scale-factor=x is used. As with firefox, number less than 1.0 will make things smaller and number larger than 1.0 will make thing bigger.

Since I usually launch chrome from cwm application menu, I have a simple shell script that add the CLI option.

#!/bin/sh

SF=${SCALE_FACTOR:-1.0}
ARGS="--force-device-scale-factor=${SF}"

/usr/local/bin/chrome ${ARGS} $@

start_chrome.sh

In the ~/.xsession script I set an environment variable of SCALE_FACTOR and then use that in this script to set the scale factor for chrome.

Conclusion

Other window managers / desktop environments like XFCE, Gnome, KDE seem to have their own ways of handling the display DPI discrepancy. They each have their plusses and minuses, but all require more dependencies. Choose what is right for you.

This doesn't take care of 100% of the applications, but covers the ones I use most.

HA @ Home

2024-08-05T20:48:35Z

Photo by Nathan Walker / Unsplash

I, like many of you, have been working from home since the pandemic. As a network engineer, I worked from anywhere I needed to as the job demanded. I met with my team and customers over WebEx, Zoom, Meet and later Teams. I was already pandemic ready, so when I had to stay home, it really wasn't a huge change... except for the network.

At work I had build a global, highly redundant, scalable network that always provided best experience no matter what was thrown at it. At home... well, you've heard of the cobblers children? Don't get me wrong, the network worked. I could watch easily watch streaming shows all day long. However, throw streaming plus online gaming, online learning, et cetera and the network was not something I was going to show off.

I am not going to go through all of the changes/upgrades I made in this article. I will cover how I went from a single ISP point of failure to highly available, resilient internet connection.

First step... get another ISP. I do not plan to name the ISP's I ended up using. Depending on where you are in the world, you may only have access to a limited number of ISPs. For me, my original ISP (cable provider) was charging nearly $200 a month. The service wasn't bad, for the most part but this wasn't a business class service. So my first step was to go 5G. Getting that setup was fairly easy. The modem I was provided had the ability for an external antenna so I pick one up as well. That seemed to help some, but not as much as I would have hoped. While the service was good, I did notice that there was a large amount of jitter and the latency increased and the bandwidth increased. So I canceled the cable and ordered a 'fiber to the home ISP' that just happened to be expanding into my area.

Basic Setup

I run OpenBSD as my firewall (big surprise if you have read any of my other articles). Here is my basic setup.

#
# Firewall PF
#
all_nets = "192.168.0.0/16"
all6_nets = "2001:db8::/32"
vpn_subnet = "192.0.2.0/24"
wan_if="em3"
lan_if="em1"

set skip on lo
set skip on enc
set state-defaults pflow

table <inet_doh> persist file "/var/db/pf.internet_doh_servers"
table <internal_dns> persist file "/var/db/pf.internal_dns_servers"
table <banned> persist file "/var/db/pf.banned"
table <sshguard> persist
table <time_blocked> persist

partial /etc/pf.conf

Pretty standard, first specify macros, next setup 'skip' and other global settings, followed up by common tables.

# default block
block log
# block anything in the time_blocked table
block quick from <time_blocked>
# block bad hosts
block quick from <banned>
# capture NTP from any internal sources and respond for them
pass in proto udp from $all_nets to port 123 rdr-to 127.0.0.1 port 123
pass in inet6 proto udp from $all6_nets to port 123 rdr-to ::1 port 123

pass out inet6 proto icmp6 from self to $all6_nets

# pass all and pfsync carp protocol
pass out quick on carp proto carp
pass quick on $lan_if proto pfsync

partial /etc/pf.conf

Next setup the common rules. Default block and log, block time based addresses, quick block of banned addresses. Capture all NTP and respond from ourselves. And finally permit CARP and PFSYNC.

Finally specify my zones.

include "/etc/pf.d/pf.anchor.lan"
include "/etc/pf.d/pf.anchor.wan"

partial /etc/pf.conf

My actual configuration has a few more zones than this, but for this example we'll focus on 2.

# WAN Anchor
# NOTE: (WAN:0) will be different per FW pair
#

anchor "wan" on WAN {
  # block everything in
  block in log
  # match outbound traffic to WAN interfaces
  match out nat-to (WAN:0)
  # pass traffic out
  pass out set queue(DEFAULT, ACK)
  # stop DNS/DOH bypass
  block out quick log proto tcp to <inet_doh> port 443 ! tagged DNS
  block out quick log proto { udp tcp } to port { 53 853 } ! tagged DNS
  # allow ourselves out
  pass out proto { udp tcp } to port { 53 853 } tagged DNS set queue(DNS)
  pass out proto { tcp udp } from self user = "_unbound" set queue(DNS)
  pass out proto { tcp udp } to port { 80, 443, 8443 } set queue(WEB, ACK)
}

/etc/pf.d/pf.anchor.wan

Anything tagged as a WAN interface will have this NAT applied. Basically it will block all inbound traffic and NAT all outbound traffic. It will block traffic to known DoH servers but allow my DNS servers out.

For the LAN interface

# PF LAN
anchor "lan" on LAN {
  pass log set queue(LAN_DEF,LAN_ACK)
  pass in inet6 from any to 64:ff9b::/96 af-to inet from (WAN:0) set queue(LAN_DEF,LAN_ACK)
  match in from <internal_dns> tag DNS set queue(LAN_DEF,LAN_ACK)
}

/etc/pf.d/pf.anchor.lan

Anything on the LAN is good?! Right. For now let's assume so, so pass it. If traffic comes from my internal DNS servers tag it for the WAN rule processing.

One thing that may not be familiar is the use of the queue() syntax. More on that later when we talk about buffer bloat.

Now with the PF rules in place, just set the groups on the interfaces.

group WAN
inet autoconf
up

/etc/hostname.em3

group LAN
inet 192.168.1.2/24
inet6 2001:db8::2 64
up

/etc/hostname.em1

Don't forget to enable IP forwarding or the firewall will not work as expected.

# sysctl net.inet.ip.forwarding=1
# sysctl net.inet6.ip6.forwarding=1

sysctl forwarding commands

Since we plan on having redundant firewalls to eliminate a device failure we will need to setup carp(4). This will provide a first hop redundancy protocol (FHRP). The setup of carp is very easy.

group LAN
vhid 3
carpdev em1
pass letmein
state master
inet 192.168.1.1/24
up

/etc/hostname.carp1

Match the group of the main interfaces. Set the ID. This much match on the peer router. Tie the carp interface to a physical interfaces and set a password (optional). Set the default state. Once should be 'master' the other 'backup'. Finally set the floating IP and bring the interface up.

By default the sysctl(8) for carp are setup to have carp enabled. To see what the current settings are use the following command

# sysctl net.inet.carp
net.inet.carp.allow=1
net.inet.carp.preempt=0
net.inet.carp.log=2

sysctl command and output

Buffer Bloat... simply put

Network devices, especially router, have different speed interfaces. Today it is not unusual to have 1G / 10G on a home LAN and less than that for the WAN interface. Sometime an order of magnitude smaller. To compensate for this difference bandwidth, routers use temporary storage, called buffers, so store packets while they wait their turn to be sent on the slower interfaces. Without special configurations, packets are processed in a FIFO (first in, first out) queue. Meaning that if a larger packet is queued before a smaller packet, the smaller packet will need to wait until the larger packet is sent before it gets a turn.

Now let's consider that the smaller packet is an ACK. And that ACK is being sent in response to a on-line game. Do we want that packet waiting for some other non-time sensitive packets... say a web page? Probably not.

Fighting the bloat... Queuing

Fortunately, OpenBSD has an answer ready to go. Since version 6.2 OpenBSD has a queuing mechanism in PF that will help us overcome buffer bloat (called FQ-CoDel for those that are curious). We can specify our upload speed of the main FIFO queue and setup flow based sub-queues for our traffic.

# WAN
queue INET on $wan_if bandwidth 95M max 95M
queue ACK parent INET bandwidth 1M max 10M flows 1024 qlimit 1024
queue DNS parent INET bandwidth 5M flows 1024 qlimit 1024
queue WEB parent INET bandwidth 50M flows 1024 qlimit 1024
queue SSH parent INET bandwidth 20M flows 1024 qlimit 1024
queue GAME parent INET bandwidth 20M flows 1024 qlimit 1024
queue DEFAULT parent INET bandwidth 90M flows 1024 qlimit 1024 default
# LAN
queue LAN on $lan_if bandwidth 1G
queue LAN_ACK parent LAN bandwidth 1G flows 1024 qlimit 1024
queue LAN_DEF parent LAN bandwidth 1G flows 1024 qlimit 1024 default

partial /etc/pf.conf

As mentioned the INET queue has a bandwidth set to match the upload speed of the connection as well as the max bandwidth setting. It is important, for queuing to work correctly that the bandwidth be set around 95% of the available upstream bandwidth.

Next create one or more child queues and set the bandwidth they can should consume. If max is not specified, the queues will be allowed to take more, if available. Here a qlimit and flow limit for each queue is also specified increasing from the default. This is optional. To see if a queue is getting maxed out and dropping traffic you can use the following command. Make notes of any dropped pkts or qlength being larger than 0.

# pfctl -v -sq
queue INET on em3 bandwidth 100M
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
queue ACK parent INET flows 1024 bandwidth 1M qlimit 1024
  [ pkts:   31459517  bytes: 1770765239  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue DNS parent INET flows 1024 bandwidth 5M qlimit 1024
  [ pkts:     465724  bytes:   43885028  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue WEB parent INET flows 1024 bandwidth 50M qlimit 1024
  [ pkts:    2913536  bytes: 2104425594  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue SSH parent INET flows 1024 bandwidth 20M qlimit 1024
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue GAME parent INET flows 1024 bandwidth 20M qlimit 1024
  [ pkts:       5751  bytes:     596945  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue DEFAULT parent INET flows 1024 bandwidth 90M default qlimit 1024
  [ pkts:    1723990  bytes:  633385764  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue LAN on em1 bandwidth 1G
  [ pkts:          0  bytes:          0  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]
queue LAN_ACK parent LAN flows 1024 bandwidth 1G qlimit 1024
  [ pkts:     749181  bytes:   50240090  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]
queue LAN_DEF parent LAN flows 1024 bandwidth 1G default qlimit 1024
  [ pkts: 1229933940  bytes: 320469964876  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/1024 ]

The systat(1) command can also give you a real time look at the queues in action.

# systat queue

systat command example

Got one... now repeat for two

Now that we have one firewall up and working we can repeat the process for the second firewall with the second ISP. After that we will need to tie them together.

Since both firewalls are up and functioning, let's make sure that CARP is working as well as PFSYNC. The command

# ifconfig carp

example showing CARP interfaces

will show all carp and pfsync interfaces and their current state. On one firewall you should see the interface be MASTER and on the other firewall it should state BACKUP. If it states anything else, CARP is not working and this problem will need to be solved first.

Now to test failover of CARP. The easiest way to do this is on the master systems demote the carp interface. To check the current demotion use this command

# ifconfig -g carp
carp: carp demote count 0

example showing carp demotion counter

Now let's demote the master firewall and check again.

# ifconfig -g carp carpdemote
# ifconfig -g carp
carp: carp demote count 1

example demoting carp

Now check the status of the "backup" firewall. We should see the carp interfaces goto MASTER state. Checking the dmesg should also show the CARP interface state transition.

To remove the demotion us the following command

# ifconfig -g carp -carpdemote

If we have not enabled preempt in the sysctls, the MASTER status will not fail back over to the "primary" firewall until the "backup" firewall is demoted.

There may be reasons why we want to keep one firewall/ISP as the "primary" unless it has failed. In those cases make sure the set the net.inet.cap.preempt to 1.

Automatic demotion

Now that we have the firewalls and ISP setup and we've tested performance and failover we need failover to happen automatically. In OpenBSD this is easily accomplished with ifstated(8) and some shell scripts.

Before we get to the config, lets consider what we want to happen. First if the WAN interface is down, we can't pass packet we want the failover to occur. But what happens if the interface is up but he ISP's router isn't available. We should test for the default gateway to be up. Next what happens if the default gateway is up but there is no path to the Internet? Next what happens if there is a path to the Internet but performance is really bad? So here we have 4 tests...

WAN interface up
WAN default gateway present
Internet accessable
Internet performance ok

We can check all those items with ifstated daemon. Below is a config an helper script to continually test for the "health" of the connection.

# 
dmz_if_up = 'em3.link.up'
dmz_if_down = 'em3.link.down'

external_gw='"ping -q -c 1 -w 1 `route -n show -inet | grep default | awk \'{ print $2 }\'` >/dev/null 2>&1" every 60'

performance='"/root/check_perf.sh X.X.X.X >/dev/null 2>&1" every 60'

# ifstated starts up with the first defined state
state neutral {
        if $dmz_if_down {
                run "logger -st ifstated 'uplink down'"
                set-state demoted
        }
        if ! $external_gw {
                run "logger -st ifstated 'could not reach cable router'"
                set-state demoted
        }
        if ! $performance {
                run "logger -st ifstated 'perforamce degraded'"
                set-state demoted
        }
}

state demoted {
        init {
                run "ifconfig -g carp carpdemote" 
        }
        if $dmz_if_up && $external_gw && $performance {
                run "logger -st ifstated 'cable router ok'"
                # remove our carp demotion
                run "ifconfig -g carp -carpdemote" 
                set-state neutral
        }
}

## commands in the global scope are always run
if carp3.link.up
        run "logger -st ifstated 'carp3 is master'"
if carp3.link.down
        run "logger -st ifstated 'carp3 is backup'"

/etc/ifstated.conf

#!/bin/sh

if [ $# -lt 1 ]; then
        echo "${0##*/} <host>"
        exit 2
fi

LIMIT=${LIMIT:-200}
TRIES=${TRIES:-5}
WAIT=${WAIT:-1}

rtt=`ping -q -c ${TRIES} -w ${WAIT} ${1} 2>&1 | sed -n '/^round-trip/p' | cut -f4 -d" " | cut -f2 -d"/"`

if [ -z "${rtt}" ]; then
  echo "failed to get RTT" >&2
fi
rtt=${rtt%%.*}
if [ $rtt -gt $LIMIT ]; then
  echo "too high"
  exit 1
else
  echo "ok"
fi
exit 0

/root/check_perf.sh script

The first checks dmz_if_up/down check to make sure the WAN interface is up. The next check external_gw verifies that the gateways is pingable every 60 seconds. The final test performance checks for both internet reachability as well as performance. Select and IP or host what we want to ping every 60 seconds. The script below will send 5 pings and calculate the average response time. If any test fails then the performance/reachability is considered "failed" and state is set to demoted. When entering demoted state, the first thing that is ran is the ifconfig demotion of CARP. Once the tests all succeed, the state is change back and the demotion is reversed.

Conclusion

After running in this environment for a while it has worked pretty well. By and large my "home user community" has been generally happy. The environment seem to "do the right thing" when failures occur.

Since traffic is split across 2 different ISP's when a failover occurs, all sessions are dropped and have to be re-established. By and large this is not a big deal. But since the IP addresses on the WAN interfaces are different there are a ton of errors with pfsync making me wonder I should even be worrying about pfsync across these firewalls.

OpenBGPD + RTBH = dynamic PF rules

2023-08-28T15:39:39Z

Photo by Andrew Kliatskyi / Unsplash

If you have been networking for a while you will come across the need to route
traffic to Null0 (black hole). For the uninitiated, this means that traffic will just be
dropped by the router. What if there was a way to automate adding these routes to firewall rules. Good news! There is.

OpenBGPD is capable BGP daemon in it's own right. It has a hidden capability. It can add routes to a PF table.

Why would you want to use this if you do not run a large routing network? Glad you asked. Let's say you have several servers on a hostile network doing server stuff (email, web, blog, etc). And being a good admin the have things pretty locked down. The servers keep getting harassed by ne'er-do-wells, filling logs with noise, etc. By using RTBH and OpenBGPD, harassment of one server is a block on all servers.

Headend Setup

First thing is to configure the BGP Headend router. This router, and in fact
all of the routing daemons, will not update the routing tables on the systems. They
are just setup to exchange routes between each other.

# The ASN does not need to be registered.  In fact having it not registered
# will add protection that these records will not slip out into the
# greater Internet.
ASN="65001"
# The RTBH community can also be anything we want.
RTBH="65535"
# This list cooresponds to the list of PF tables.  There can be as many or
# few as required.
# Local
#  001 sshguard
#  002 banned
#  003 aggressive

# global configuration
AS $ASN
# prevent adding routes to host routing table
fib-update no
nexthop qualify via default

# Devices to peer with. Use of a password, may allow the neighbors to be
# wild carded, but may increase risk.
group "peers" {
        announce IPv4 unicast
        announce IPv6 unicast
        remote-as $ASN
        # since we are utilizing iBGP, we need to have the headend
        # be the route-reflector so we will not need a full mesh.
        route-reflector
        # Let out peers come to us.
        passive
        tcp md5sig password "super secure password here"
        neighbor <IP address of spoke>
}
# Allow iBGP routes in and out.  If this configuration is part of a router then
# the allow filters will be more involved.
allow from ibgp
allow to ibgp
# Match any routes that have the RTBH community and add them to the rtbh table.
# Enable if headend box will also participate in RTBH.  Dont forget to update
# the PF config.  Will need, at a minimum, a rtbh table.
#match from ibgp community $RTBH:* set { pftable rtbh }

headend bgpd.conf

Now enable BGP on the headend and start it.

rcctl enable bgpd
rcctl start bgpd

Spoke Setup

Next each spoke will need 3 parts: PF config update; BGP config; and a helper program. The first part is an update to the PF rules running on the device. The
following updates will add PF table(s) and a drop rule.

###### partial PF configuration
# store the routes learned from BGP
table <rtbh> persist
# locally banned hosts added manually
table <banned> persist
# aggressive hosts added automatically by PF
table <aggressive> persist

block quick from <rtbh> label "rtbh host"
block quick from <banned> label "banned host"
block quick from <aggressive> label "aggressive host"

Validate and restart PF.

pfctl -nf /etc/pf.conf && pfctl -f /etc/pf.conf

The second part is the BGP config. This config will be exactly the same on all
the spoke servers.

# Almost exactly the same as the headend config
ASN="65001"
RTBH="65535"
# Local
#  001 sshguard
#  002 banned
#  003 aggressive

# global configuration
AS $ASN
fib-update no
nexthop qualify via default

group "peers" {
        announce IPv4 unicast
        announce IPv6 unicast
        remote-as $ASN
        tcp md5sig password "super secure password here"
        neighbor <IP address of headend>
}
allow from ibgp
allow to ibgp
match from ibgp community $RTBH:* set { pftable rtbh }

spoke bgpd.conf

Now enable and start bgpd.

rcctl enable bgpd
rcctl start bgpd

Now for the last part... the watcher daemon. OpenBGPD has the ability to add
and remove addresses/networks using a PF table. However, it does not monitor
PF tables for additions/subtractions that are made outside the BGP daemon.
This is where the watcher daemon comes in. It watches one or more tables for
additions/subtractions and utilizes bgpctl commands to add/remove routes as
they are added and removed from the table(s).

The script can be downloaded from github. Word of warning, this is a
minimal viable script. There are probably better programs out there. (If you
know of any feel free to drop me an email.) However, it is lightweight and
should run on any flavor of *nix out there.

Assuming you are configuring OpenBSD, there is a rc.d script. Enable the
daemon, set the tables to watch, and then start.

rcctl enable watchtabled
rcctl set watchtabled flag -t banned:65535:102 -t aggressive:65535:103
rcctl start watchtabled

Now with everything running lets take a look at the tables.

pfctl -T show -t table1 | wc -l

Typing this can get tedious quickly. I created a this script to help.

One thing we should notice is there are no records. While we have setup the
environment to watch and replicate records, we need to add records. There are
several ways this can happen. The most straight forward is to add them
manually to the table.

pfctl -T add -t banned 1.2.3.4

To see if the address is in the BGP daemon, use the bgpctl command. NOTE:
that the community filter will need to match the community that references the
specific PF table.

bgpctl show ip bgp community 65535:102

Another option for adding records more automatically is to utilize the PF
'overload' option.

#### partial PF config

pass in inet proto tcp from any to ($ext) port ssh keep state \
  (source-track rule, max-src-conn-rate 20/10, \
   overload <aggressive> flush global)

Here aggressive hosts will trigger an addition to the aggressive PF
table. NOTE: the max connection rate rule will need to be tailored to the
service. SSH on a console server is much different that SSH on an Email
server.

Last, but not least, is the use of some other application like sshguard or
fail2ban. The setup of these services is outside the scope of this post, but
once setup to add 'bad hosts' to PF tables, these tables can be watched and
trigger RTBH bans.

Conclusion

Hopefully this article has help explain remote triggered blackholes and how they can be used to protect multiple systems from outside attacks.

"IT Security Is Broken?"

2023-01-15T15:01:03Z

Photo by Egor Vikhrev / Unsplash

Go listen to Day Two Cloud 177: IT Security is Broken; Here's Ideas On How To Fix It. Great interview between Ethan Banks and Mick Douglas discussing some of the failures in approach to security in business protection.

Some key points I took from this

Prevention is good. Detection is key!
My security is against not only attackers, but also compliance and auditors. I need to survive all of them.
Mono culture of OS and cloud deployments mean that the attackers have "my" container, "my" apps in their labs to test against before they get to me.
Does security really protect the risk? Or am I spending $1M to protect $10.
And finally... "Security is a journey and not a destination" or maybe better put "Security is a process and not a product".