The Outage That Wrote the Specification

When a Scheduling Service Broke, We Built Our Own

Sometimes a project starts with a roadmap.

Sometimes it starts with an error message.

In this case, a hosted booking system that had been happily talking to our on-premises Exchange servers simply stopped.

The error?

Cannot read properties of undefined (reading 'headers')

Nine hours later, we had traced the failure, replaced the hosted service with a self-hosted installation, discovered exactly where that solution falls short, and accidentally written most of the specification for the scheduler we really want.

So, naturally, we're probably going to build one.

First assumption: it's us

When an outside service suddenly can't reach something you operate, the responsible place to start is your own infrastructure.

And we have plenty of places capable of saying "no."

DNS filtering. An IP blocklist shared with the edge firewall. User-agent filtering on the Exchange front end. Anti-spam enforcement. Certificates. Authentication. Firewall rules.

So we checked them.

The service's addresses weren't blocked. DNS wasn't filtering them. The user-agent rules didn't match. Certificates were valid and chained correctly. The Exchange service accounts were enabled, unlocked, and unchanged.

Then the firewall logs provided the useful clue.

The last connection from the scheduling provider arrived on August 29:

2026-08-29 16:20:31
3.x.x.x:4180 → 192.x.x.x:443
ACCESS FORWARD … DNAT Packet, ACCEPT

It wasn't rejected.

It was accepted.

And then their traffic simply stopped arriving.

During a live reconnection attempt, while the scheduling service was actively reporting a failure, the firewall recorded 844 connections to the same Exchange servers.

All forwarded.

None blocked.

Outlook and Apple Mail continued working normally.

That changed the question completely.

What it looked like: Something on our network had started dropping their traffic.

What was actually happening: Their traffic wasn't reaching our network at all.

The error message was hiding the real error

That wonderfully descriptive undefined headers message turned out to be a second failure masking the first one.

Reduced to its basic shape, the code handling the Exchange Web Services connection effectively did this:

const response = await axios(opt).catch(err => err.response);
let type2msg = decodeType2Message(
    response.headers['www-authenticate']
);

That works if the server returns an HTTP response.

But what happens if there is no response?

A DNS failure, TLS failure, timeout, routing problem, or refused connection can leave err.response undefined.

The code then throws away the original network error and immediately tries to read:

response.headers

...from something that doesn't exist.

The useful error disappears and is replaced by a JavaScript type error.

That little mistake sent the investigation in entirely the wrong direction.

A swallowed error doesn't merely hide the cause. Sometimes it actively points you somewhere else.

At that point, we stopped debugging someone else's outage

The product also has a self-hosted edition.

So rather than wait for the hosted service to recover, we spun one up.

By the end of the evening, it was taking real bookings.

The new environment consists of:

Windows Server 2025
Node
PostgreSQL 16
Reverse proxying through an existing public web server
Let's Encrypt certificates using DNS-01 validation
HSTS and branded error pages
Multiple Exchange mailboxes accessed through EWS for conflict checking
PayPal payments with webhook verification
Public booking pages with account creation restricted at the proxy
Nightly PostgreSQL backups using a group Managed Service Account, with no stored password

And importantly, the backup wasn't considered finished just because pg_dump successfully produced a file.

It was test-restored.

A backup you haven't restored is mostly an optimistic theory.

The installation had a few surprises of its own

The self-hosted version worked, but getting there exposed several things worth mentioning.

The build can fail out of the box

One build script runs a formatter against a generated file that the project's own formatter configuration explicitly excludes.

The formatter returns a failure code.

The script treats that as fatal.

Build over.

Remote PowerShell ate a background process

A dependency installation launched from a remote PowerShell session mysteriously vanished halfway through execution.

No useful error. Just a truncated log.

The reason was straightforward once identified: processes launched that way can be terminated when the remote session closes.

A tracked configuration file could have exposed secrets

The database configuration is distributed as a symbolic link.

On Windows, the documented workaround causes it to become an ordinary file containing real database credentials and encryption keys.

Unfortunately, it's also tracked by Git.

One casual:

git add -A

...and those credentials could become part of the repository's permanent history.

That was corrected before Git got any exciting new secrets to remember.

Then our own spam filter caught us

This was probably my favorite failure of the evening.

The new scheduler sent its verification message.

Our anti-spam system quarantined it.

The analysis essentially said:

Origin: unknown

DMARC=fail
SPF=none
DKIM=none

LLM spam score: 95

Message claims one domain but links to another domain
using a tokenized URL.

Verdict: credential phishing

My first reaction could have been:

"Fine. Add it to the allow list."

Except the filter was right.

Look at that message from the perspective of someone who doesn't know we just installed the application.

A sender claiming a domain with a DMARC reject policy.

No SPF alignment.

No DKIM signature.

An unknown origin.

A link pointing somewhere else.

A token embedded in the URL.

That's not the filter being overzealous.

That's us sending something that looks remarkably like credential phishing.

So we didn't add an exception.

We fixed the mail submission.

SPF aligns. DKIM signs. DMARC passes. The sender is authenticated.

Allow-list entries added: zero.

That's exactly how I like that number.

And then we found the wall

By midnight, the replacement scheduler worked quite well.

It also became very clear what it cannot become without substantial modification.

The self-hosted edition intentionally excludes much of the product's multi-tenant functionality.

Teams.

Organizations.

Single sign-on.

Other enterprise features.

Those aren't forgotten features. They're part of the commercial offering.

For us, the most important limitation is even simpler:

The sender address is global to the entire installation.

One scheduler installation.

One outbound identity.

For a single business, that's perfectly reasonable.

For hosting scheduling services for multiple clients, it's a hard architectural boundary.

And the edition that solves those problems is precisely the edition that isn't part of the self-hosted community product.

Which raises an interesting question

What would it take to build the scheduler we actually need?

Probably less than rebuilding the product we just installed.

Our requirements are oddly specific.

We don't need to support millions of unrelated users.

We don't need Google Calendar.

We don't need Microsoft 365 integration.

We don't need to become another general-purpose scheduling platform.

What we do want is something surprisingly difficult to find:

A multi-client scheduling system that works extremely well with on-premises Exchange.

For our use, multi-tenancy doesn't require an elaborate enterprise organization framework.

It can start with a tenant ID in the database from migration number one.

We already operate the pieces that tend to make projects like this unpleasant:

Authentication across multiple directories
Exchange
Mail delivery and authentication
Web hosting
Reverse proxies
TLS
Databases
Payments
Deployment
Monitoring

And after this exercise, we now have a considerably deeper understanding of the EWS and NTLM side of the equation than we had the day before.

That doesn't make the rest trivial.

Scheduling software contains some deceptively deep rabbit holes.

Time zones and daylight saving time alone can consume an alarming quantity of caffeine.

Then come buffers, minimum notice periods, availability overrides, cancellation rules, rescheduling, reminders, no-shows, recurring availability, payment states, and all the little exceptions that individually sound easy and collectively become the actual product.

But something useful happened during those nine hours.

We weren't merely installing somebody else's scheduler.

We were discovering our requirements.

By the time the replacement was running, we knew exactly what we liked, exactly what we didn't, and exactly where its architecture stopped matching ours.

So the replacement is staying.

It's working.

It's taking bookings.

It's earning its keep.

And it isn't going anywhere until whatever replaces it is genuinely better.

Not merely ours.

9 hours from failure to replacement 18.7 GB of firewall logs examined 844 connections observed during testing 0 firewall blocks found 0 spam-filter exceptions added 1 unexpectedly detailed software specification produced

Not quite how the evening was supposed to go.

But considerably more interesting.