Don’t take this as an endorsement as it’s not really intended to be used for this purpose and I’m probably going to switch, but I’ve been using sendgrid for a few years as I was familiar with it from business use. I was using mandrill before that until they changed their pricing and some other things.

EDIT: also worth mentioning I have a grandfathered free plan.

I can reply yo emails. It’s what jaybone said – I use a 3rd party outbound (SMTP).

I use Thunderbird as my mail client. In there you can define multiple IMAP/pop accounts (inbound) and multiple SMTP (outbound). So when I click reply, the SMTP I have set for that email address is a 3rd party service that sends emails using my domain.

This does have a privacy implication, so whether that’s appropriate for you is for you to decide. The 3rd party can read and save a copy of your outbound email inclusive of any content included with it (like the email you’re replying to if you include it).

I’ve run a mail server since the late 90s. I gave up running my own outbound years ago. Whether it gets received is too inconsistent. Primarily, make sure the IP you use isn’t on any spam lists. If it is, get a new IP or try to submit appeals. Make sure you have the standard anti spam/authenticity measures implemented (they’re listed in the OP).

These are the minimum. Years ago it was enough to have reasonable delivery. Unfortunately these aren’t that helpful anymore because spammers also set them up (often with seemingly randomly generated domain names). I assume because not having them was a fairly strong signal for being a spammer, so now they also set them up, which also means there’s not really any configuration / setup you can do to seem legitimate (aside from mail volume and the content itself which will be analyzed).

Anyway running your own SMTP itself is easy. The problem is getting your mail to the inbox or at least spam/junk folder (your mail may get blackholed and not even show up in spam). At least when you get a rejection, you know something is wrong, but many times the receiving server will indicate a success code and simply not deliver it to the user (not even to their spam).

FOSS, learning curve, and niche field? I got you. VLSI layout software or SDR (software defined radio) software.

VLSI layout is kind of software allows you to design electronics hardware at the silicone/transistor level. IOW you have to draw the layers of materials that form transistors (cross poly silicon and p/n-diffused silicon to form) and connect them to metal layers. Learn how NAND and NOR gates are actually formed. Make a 1 bit data storage (flip flop), half adder, full adder. Go as far as you want.

Example software:

• Magic (what I used in school): https://opencircuitdesign.com/magic/
• klayout: https://www.klayout.de/

SDR. Less intense, but also niche and gives you lots of related things to learn/branch in to.

Can be used to receive and decode radio/satellite transmissions. Normally people would buy hardware and use software locally, but some people make their receivers accessible on the internet (websdr) which can give you a first intro to it. After you capture data you can run it through other software to decode (depending on what it is). For example, you can capture raw weather image data from satellites and decode it to get the images that the satellite is transmitting as it’s capturing the data.

A list of some SDRs you can access over the internet (doesn’t require hardware purchase to use): http://websdr.org/

It probably won’t be immediately obvious how to use either SDR or VLSI software but you asked for niche and learning curve, so enjoy. ;)

EDIT: Fixed some typos and grammar.