It sounds pretty reasonable. As long as you keep SSH patched and keep the key safe it should be quite locked down. Do double-check that password login isn’t allowed (or that all users have a very strong password).
One non-security note is be careful with rsync backup. Generally rsync isn’t considered a backup as any mistakes made in the source will be propagated to the “backup” on next sync. Although there are ways to use rsync to take good backups (like copying to a new directory for each backup).




Generally speaking it will be fine. SSH will also refuse keys with open permissions so you would notice if it was wide-open to other users of the device.
But you know if you are running random code or AI harnesses as that user it can be at risk. Or if you copy around the key all over the place it is more likely to leak. But generally speaking you are secure by default, just don’t do something dumb with the key and you’ll have no problems.